Add Redis Cloud Pro and Trino full-data ClickBench results

redis-cloud/.gitignore

@@ -0,0 +1,8 @@
+/.venv/
+/hits.parquet
+/__pycache__/
+/cloud-etc/
+/cloud-results/
+/connector/
+/plugin/
+/*-count.csv

redis-cloud/README.md

@@ -0,0 +1,136 @@
+# Redis Cloud Pro (Trino)
+
+ClickBench's 43 SQL queries run through the Redis SQL Trino connector. Data lives
+in Redis Cloud Pro all-RAM hashes with a Redis Query Engine index. Trino evaluates
+SQL operations the connector cannot push into Redis. This measures the combined
+Redis + Trino deployment, including receiving all query result pages at the client.
+
+## Setup
+
+Use Ubuntu 24.04 on a dedicated AWS runner in the Redis subscription's VPC, region
+and availability zone. Use AZ IDs to verify physical placement when accounts differ.
+The configuration below uses a single-AZ, non-replicated database for benchmarking.
+It does not provision Flex. Flex v2 is outside this entry until tested separately.
+
+The initial test deployment uses:
+
+- AWS us-east-1a / use1-az6; same AWS account and VPC for client and server.
+- Redis Cloud Pro 8.6, 1,000 GB dataset capacity, 40 shards, RESP3, OSS Cluster API
+  enabled, default query performance factor, `noeviction`, AOF every second.
+- Redis hosts: three r8g.16xlarge instances. Each data disk is gp3, 1,598 GB,
+  16,000 IOPS, 1,000 MiB/s. Root volumes retain their defaults.
+- Trino client: r7a.4xlarge (16 vCPUs, 128 GiB RAM); 500 GB gp3 data disk, 16,000 IOPS,
+  1,000 MiB/s, plus a 20 GB gp3 boot disk with default 3,000 IOPS / 125 MiB/s. Trino JVM: 96 GiB heap, 64 GB per-query memory, 24 GB heap headroom.
+- Trino 483 and the exact connector revision in `versions.env`.
+
+These are provisioning settings, not measured results. Record actual versions,
+capacity, shard count, worker settings, billing and disk settings with every run.
+A 10,000-row real-data sample used about 75 MB including hashes and indexes;
+monitor the full load because high-cardinality index growth is not necessarily linear.
+
+Install the connector and loader:
+
+```sh
+./install
+```
+
+Create a private JSON connection file outside the checkout, mode 0600:
+
+```json
+{"host":"PRIVATE_REDIS_HOST","port":14756,"password":"DATABASE_PASSWORD"}
+```
+
+Use `"ssl": true` when connecting with TLS; the server certificate must validate.
+The Cloud API account/user keys are provisioning credentials and are not needed
+by the benchmark runner. Never put any credentials in results or committed files.
+The default and OSS Cluster API paths use the same private endpoint, database,
+credentials, data and Trino settings; the database's `supportOSSClusterApi` flag and `redisearch.cluster` change together.
+
+```sh
+export REDIS_CONNECTION_FILE=/secure/path/redis-connection.json
+export PLUGIN_DIR="$PWD/plugin"
+./start-cloud
+curl --fail --location --output ../hits.parquet \
+  https://datasets.clickhouse.com/hits_compatible/athena/hits.parquet
+./load-cloud | tee load.log
+RESULT_DIR=cloud-results-default ./run.sh --mode default | tee default-run.log
+# Enable supportOSSClusterApi on the Cloud database; wait until active.
+RESULT_DIR=cloud-results-cluster ./run.sh --mode cluster | tee cluster-run.log
+```
+
+Wait for Trino `/v1/info` to report `starting: false` before loading. The dataset
+must be completely downloaded before the load timer starts. Loading includes
+indexing and verifies exactly 99,997,497 source rows and indexed documents.
+Hash keys use physical row ordinals, because WatchID is not unique. Numeric IDs
+are written as decimal integers without conversion to doubles. Dates are ISO dates;
+Parquet timestamp seconds become epoch milliseconds, as the connector requires.
+All 105 columns are created with the connector's standard schema; no extra
+materialized views, preaggregations, or manually tuned indexes are used.
+
+For a database already using OSS Cluster API, load with `BENCHMARK_MODE=cluster`.
+The default `LOAD_WORKERS=8` can load independent Parquet row groups in parallel; each
+row retains its original physical ordinal, and all worker errors abort the run.
+
+Redis query timeouts must fail explicitly (`CONFIG SET search-on-timeout fail`)
+so partial results are not treated as valid. Preserve this setting with the run metadata.
+
+`load-cloud` requires an empty dedicated database. It never flushes a database.
+For another run, create a fresh database or explicitly remove only the benchmark
+data. An interrupted parallel load of the identical source file can use
+`load.py --workers 8 --resume`; completed row groups are reused, partial groups
+are rewritten by the same physical ordinals, and the final global index count
+is still required. Do not publish resumed wall time as a fresh-load measurement. `load.py --limit` exists for smoke tests; never submit subset timings.
+
+## API comparison and result reporting
+
+`run-cloud.py` runs three tries per query in the selected mode. Run the default
+API phase with Cloud OSS Cluster API disabled, then enable it for the cluster
+phase. Changing only the client flag while OSS Cluster API is enabled produces
+MOVED errors for ordinary clients. Preserve the same data and hardware across
+phases; record their order, and repeat in reverse order if comparing cache effects. It restarts Trino and clears the runner's page cache
+before each mode/query block. Redis Cloud remains running. Thus **both result files
+must have the `no-cold` tag**: the first try is not a true cold Redis measurement.
+There is no query-result cache. Each query consumes and writes all returned results.
+Failed queries have `null` timings, with errors retained in `samples.jsonl`.
+
+The selected `RESULT_DIR` contains the three timings per query, result CSVs, errors, and
+checksums. Retain these as raw evidence. Compare successful query outputs between
+modes, accounting for unspecified order and ties at a LIMIT boundary, before
+publishing timings. Include correctness failures as `null`, not fast successful
+numbers. Check representative numeric aggregates against a reference engine.
+
+The connector revision includes a correctness fix for grouping NUMERIC BIGINTs:
+Redis's double-based grouping can merge adjacent IDs beyond 2^53. Those groups are
+computed by Trino from the exact hash values. See merged redis-sql-trino PR #125. The connector also sets an explicit 20-minute
+aggregation timeout (PR #127), because Redis Cloud rejects changing its server
+search timeout through `CONFIG SET`. The Redis URI includes `?timeout=1200s` so Lettuce commands have the same limit.
+Trino retains the same 20-minute query limit.
+
+After a full verified run, add two JSON files under `results/YYYYMMDD/`, labeled
+`Redis Cloud (Trino)` and `Redis Cloud (Trino, OSS Cluster API)`. Include all 43
+three-value arrays, load time, actual data size, machine metadata, and `no-cold`.
+For data size include hash/index memory and persistent log bytes where available;
+do not substitute provisioned capacity or the Parquet source size. If persistent
+log size is unavailable, leave `data_size` null and record measured Redis RAM
+bytes separately; an AOF-enabled Cloud database reporting zero AOF bytes is not
+evidence that its transaction logs occupy zero bytes. Do not create
+result files until the full run and correctness checks finish.
+
+Stop and remove the task's Trino container and delete the dedicated Redis Cloud
+subscription and runner after downloading results. Deleting the subscription's
+managed VPC also requires terminating the runner and deleting its security group
+first. Retain source/results; discard private credential files and task SSH keys.
+
+## Current-code requirement
+
+This submission pins `d83d504d65cd5b3bd14fd9bd559f5acad6ffedf2` (merged PR154), including automatic parallel scans. `install` checks the pin
+against current origin/master and builds a Git archive of that exact commit,
+excluding stale compiled files and local edits. It records every plugin JAR's
+SHA-256 and the build image ID in `plugin/connector-build.json`.
+
+`start-cloud` requires the pin to match live master and validates all plugin JARs.
+It labels the Trino container with the revision and manifest checksum. Both query
+runners recheck live master, plugin bytes, container labels and its read-only
+plugin mount before testing. If master advances, update `versions.env`, rebuild,
+and recreate Trino. Runs stay pinned once started; do not replace artifacts in an
+active run. Network verification failures stop the run rather than using stale code.

redis-cloud/benchmark_version.py

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+"""Fail closed on stale/mismatched benchmark connector artifacts."""
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+
+REMOTE = 'https://github.com/redis-field-engineering/redis-sql-trino.git'
+MANIFEST = 'connector-build.json'
+
+
+def digest(path):
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def jars(plugin):
+    return {str(path.relative_to(plugin)): digest(path) for path in sorted(plugin.rglob('*.jar'))}
+
+
+def latest_revision():
+    output = subprocess.check_output(['git', 'ls-remote', REMOTE, 'refs/heads/master'], text=True)
+    lines = output.splitlines()
+    if len(lines) != 1 or not re.fullmatch(r'[0-9a-f]{40}\s+refs/heads/master', lines[0]):
+        raise RuntimeError('Cannot establish latest master revision')
+    return lines[0].split()[0]
+
+
+def record(plugin, revision, image_id):
+    if not re.fullmatch(r'[0-9a-f]{40}', revision):
+        raise RuntimeError('Require an immutable full connector revision')
+    files = jars(plugin)
+    if not any(Path(name).name.startswith('redis-sql-trino-') and not Path(name).name.endswith(('-sources.jar', '-javadoc.jar', '-tests.jar')) for name in files):
+        raise RuntimeError('Connector JAR missing from plugin')
+    manifest = dict(connector_revision=revision, build_image_id=image_id, jar_sha256=files)
+    (plugin / MANIFEST).write_text(json.dumps(manifest, indent=2) + '\n')
+    return manifest
+
+
+def verify(plugin, expected, latest=True, container=None):
+    manifest_path = plugin / MANIFEST
+    manifest = json.loads(manifest_path.read_text())
+    if manifest['connector_revision'] != expected:
+        raise RuntimeError('Plugin revision differs from benchmark pin; rebuild')
+    if latest and latest_revision() != expected:
+        raise RuntimeError('Benchmark pin is behind master; update versions.env and rebuild')
+    if not manifest.get('jar_sha256') or manifest['jar_sha256'] != jars(plugin):
+        raise RuntimeError('Plugin JARs differ from recorded build; rebuild')
+    if container:
+        info = json.loads(subprocess.check_output(['docker', 'inspect', container], text=True))[0]
+        labels = info['Config'].get('Labels') or {}
+        if labels.get('clickbench.connector-revision') != expected or labels.get('clickbench.plugin-manifest-sha256') != digest(manifest_path):
+            raise RuntimeError('Running Trino has an older connector build; recreate container')
+        mounts = [m for m in info.get('Mounts', []) if m.get('Destination') == '/usr/lib/trino/plugin/redisearch']
+        if len(mounts) != 1 or Path(mounts[0]['Source']).resolve() != plugin.resolve() or mounts[0].get('RW', True):
+            raise RuntimeError('Running Trino does not mount the verified plugin read-only')
+    return manifest
+
+
+def require_current_run(target):
+    root = Path(__file__).resolve().parent
+    values = dict(line.split('=', 1) for line in (root / 'versions.env').read_text().splitlines() if '=' in line)
+    plugin = Path(os.environ.get('PLUGIN_DIR', str(root / 'plugin'))).resolve()
+    manifest = verify(plugin, values['CONNECTOR_REVISION'], container='clickbench-cloud-trino')
+    path = target / MANIFEST
+    if path.exists() and json.loads(path.read_text()) != manifest:
+        raise RuntimeError('Result directory contains another connector build; use a fresh directory')
+    if not path.exists() and ((target / 'samples.jsonl').exists() or (target / 'complete.json').exists()):
+        raise RuntimeError('Existing results lack build provenance; preserve them and use a fresh directory')
+    target.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(manifest, indent=2) + '\n')
+    return manifest
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('action', choices=['record', 'verify'])
+    parser.add_argument('--plugin', type=Path, required=True)
+    parser.add_argument('--revision', required=True)
+    parser.add_argument('--image-id')
+    args = parser.parse_args()
+    if args.action == 'record':
+        if not args.image_id:
+            parser.error('record requires --image-id')
+        record(args.plugin, args.revision, args.image_id)
+    else:
+        verify(args.plugin, args.revision)
+    print(args.revision)
+
+
+if __name__ == '__main__':
+    main()

redis-cloud/create.sql

@@ -0,0 +1,108 @@
+CREATE TABLE hits
+(
+    WatchID BIGINT,
+    JavaEnable SMALLINT,
+    Title varchar,
+    GoodEvent SMALLINT,
+    EventTime timestamp(3),
+    EventDate Date,
+    CounterID INTEGER,
+    ClientIP INTEGER,
+    RegionID INTEGER,
+    UserID BIGINT,
+    CounterClass SMALLINT,
+    OS SMALLINT,
+    UserAgent SMALLINT,
+    URL varchar,
+    Referer varchar,
+    IsRefresh SMALLINT,
+    RefererCategoryID SMALLINT,
+    RefererRegionID INTEGER,
+    URLCategoryID SMALLINT,
+    URLRegionID INTEGER,
+    ResolutionWidth SMALLINT,
+    ResolutionHeight SMALLINT,
+    ResolutionDepth SMALLINT,
+    FlashMajor SMALLINT,
+    FlashMinor SMALLINT,
+    FlashMinor2 varchar,
+    NetMajor SMALLINT,
+    NetMinor SMALLINT,
+    UserAgentMajor SMALLINT,
+    UserAgentMinor varchar,
+    CookieEnable SMALLINT,
+    JavascriptEnable SMALLINT,
+    IsMobile SMALLINT,
+    MobilePhone SMALLINT,
+    MobilePhoneModel varchar,
+    Params varchar,
+    IPNetworkID INTEGER,
+    TraficSourceID SMALLINT,
+    SearchEngineID SMALLINT,
+    SearchPhrase varchar,
+    AdvEngineID SMALLINT,
+    IsArtifical SMALLINT,
+    WindowClientWidth SMALLINT,
+    WindowClientHeight SMALLINT,
+    ClientTimeZone SMALLINT,
+    ClientEventTime timestamp(3),
+    SilverlightVersion1 SMALLINT,
+    SilverlightVersion2 SMALLINT,
+    SilverlightVersion3 INTEGER,
+    SilverlightVersion4 SMALLINT,
+    PageCharset varchar,
+    CodeVersion INTEGER,
+    IsLink SMALLINT,
+    IsDownload SMALLINT,
+    IsNotBounce SMALLINT,
+    FUniqID BIGINT,
+    OriginalURL varchar,
+    HID INTEGER,
+    IsOldCounter SMALLINT,
+    IsEvent SMALLINT,
+    IsParameter SMALLINT,
+    DontCountHits SMALLINT,
+    WithHash SMALLINT,
+    HitColor varchar,
+    LocalEventTime timestamp(3),
+    Age SMALLINT,
+    Sex SMALLINT,
+    Income SMALLINT,
+    Interests SMALLINT,
+    Robotness SMALLINT,
+    RemoteIP INTEGER,
+    WindowName INTEGER,
+    OpenerName INTEGER,
+    HistoryLength SMALLINT,
+    BrowserLanguage varchar,
+    BrowserCountry varchar,
+    SocialNetwork varchar,
+    SocialAction varchar,
+    HTTPError SMALLINT,
+    SendTiming INTEGER,
+    DNSTiming INTEGER,
+    ConnectTiming INTEGER,
+    ResponseStartTiming INTEGER,
+    ResponseEndTiming INTEGER,
+    FetchTiming INTEGER,
+    SocialSourceNetworkID SMALLINT,
+    SocialSourcePage varchar,
+    ParamPrice BIGINT,
+    ParamOrderID varchar,
+    ParamCurrency varchar,
+    ParamCurrencyID SMALLINT,
+    OpenstatServiceName varchar,
+    OpenstatCampaignID varchar,
+    OpenstatAdID varchar,
+    OpenstatSourceID varchar,
+    UTMSource varchar,
+    UTMMedium varchar,
+    UTMCampaign varchar,
+    UTMContent varchar,
+    UTMTerm varchar,
+    FromTag varchar,
+    HasGCLID SMALLINT,
+    RefererHash BIGINT,
+    URLHash BIGINT,
+    CLID INTEGER
+);

redis-cloud/install

@@ -0,0 +1,32 @@
+#!/bin/bash
+set -euo pipefail
+cd "$(dirname "$0")"
+sudo apt-get update
+sudo apt-get install -y docker.io python3-venv git curl wget
+python3 -m venv .venv
+.venv/bin/pip install -r requirements.txt
+source ./versions.env
+if [ ! -d connector ]; then git clone https://github.com/redis-field-engineering/redis-sql-trino.git connector; fi
+git -C connector fetch origin master
+latest=$(git -C connector rev-parse origin/master)
+[ "$latest" = "$CONNECTOR_REVISION" ] || { printf 'Update CONNECTOR_REVISION to latest master and rebuild\n' >&2; exit 1; }
+# Archive the pinned tree: ignored target/ and other local files cannot enter the build.
+context=$(mktemp -d)
+plugin_stage=$(mktemp -d)
+cid=
+cleanup() {
+    [ -z "$cid" ] || docker rm "$cid" >/dev/null 2>&1 || true
+    rm -rf "$context" "$plugin_stage"
+}
+trap cleanup EXIT
+git -C connector archive "$CONNECTOR_REVISION" | tar -x -C "$context"
+docker build --pull -t "clickbench-connector:$CONNECTOR_REVISION" "$context"
+cid=$(docker create "clickbench-connector:$CONNECTOR_REVISION")
+docker cp "$cid:/usr/lib/trino/plugin/redisearch/." "$plugin_stage/"
+image_id=$(docker image inspect --format '{{.Id}}' "clickbench-connector:$CONNECTOR_REVISION")
+.venv/bin/python benchmark_version.py record --plugin "$plugin_stage" \
+    --revision "$CONNECTOR_REVISION" --image-id "$image_id"
+.venv/bin/python benchmark_version.py verify --plugin "$plugin_stage" --revision "$CONNECTOR_REVISION"
+# Retain the previous build instead of mixing dependencies from two versions.
+if [ -e plugin ]; then mv plugin "plugin-previous-$(date +%s)"; fi
+mv "$plugin_stage" plugin