Repository navigation
Conversation
…oducer Stack Encrypt can produce one EQL type because its ciphertext, equality and order encodings disagree about which kinds exist, and the mapping from a kind to a Rust type lives in stack-encrypt (Scalar, dynamic::Value) instead of vitaminc. EQL also cannot tell a Stack Encrypt column from a cipherstash-client one, so a query from one producer against the other's column matches nothing, silently. ADR-0002 records the decisions from the value-encodings RFC review: the new kinds and their frozen tags, one canonical form per kind for terms, term derivation moving into vitaminc-prf and a new vitaminc-ore, and EQL v4 as the v3 SQL emitted under a second name with "v": 4 envelopes, so Postgres keeps the two producers apart. The plan-builder plan defined "EQL v4" as a name for a Stack Encrypt payload in the v3 envelope and eql_v3 domains; it now points at the ADR.
…m stack Starting vitaminc-ore under ADR-0002 surfaced four problems: order encodings, scheme traits and term derivation spread over three repositories; scheme crates that accept raw, hand-encoded bytes; vitaminc-prf reaching orderable-bytes through aead-value's canonical module; and one canonical form shared by equality and order, which rules out leakage-tuned domains such as an exact timestamp for equality with day resolution for order. The plan settles the replacement design: vitaminc-ore owns the order encodings and an OrderScheme trait with a sealed input, and ore-rs and cllw-ore implement it; each term crate applies a semantic baseline and per-domain transforms named in its labels; PRF has no ordering dependency; orderable-bytes is frozen. Exhaustive matching over Value is left open, with both approaches and their shared prerequisite laid out. It includes current and target dependency diagrams and the steps, and ADR-0002 now points to it. Claude-Session: https://claude.ai/code/session_012zHhtTEo968WazExKxA5oT
The ADR said Postgres refuses to compare an eql_v4 query term with an eql_v3 column at plan time. It does not. EQL's domains are over jsonb, and with no operator defined for the exact pair Postgres resolves `=` as the built-in jsonb = jsonb, so the query returns no rows silently, as a mismatched pair within v3 already does. Verified on Postgres 17 with both bundles from #1141 installed in one database. The decision stands: separate names still reject a wrong-version insert, make the function form ambiguous, and give typed clients distinct types. Consequences now says where the protection stops, and Deferred names the cross-version blocker operators that would close the gap (#1160).
The ADR deferred to a plan whose own banner disclaims being a specification, so an implementer had no authoritative text for the canonical forms, the order-scheme layering or the release order. The amended decisions now live in the ADR body, with exhaustive handling of Value recorded as the one open decision. Review fixes: pin the timestamp microsecond rounding to Postgres's (round half to even, checked on 17.5) and the text fold's step order; let unassigned code points pass through instead of refusing writes; forbid lossy order transforms on _ord domains that answer = with the order term; choose a binding's kind from the declared EQL target, with a kind-to-family table, and take a Go date in the value's own location; copy the exact leaf byte layouts from tags.rs; say why float equality is now acceptable and that text order terms change; keep cllw-ore's Bool impl; reintroduce --eql-version deliberately on install and migration; list the eql_v3_-prefix detectors, installer schema lists and single-bundle release tooling v4 needs; and name the owner of the deferred cross-version blockers.
…ic.=) as a loud path ADR-0002's host-language table claimed the seven non-EQL integer kinds and Bytes go through Stack Encrypt's own targets, but no target names them yet. Say so, and say how Go's narrow integers reach a target today. The Deferred section listed typed clients and the function form as the only things that catch a cross-producer query, while Consequences also names OPERATOR(public.=). List all three.
ADR-0002 separates the two producers of EQL payloads by Postgres type: cipherstash-client's terms stay in eql_v3, Stack Encrypt's move to eql_v4, and both come from one SQL source. This is the first step of #1141: the source stops naming its schemas and payload version literally. The hand-written SQL under src/v3, the version template, the search_path pin script and the uninstaller now write {{prefix}} where they wrote eql_v3, and {{eql_version}} where domain CHECKs compared VALUE->>'v' to '3'. eql-codegen emits the same placeholders: its schema, typname-prefix and version constants are placeholder strings, and its unit tests render output as v3 before asserting. The Rust bindings keep the literal eql_v3_ prefix because they describe v3 payloads. A new `eql-codegen render <3|4> <out> <file>...` substitutes them. It refuses an unknown or unterminated placeholder and any literal eql_v3 or eql_v4, since a literal name would render the same for both versions and let the v4 bundle reach into v3. build.sh renders the ordered surface into build/render/eql_v3/ and runs the symbol-order and installer-completeness gates on the rendered files. The docs tooling reads rendered SQL too. The committed generated surface is regenerated in the next commit, so codegen:parity fails at this one alone. Refs #1141
Output of `cargo run -p eql-codegen` after the previous commit; no hand edits. `mise run codegen:parity` reproduces it. Only v3 is rendered so far, and the installer and uninstaller built from this commit are byte-identical to the ones built before the placeholders. Refs #1141
Gate 1 of verify_installer_complete.sh compared the order against the rendered tree, which build.sh writes from that same order, so a dropped file could no longer fail it. It now lists the source tree and maps the paths into the rendered namespace (a truncated order fails again). The Doxygen filter read through a process substitution, whose exit status is lost, so an unreadable input passed docs_v3_grep.sh with empty output; it is a plain pipe under pipefail now. eql-codegen render refuses an absolute or .. path instead of writing over the source. build/ is anchored so tasks/build/ is not ignored, the codegen templates are build cache sources, a moved doc comment is back on its function, and the schema, JSON and pin-script comments name the rendered version (v3 output is byte-identical).
- `eql-codegen render` refuses an empty file list instead of exiting 0
having written nothing.
- The literal-name guard no longer matches a longer version such as
`eql_v30`, and its error says to write the placeholder, comments
included.
- `list-schemas [3|4]` prints rendered names, and the schema parity gate
renders schema.sql with the build's renderer before comparing.
- The Doxygen filter refuses any `{{` left after its substitution.
- verify_installer_complete.sh takes its rendered root from the order
file's directory by default.
- The render tests clean up their scratch directories on failure.
- pin_search_path_v3.sql's header says it is rendered, not appended
verbatim. This is the only change to the v3 installer, and it is a
comment.
…endering it The filter substituted the placeholders with its own sed, a second renderer without eql-codegen's checks. It now reads a src/ or tasks/ file's rendered copy from build/render/eql_v3/, which every caller already builds, and refuses a file with no rendered copy. Other files, such as the filter's own test fixtures, are read as given. The Doxygen XML is byte-identical (297 files). docs_v3_grep.sh counts the hidden ste_vec_contains aliases in the rendered file too, so it no longer depends on the placeholder's spelling, and now depends on build. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
… traits on v4 only Stack Encrypt's output is not byte-for-byte compatible with cipherstash-client's, which writes the eql_v3 domains, so an eql_v3 struct deriving EncryptFrom or DecryptInto claims a compatibility that does not exist. ADR-0002 gives Stack Encrypt its own name, eql_v4. eql-codegen now writes the scalar domains a second time into eql_bindings::v4: eql_v4_* names, a SchemaVersion pinned to 4, a schema id under eql/v4, and an inventory. encryption_gap refuses every v3 domain, so TextEq and TextEqQuery derive their traits only in v4, and a codegen test asserts no v3 struct derives either. The Stack Encrypt target table moves to v4/targets.rs and names eql_v4 domains; the Go package encrypt/eql is regenerated from it. v4 has no SteVec bindings, payload enums, TypeScript export or JSON Schema yet. No published eql-bindings carried the stack-encrypt feature, so nothing released loses a trait. The encryption test crate's PostgreSQL test now installs the v4 bundle; the Go guest and the stack-encrypt fixtures name the v4 domains. Refs #1142
ADR-0002 installs one SQL source under two names, so a column's domain says which engine wrote its terms. The build now renders the placeholder source a second time as eql_v4 and writes release/cipherstash-encrypt-v4.sql and its uninstaller. The v3 artefacts are unchanged, byte for byte. verify_no_v3_in_v4.sh fails the build if the v4 bundle names eql_v3 or checks for payload version 3: a stray v3 reference would install cleanly and then compare v4 data with v3 terms. The symbol-order check stays on v3 only, since v4 is the same files in the same order with every name renamed. v3_v4_coexistence_tests installs both bundles into one database and pins what the separate names enforce: a wrong-version payload fails the domain CHECK, and the cross-version function form is ambiguous. It also pins the gap the corrected ADR records: a plain `=` across versions falls back to jsonb = jsonb and returns no rows, silently, until cross-version blocker operators exist (#1160). Refs #1141
The guest's eql-feature round-trip test still asserted "v": 3 after Stack Encrypt moved to writing EQL v4, failing the WASI/Go CI job. Also update two comments that named the v3 envelope and the eql_v3.query_<name> operand.
…dles verify_no_v3_in_v4.sh piped grep into head under pipefail. When the matches outgrew the pipe buffer, grep died of SIGPIPE, the condition read false, and a wholesale eql_v3 leak passed while a single stray reference failed. It now captures the matches before printing them, and a new self-test (test:v4_isolation, run in CI) feeds it a 20,000-line leak. With v3 and v4 in one database, each lints() reported 3,840 errors about the other version's blocker operators: encrypted_domain_types took every jsonb domain in public, while blockers are recognised only in the lint's own schemas. The public arm is now limited to the lint's own prefix, and the coexistence test asserts both lints are clean with both installed. The coexistence tests also pin OPERATOR(public.=) between versions (ambiguous) and the eq_term text comparison (runs, refuses nothing), and build.sh and mise run clean remove the v4 artefacts too.
…he version boundary The inventory boxes PhantomData<T> handles, and their DomainType impl did not forward schema_id, so every v4 entry reported an /eql/v3/ $id. A type-level schema_id_base_static() now carries each type's base through the handle, and the generated v4 types override it. That surfaced in a new v4 parity test, the counterpart of catalog_parity.rs: v4::all() is the scalar catalog under eql_v4 names in catalog order, each query twin lives in eql_v4, term keys match the catalog, and each generation's TextEq refuses the other's version, both directly and through the inventory. The encryption suite refuses a real v4 payload whose version alone was set back to 3. types:generate snapshots and restores src/v4 as well as src/v3, types:check diffs it, and the pre-commit hook triggers on it. The v4 module doc no longer links a feature-gated module, and the Go READMEs name the eql_v4 column and query operand the guest now writes.
- The gate accepts a payload-version read only in the form `->> 'v' = '4'`, so IN (...), swapped operands, casts and jsonb comparisons are refused without being listed. A grep fault ends the gate with 2 rather than reading as no match. - The self-test asserts the exact exit status for every case: 0 clean, 1 leak, 2 bad input. A crashed gate no longer passes the rejection cases. - test:installer_complete checks the v4 installer as well as v3. - lints() matches its own public domains with starts_with: in LIKE, the underscores inside the prefix are wildcards.
v3 and v4 each had a hand-written SchemaVersion that differed only in the constant and the schema name. They are now one type, SchemaVersion<const V: u16 = 3>: v3::SchemaVersion and v4::SchemaVersion are SchemaVersion<3> and SchemaVersion<4>, and plain SchemaVersion still means v3. The TypeScript binding is written by hand, since ts-rs's derive has no const generics, and bindings/v3/SchemaVersion.ts is byte-identical. Also: - The changeset says what moved to v4 was never released. - The Go README says how to install EQL v4 before using an eql_v4 column. - The targets test exempts only the SteVec rows from the compiled-type check, and asserts they are not producible. - packages/eql/AGENTS.md: types:check covers src/v4, and v4 shares v3's term types. - A private intra-doc link in encryption::targets no longer fails rustdoc -D warnings with the stack-encrypt feature. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…atalog operator_across_versions_returns_no_rows read the planner's choice from EXPLAIN text, whose deparse can change between Postgres majors. It now stores the comparison as a view and reads the operator it bound: an EQL operator from pg_depend, and the built-in jsonb = jsonb, which is pinned and has no dependency row, from the view's stored query tree. A control checks that a same-version comparison binds the EQL operator, so an empty list is a finding rather than a probe that sees nothing. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
The hook never ran in cipherstash/stack, and would have matched nothing if it had: - `mise run install-hooks` set core.hooksPath to `tasks/githooks`. Git resolves a relative hooksPath against the root of the working tree, which here is the monorepo root, so git looked for `tasks/githooks` there and ran no hook. It now writes the path from the root, through `git rev-parse --show-prefix`. - Git runs a hook from the root of the working tree, so the staged paths carried a `packages/eql/` prefix that the hook's patterns, anchored at `crates/` and `src/`, never matched. The hook now lists only the staged paths under this package, relative to it. - `mise run` ran from the monorepo root, where this package's tasks are not defined. The hook now works from the package root. The prefix is computed from paths rather than with `cd` and `git diff --relative`, because git exports GIT_DIR to a hook, and with GIT_DIR set git takes the current directory as the top of the working tree. Checked with `git hook run pre-commit` from the monorepo root: a staged `src/v3/schema.sql` runs `test:schemas:parity`, and an unrelated staged file runs nothing. Running the hook from the package directory still works. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…text types A Go partner needs a pre-release of EQL v4 with Stack Encrypt before vitaminc 0.6 settles ordering. These are the v4 types that need no ordering term and whose value kind is already in vitaminc 0.5's value enum: Text, Integer, IntegerEq, Bigint, BigintEq, Boolean, Real and Double, beside TextEq. vitaminc 0.5 implements Encrypt for String but not for i32, i64, bool or the floats, and stack-encrypt deliberately has no combinator that re-encodes a plaintext. So eql_bindings::encryption::plaintext adds Int32, Int64, Bool, Float32 and Float64: each seals vitaminc's tagged leaf, the bytes FfiValue seals for the same value, opens only its own tag, and the integers derive the equality term of the integer inside. A value written through these types and one the Go SDK writes as a plain field open each other, and IntegerEq's term is a plain i32's. The derive reasons now have one source, encryption_gap: ordering terms wait for vitaminc-ore, float equality for vitaminc 0.6's canonical float, smallint/date/timestamp/numeric for their value kinds. Json stays unproducible: a JSON value seals as a tree of leaves and EQL's c holds one, and how a document becomes one leaf is a wire decision not made here. The encryption test crate checks each type through both entry points, against the value enum's leaf and the plain integer's term, and in PostgreSQL on an eql_v4 install: equality queries on IntegerEq and BigintEq return the right rows and every stored value opens. Refs #1130, #1133
Text became producible in this PR, so it is now the live storage-only type the no-query rule describes, and TextOrdOre's refusal names vitaminc-ore rather than CLLW. Both unit tests still asserted the old table and failed under --all-features.
Their equality terms came from vitaminc 0.5's per-width PRF domains (i32-le/v1, i64-le/v1). vitaminc main replaces those with one integer-orderable/v1 domain for every width, so shipping them now would either break the promise that IntegerEq matches a Go int32 equality field or leave every stored hm unmatchable after the upgrade. Both leave ENCRYPTION_DOMAINS with a reason that names the domain, the generated dispatch and the Go table mark them unproducible, and the PRF impls on Int32 and Int64 are removed so nothing derives a term under the old domains. The storage-only types, TextEq and the plaintexts stay. v4_types.rs now covers what the review found missing: a leaf the value enum sealed opens as each typed plaintext, every producible row goes through all three by-name entry points (a swapped type or plaintext in the generator fails), Float64 is checked against the value enum, and the Postgres test reads every column back. The width test that pinned the per-width design is gone. The guest's producibility assertions and the eql-bindings unit tests use the final producible set, and the stashgen README lists each producible type with the Go field type it takes.
… bits, check widths - plaintext.rs: Float32 and Float64 compare by bit pattern, so -0.0 is not 0.0 and a NaN equals itself, the equality a round trip preserves. Unit tests pin each leaf's tag, width and little-endian order through the same reader Decrypt uses, and refuse another tag or width. - The generated plaintext doc line shows without the stack-encrypt feature again; only its link to crate::encryption is feature-gated. - stashgen refuses a field of the right kind and the wrong width (int64 into Integer, float64 into Real) itself, naming both widths, as its README says. - enginetest.Static lists the seven types the engine produces, with their widths. - The eql-bindings README no longer says TextEq is the only producible type. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…ch terms TextMatch needs no ordering term, and the engine already derives match terms. Under ADR-0002 the terms only have to be ones EQL's SQL can compare, not cipherstash-client's bytes: both the stored value and the query come from Stack Encrypt, and cipherstash-client never writes an eql_v4 column. EQL's bloom filter is a smallint[] of set bit positions, and @@ tests containment. The new BloomFilter adapter writes the engine's positions under its default match configuration (3-grams, lower-cased, k = 3, a 256-bit filter), so every position fits a smallint, and refuses one that does not rather than wrapping it. It names the match term MatchTerm, the spelling that compiles against both the tree and the crates.io stack-encrypt 0.2.0 the published crate builds against. Proven in PostgreSQL on an eql_v4 install: @@ returns every row for a shared substring, one row for a unique one and none for a miss. Text too short to make a 3-gram is refused rather than stored with a filter no query could match. Refs #1131
…age against crates.io The adapter's turbofish named four type parameters for target::matching, and stack-encrypt 0.2.0 on crates.io, which eql-bindings publishes against, takes three, so cargo publish --dry-run --all-features failed. Naming the Encryption type instead lets inference fill the rest under either version. The match configuration was Stack Encrypt's default (k = 3, 256 bits), taken from another crate and recorded nowhere. A 256-bit filter saturates on a few hundred characters, so @@ returned most of a table, and a change to that default would have left stored filters disagreeing with new queries. eql-bindings now pins its own EqlMatch (3-grams, lower-cased, k = 6, 2048 bits); a unit test pins the values and an encryption test pins the filter positions for one query, so a change has to be deliberate. Also: TextMatchQuery's generated docs say match query terms, the Postgres test pins the reverse operand order (query @@ stored) as the known SQL gap it is, the Go guest lists TextMatch as producible, and the READMEs and changeset say that short text cannot be stored yet. Storing an empty filter for it needs a stack-encrypt release first: 0.2.0 refuses token-less text inside the term derivation, before anything here can see it.
…he test engine TextMatch - The eql-bindings README, the stashgen README and the changeset say that only `stored @@ query` matches until #1171 is fixed, and that a TextMatchQuery for text shorter than three characters is refused too. - The PostgreSQL test's reversed-order assertion names #1171, and the filter test says its negative case is a property of one key source and configuration, not of Bloom filters. - enginetest.Static produces TextMatch; the refusal case for an EQL type the engine cannot produce now names IntegerEq; the users golden case stores Name as TextMatch, so generation of its field methods is exercised. - encryption_gap drops its Bloom branch, which no catalog domain reached once text_match became producible. - The generated TextMatchQuery doc keeps the plaintext line without the stack-encrypt feature. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…eSQL stashgen needed no change: it maps int32, int64, bool, float32, float64 and string to their kinds and accepts any type the engine's se_targets table marks producible, so every type the previous commits made producible already generates typed Encrypt and Query methods. A Profile fixture stores one field in each new type. The hermetic test seals it through generated code over the deterministic EQL guest, checks every value is a v4 envelope with the type's keys, and opens it back. A cross-language fixture, v4_queries.json, holds the IntegerEq, BigintEq and TextMatch query operands the Rust dispatch derives under the shared test index key; the Go fields derive them byte for byte. Checked by hand against PostgreSQL: rows sealed by this Go code, loaded into an eql_v4 install, answer =, = and @@ with exactly the expected ids. The Go module has no PostgreSQL driver, and adding one for a test would change every consumer's dependency graph, so that check is not committed. The encrypt and stashgen READMEs name eql_v4 and list the producible types with the Go field type each takes. Refs #1144
… withdrawn integer equality types Nothing in the repository put a value the generated Go code sealed into an eql_v4 domain or operator; the PostgreSQL check was manual. The Go test now writes v4_rows.json, the Profile rows the generated code seals over the deterministic EQL guest. On every later run it checks that file against today's output, envelopes and match terms byte for byte and ciphertexts by opening them (their AEAD nonce is random). The EQL suite opens those rows with stack-kms's DeterministicSource over the same seed, and its PostgreSQL test, which test:encryption:postgres runs in CI, inserts them into eql_v4_* columns and queries name @@ with the Go query operand from v4_queries.json. With IntegerEq and BigintEq held back, Profile drops Age and stores Balance as Bigint, and v4_queries.json carries TextMatch only. The one stored-versus-query check had used Age, so it is replaced by a TextMatch check that a contained needle's filter bits are a subset of the stored filter and an absent one's are not. Every stored field's i is checked against its table and column. The Go workflow now runs when the fixtures or EQL's lockfile change, and biome leaves the generated rows file alone.
…re stable - TestV4RowsMatchTheCommittedFixture writes the rows it seals to EQL_V4_ROWS_OUT, and the Rust suite opens the file EQL_V4_ROWS names. The Go job runs both, so the ciphertexts the Go SDK seals in that run are opened by native Rust, not only the committed ones. - EQL_UPDATE_FIXTURES rewrites v4_rows.json only when something besides the ciphertexts moved, so a regeneration that changes nothing leaves no diff of fresh nonces. - The Go and Rust TextMatch checks say that their negative cases hold for one seed and configuration, the PostgreSQL expectation folds case as `@@` does, and a needle shorter than three characters is asserted to be refused. - tests-golang.yml runs on any change under packages/eql/tests/encryption. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
The encrypt README and package doc opened with the index= grammar (Ciphertext plus term byte columns, Fields.Email.Equality), while the stashgen README and the EQL v4 alpha setup guide use encrypt_into= and Fields.Email.Query. A reader starting from go doc wrote tags the guide does not describe, for a storage model it does not cover. Both quickstarts now declare encrypt_into fields, build the query with Query, and store each field as one eql_v4 column, with the install step and the query cast. The index= form stays documented and is named as the alternative; the example program still uses it, and the README now says so rather than calling it the eight steps.
…rom Go, and store Go's rows in PostgreSQL The Go side of the previous commit: an encrypt.Date, a time.Time and an encrypt.Decimal field reach Date/DateEq, Timestamp/TimestampEq and Numeric/NumericEq through encrypt_into, as ADR-0002's host-language rules map them. Nothing in the SDK changed for it; this proves it end to end. A Ledger test model holds one field of each type. It is a model of its own, under Profile's table with columns of its own, because Profile's rows are frozen in stored_v4_rows_v1.json. The Go test checks that every field seals with the right keys and opens back (a decimal with its scale, an instant in UTC with its nanoseconds), that Eq normalises a decimal's scale and compares an instant to the microsecond and by instant rather than wall clock, that out-of-range values are refused with ErrEncoding, and that its Eq queries match the Rust fixture byte for byte. TestV4LedgerRowsMatchTheCommittedFixture writes v4_ledger_rows.json. Rust opens those Go-sealed rows into eql-bindings' plaintexts, and a PostgreSQL test stores them in eql_v4_* columns and runs = with the fixture's queries. The Go README, doc.go, the stashgen README and the CHANGELOG list the six types and the Go type each takes.
coderdan
removed this pull request from stack #1282
October 19, 2026 09:41
…ering feat(stack-encrypt): order text over ADR-0002's folded order key
feat(stack-encrypt)!: derive every order term over its ADR-0002 order key, and order floats
feat(eql-bindings): produce the OPE-ordered EQL v4 types
…numeric feat(eql-bindings): produce the date, timestamp and numeric EQL v4 types
…ains # Conflicts: # packages/eql/crates/eql-codegen/src/go_eql.rs # packages/eql/mise.toml # packages/eql/tests/encryption/Cargo.toml
The range domains' CHECK compared `VALUE->>'v' = '4'`. For `"v": null` that comparison is SQL NULL, the whole CHECK is NULL, and PostgreSQL treats a NULL CHECK as passing, so a range with no v4 version cast cleanly and `eql_v4.between` ran on it. The check is now `COALESCE(VALUE->>'v' = '4', VALUE IS NULL)`: false for a JSON null version, still true for a NULL range (which matches no row). The comparison keeps the one form the v4 isolation gate accepts. `postgres_range_domains_check_their_shape` now casts a `"v": null` range to both the ORE and the OPE domain and expects 23514. It fails without the template change. The same `->>'v' = '<n>'` predicate is in every other domain CHECK, most of them shared with v3; that is left to a separate change so the v3 bundle stays byte-identical here.
… handles 64 bits Review follow-up on the step loop (#1064). A Go caller past the guest's MAX_OPEN of 64 paused operations had its first step refused with STATUS_STATE, which surfaced as ErrState, the error that means the client is closed. Before the step loop such a caller queued on the client's lock. The client now holds a semaphore of the same size: an operation that can reach ZeroKMS takes a slot before its first step and gives it back when it ends, so callers past the cap wait (and leave the queue when their context ends) instead of failing. A test pins maxPaused to the guest's MAX_OPEN. Handles were u32, issued once each and never reused, so a long-lived instance spent them: every Seal and Open pauses at least once, and after 2^32 pauses every ZeroKMS-reaching call failed with ErrState. They are u64 now, through se_resume, se_abort and the step frame. Also: responses for a resume step that never ran (the client closed while the request was in flight) are wiped rather than left to the collector; takeIO keeps the allocator's Enter/Exit bracket like every other guest entry; and the docs and CHANGELOG say the token source is asked once per call that can reach ZeroKMS, including calls the guest answers from cache.
feat(eql): give EQL v4 range query domains and eql_v4.between
feat(golang)!: the guest steps instead of blocking on the network, and the client drives it outside its lock
… to queries on its plan A typed plan answers an operator through its target type's `op::Answers` impl, but the chain required `Typed<T>: ValueShape<S>`: a label that converts into `T`'s context. An EQL v4 type is sealed under its column (`NonEmpty<Identifier>`), so no plan for one could be queried, and eql-bindings had nowhere to implement `Answers` (#1238). `op::LabelledTarget` is the opt-in: a target type says its stored terms were derived under the context a plan's label names. For such a type, `Chooses` asks only `ValueLayout<S>` (the indexes `T` declares), since a query derives through the index and never runs `T`'s encryption. Every other type keeps needing `ValueShape`, so a type sealed under, say, a tenant id cannot be queried under a label that would match nothing. This crate's term targets implement the trait, which keeps every query that compiled before compiling. `Chooses` now resolves the plan itself (`choose`) rather than through `Resolves`, which keeps `ValueShape`: a named index (`.via(..)`) on a plan whose type only opts in would send a term the type does not send. An EQL query value stores the column it queries, so its index needs the context its terms were derived under, and must refuse one it cannot record. `QueryToken`, which only the chain makes, now carries the label and whether the call extended it (`label`, `is_extended`): with `.extend(..)` the terms are derived under the label and the extension, so a term that records the label alone would match nothing. The defaulted `Supports::check_label` refuses such a context, and the chain runs it in `check`, where the label is resolved, before any keyset is loaded or term derived (and again in `prepare`, as for `check`). `Supports::query` stays infallible: the chain calls it only after `check_label` accepted the token's context. Every index here takes any context. Tests: a labelled target is queried under the plan's or the call's label, and its term reads whether the call extended it; a label its index refuses loads no keyset; and a UI case pins that a type with an `Answers` impl but no `LabelledTarget` answers no query on its plan.
…query chain Rust could store an EQL v4 type but query it only through a data plan: no v4 type implemented `op::Answers`, so `cipher.query(op::Eq, v).on(&plan)` did not compile on a plan for `TextEq`, while the Go SDK already had an `Eq` method per equality type and `Matches` with `Verify` on `TextMatch` (#1216). This gives Rust the same operators, with the same bytes (#1238). eql-codegen writes into `v4/answers.rs`, for each producible type with a query twin: `op::LabelledTarget` (it is sealed under the column a two-segment label names), the twin's assembly, and one `op::Answers` impl per operator. The operators come from the type's index keys, the input the data path's `TargetDescriptor::operators` reads: `"eq"` answers `Eq`, `"match"` answers `Matches`. It stops on any other index, or more than one, so making an ordered type producible has to extend the rule. The same rows give `ANSWERS`, which the `typed_queries` test holds to `TargetDescriptor::operators()` for every type. The file sits beside the target table rather than in it: `v4/targets.rs` is built on Stack Encrypt's `dynamic` layer and needs the `targets` feature, while a typed query needs neither, so `answers.rs` needs only `stack-encrypt`. The index each impl names is `TwinIndex<I, Q>`: Stack Encrypt's own index `I` (the same spec and derivation), whose query term is written into the twin `Q` with the column the query's label names, by the encodings the stored values use. It answers only for the twin's own plaintext (`QueryTwin::Plaintext`), so no plan can send one type's twin for another type's value, and its `check_label` refuses a label that names no column and any extension before any keyset is loaded: an extended query would be derived under more than the column the twin stores, and match nothing. Both refusals are a `QueryLabelError` in `Error::Other`, so a caller matches the variant rather than the message. `TextMatch` returns `MatchTwin<Q>`, which serializes as the twin and keeps the query's tokens for `verify` and `target_limit` under `EqlMatch`. `Hmac256` and `BloomFilter` now declare their indexes, so a type holding one declares the index a query is checked against, and `EqlMatch` implements `MatchCase`. Tests: every typed query is the data path's `query_term` bytes and domain (the float values the semantic baseline folds included), with the data path's inputs built as frames of Rust values; `TextMatch`'s `verify` and `target_limit` agree with the data path's; short text, a non-column label and an extended query are refused before any keyset is loaded; and compile-fail doctests for `op::Lt` on `TextEq`, `op::Matches` on `IntegerEq`, `.via(Equality)` and another type's `TwinIndex` on a `TextEq` plan, and `verify` on an exact term. The new module's example runs as a test in the encryption crate.
…nd pin the twin's plaintext tie `op::Answers` and the stack-encrypt CHANGELOG read as if only a type sealed under a context no label names needed `op::LabelledTarget`. Any type queried on a typed plan needs both, whatever its context; both now say so. The cross-type `TwinIndex` compile-fail doctest is rejected by `Resolves`' `ValueShape` bound before the plaintext tie is reached, so it would still pass if `QueryTwin::Plaintext` were removed. A doctest pair now checks the tie alone, through a `Supports<op::Eq, String>` bound: `TwinIndex<Equality, TextEqQuery>` meets it, and `TwinIndex<Equality, IntegerEqQuery>` fails with `<IntegerEqQuery as QueryTwin<Hmac256>>::Plaintext == String`.
feat(eql-bindings): query EQL v4 types through Stack Encrypt's typed query chain
…perator reads
eql_v4.query_text_search required hm, op and bf on every query value. bf
is the bloom filter built from 3-grams, so text shorter than one 3-gram has
none, and `col = 'Al'` or `col < 'Al'` could not be written against an
eql_v4_text_search column. That is why Stack Encrypt could not produce the
type.
In v4 the search query domains (query_text_search and its _ore twin) now
require at least one of their terms. Each extractor over them reads its
term through eql_v4_internal.require_query_term, which raises
invalid_parameter_value naming the domain, the term and the operators that
compare it when the term is absent, so `=` on a value with no hm, `<` with
no op and `@@` with no bf are refused rather than reading NULL and
matching nothing. The extractors stay LANGUAGE sql, so the wrappers still
inline and the eq_term/ord_term/match_term indexes still engage.
v3 is cipherstash-client's format and does not change. The SQL source is
rendered once per version and had no way to say "only in v4", so the
renderer gains version blocks: lines between `-- {{#v3}}`/`-- {{/v3}}`
or `-- {{#v4}}`/`-- {{/v4}}` render for that version only, and the
directive lines for neither. The generator emits them for the two search
query domains, and the helper sits in a v4 block of
src/v3/scalars/functions.sql. release/cipherstash-encrypt.sql and its
uninstaller are byte-identical to the build before this change.
ADR-0002 gains a dated amendment recording the decision.
…s operator TargetResolver::query took a type name, a label and a value, so a resolver had to build the type's whole query twin, every term of it, for every operator. EQL v4's TextSearch needs to emit only the term its operator reads: a match term cannot be made from text shorter than one 3-gram, and `=` or `<` on such text must not be refused for want of one. query_term already knows the answering index (OpSpec::answered_by over the type's indexes). It now passes it through record::query to TargetResolver::query as an &IndexSpec after the type's name. A resolver whose twins carry every term ignores it. eql-bindings' encryption::targets::query takes it the same way and refuses an index the type does not carry (TargetError::NoIndex) before deriving anything. The Go guest forwards it; an index the type lacks can only come from an engine bug, so the guest reports it as internal. BREAKING CHANGE: TargetResolver::query and dynamic::record::query take an &IndexSpec; eql_bindings::encryption::targets::query takes one after the name.
…perator's term TextSearch was unproducible because its query twin carried hm, op and bf on every query, and text shorter than one 3-gram has no bf: `=` or `<` on a two-letter name would have been refused. EQL v4's search query domain now takes any one term, so the twin can carry just the one its operator reads. In v4, TextSearchQuery (and the unproducible TextSearchOreQuery) hold each term as an Option, skipped in the JSON when None. The generator emits one projection per term, TextSearchEqualityQuery, TextSearchOrderQuery and TextSearchMatchQuery, each deriving EncryptFrom<String> over the envelope and its one term and converting into TextSearchQuery. The by-name query dispatch, which now receives the answering index, runs the projection for that index, so a match term is derived only for `matches`. The twin itself carries no derive. The stored TextSearch carries all three terms, the same bytes TextEq, TextOrd and TextMatch store, so text too short for a 3-gram is refused on write with EmptyTermText, exactly as TextMatch refuses it: a stored row with an empty filter would be one no match query finds. TextSearch joins ENCRYPTION_DOMAINS, the regenerated target table marks it producible, and the Go encrypt/eql table follows. The engine answers eq, lt, lte, gt, gte and matches on it. Tests: each index's query carries its term alone and equal to the stored one; "Al" is queried by eq and ope and refused for match; the typed projections serialize to what the dispatch writes; and, against an EQL v4 database, `=`, `<>`, `<`, `>=`, `@@` and the function forms find the plaintexts' rows, a one-term operand is a query_text_search and a termless one is not, and each operator and extractor refuses an operand missing its term with invalid_parameter_value naming the term and never the operand.
…ng one term A directory Entry model holds one TextSearch field. Generated over the deterministic EQL guest, its stored value carries bf, c, hm, i, op and v and opens back; Eq, Lt and Gte return a query with only hm or op, the stored term, for "Alice" and for the two-letter "Al"; Matches returns one with only bf, which Verify checks; and Matches on "Al", like storing "Qz" or "", is refused with ErrTerm without quoting the value. stashgen gets a golden case for the same field over its static engine, which now lists TextSearch with its three indexes, so the generated methods (Eq, Lt, Lte, Gt, Gte, Matches) are pinned and compile against the stub SDK.
…index `IndexSpec::OreBlock` (key `ore_block`) derives ore-rs 0.9.0's 6-bit block ORE under its own key domain, over the order keys the CLLW indexes use: `ore_rs::vitaminc::Bit6BlockOre` for targets of at most ten bytes (every integer to 64 bits, both floats) and `ChainedBlockOre` for text, bytes and the 128-bit integers. The legacy 8-bit scheme is not used. `sem::OreBlockTerm` is the whole ciphertext. It carries a random nonce, so it compares by the scheme's comparator, never by its bytes, and `from_bytes` checks a stored term's structure. A query is the full ciphertext too: EQL v4's comparator reads the right half of the operand a btree index passes second. Text whose fold is longer than 1023 bytes or ends in NUL is refused, on the query path before any key is touched. The CLLW `Ore` and `Ope` indexes are unchanged, byte for byte. BREAKING CHANGE: `IndexSpec` is exhaustive and gains `OreBlock`, so a `match` over it needs an arm for it.
…ional-terms feat(eql)!: EQL v4 text search queries carry only the term their operator reads; produce TextSearch
EQL v4's `ob` is one ore-rs ciphertext: the fixed-width 6-bit scheme
(header `02 02`) or the chained one (`02 03`, then the padding policy).
v3's is cipherstash-client's 8-bit blocks, which need another comparator,
so the v4 SQL holds its own in a version block:
- `{{ore_block}}_hash_bit`: ore-rs's sigma-MMO hash, the GF(2^128)
doubling on two bigint halves with no branch on the data, then AES under
the public key `ORE-rs.v2.H-pi.1` through pgcrypto.
- `{{ore_block}}_scheme`: the shape checks ore-rs's comparators make
(header, block count, exact length, padding policy and the block counts
it can produce, every symbol below 64). A malformed term raises, naming
its length and never its bytes.
- `compare_{{ore_block}}_term`: raises on terms of different schemes,
fixed-width terms of different block counts and chained terms padded
under different policies. For terms of one shape it runs a constant
operation count: one statement per block evaluates both comparisons and
latches the first difference arithmetically, the hash runs once on every
path, and the result is selected without a branch. Chained terms compare
lexicographically, the shorter first when its blocks all match.
- `compare_{{ore_block}}_terms` over an array requires exactly one term.
The v4 bundle does not reuse the 8-bit format's name. The renderer gains a
placeholder, `{{ore_block}}`, rendered `ore_block_256` in v3 and
`ore_block_v2` (ore-rs's wire version 2) in v4, and refuses a literal
`ore_block_256` outside the source directory's path. The hand-written SQL,
the pin script, the generated extractors and the ORE fallback take it.
`release/cipherstash-encrypt.sql` and its uninstaller are byte-identical
before and after.
`tests/timing/ore_block_v2/` is a hand-run timing harness for the v4
comparator, with a detached ciphertext generator. ADR-0002 is amended.
An EQL type whose `ob` term is block ORE lists the index `ore_block` (`record.OreBlock`) in `se_targets`, and the engine answers its orderings through it. The checker accepts it in a type's indexes and as an operator's index, and stashgen maps it to `IndexOreBlock`. It is not a tag word: a field reaches it only by naming such a type with `encrypt_into=`, so it has no entry in `indexNames` or `outputOfIndex`, and a plan's own outputs still refuse it. The static test engine produces `TextOrdOre`, `TextSearchOre` and `IntegerOrdOre`, so the refusal test names `BigintOrdOre` as the type it does not produce.
`IntegerOrdOre`, `SmallintOrdOre`, `BigintOrdOre`, `RealOrdOre`, `DoubleOrdOre`, `TextOrdOre` and `TextSearchOre` are producible. Their `ob` term is Stack Encrypt's block ORE term, one hex element holding the whole ore-rs ciphertext: `OreBlock256` implements `EncryptFrom` for any orderable plaintext and never decrypts. `TextSearchOre` follows `TextSearch`: its stored value takes the empty filter, and its query carries only the term its operator reads. In the target table an `ob` term is the index `ore_block`, not `ore`, which is CLLW ORE and has no EQL domain, and the block-ORE reason in `encryption_gap` is gone. The unproducible types are the date, timestamp and numeric families, which have no Rust plaintext type yet, and the two JSON types. `tests/encryption/tests/v4_ore_block.rs` compares real ore-rs ciphertexts in PostgreSQL, both ways round, against the plaintext order and ore-rs's own comparator: equal values, a first difference in every block, shared prefixes and text in three padding buckets. It checks the hash bit against ore.rs's vectors, that malformed and mismatched terms raise, and that a btree index answers `<`, `>=`, `=` and `ORDER BY`. `test:encryption:postgres` runs it. The Go catalog, the guest's tests and the Go operator test follow. ADR-0009 is amended. The EQL v4 range tests (tests/encryption/tests/v4_range.rs) now derive their order terms from the engine. They arrived from the range domains with cipherstash-client's ORE and OPE indexers writing the `ob` and `op` of every row and bound, which was the only source of block ORE terms while EQL's ordered types were not producible. The comparator this series adds refuses that 8-bit layout, and this commit is where the `OrdOre` types become producible, so the rows are real `IntegerOrdOre`, `TextOrdOpe`, ... values and each bound is the term the single-bound query of that type carries. The smoke test ranges over the families whose ordered types are producible and records that the date, timestamp and numeric ones are not.
feat(eql)!: produce the block ORE EQL v4 types through ore-rs 6-bit block ORE
coderdan
marked this pull request as ready for review
October 19, 2026 13:41
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a05affaaa5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+55
to
+56
| // Rows is the rows the query scans, such as greatest(reltuples, 0) | ||
| // from pg_class. |
There was a problem hiding this comment.
For a table that has never been vacuumed or analyzed, PostgreSQL reports reltuples = -1; the recommended greatest(reltuples, 0) therefore supplies Rows: 0. For match indexes with a declared prior, TargetLimit then budgets no false positives and returns only want, so false positives can consume the fetch and verification can silently return fewer matches even when more exist. Preserve the unknown state by selecting only when reltuples >= 0 and page without Stats otherwise.
Useful? React with 👍 / 👎.
| if l.bounded { | ||
| return l.rows | ||
| } | ||
| return page |
There was a problem hiding this comment.
When an unbounded result is given a page of at least 1<<63, this returns a high-bit uint64; database/sql rejects such parameters, and PostgreSQL's LIMIT is signed 64-bit. Bounded limits are already capped by the engine, so this fallback should likewise clamp the page to math.MaxInt64 rather than turning an otherwise valid query into a parameter-conversion error.
Useful? React with 👍 / 👎.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the integration branch for a release candidate of EQL v4 and of Stack Encrypt's Rust and Go support. EQL is the SQL that lets PostgreSQL store and search encrypted values; EQL v4 is a second copy of it for values written by Stack Encrypt, the Rust encryption engine behind the Go SDK. The open pull requests for both are merged here first, so they can be tested together before any of them reaches
main.The release version is not decided yet.
Why a branch, not main
Some of this work depends on vitaminc and ore.rs code that is not yet on crates.io, so it pins those crates to a git commit.
stack-encryptandeql-bindingsare published crates, andcargo publishdrops those git pins, so nothing on this branch can be released until vitaminc 0.6 is published (cipherstash/vitaminc#389, which waits on cipherstash/vitaminc#463). Once it is, the git pins become crates.io versions and this branch merges tomainas a normal release.What lands here, in order
feat/aead-remove-legacy-traitsto one commit on vitamincmain, and takes vitaminc's breaking changes since then.Rules for this branch
rev, neverbranch. A branch is deleted when its stack merges.packages/eqlandpackages/stack-encrypt/fuzz. Two commits mean two copies of vitaminc's traits, and the code stops compiling.Changes
Verification
nextwas created frommainat 2a70a47. Nothing has been built yet, because nothing has changed.Related