Repository navigation
refactor(stack-encrypt,eql-bindings)!: remove the plan chain, the builder and the old traits - #1666
Conversation
🦋 Changeset detectedLatest commit: 24def60 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true |
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 9 | 0 | 23 | 0 |
Every mutant in the changed lines was caught by a test.
coderdan
left a comment
There was a problem hiding this comment.
Automated review by a Claude Opus subagent, requested by the author.
The library change is sound. I found no correctness or security defect in the code that remains: the new check_indexes refusals run before any key is requested, the owned refusal path wipes its plaintext, tests pin the contexts that the three new conversions give, and the JSON openings now wipe through JsonPlaintext on every path I read. The findings are a test gap on the new wipe branch, public API that lost its only consumer, and stale guidance (rustdoc, conformance contract, pending changesets) that still describes the removed API.
Findings:
- F1 minor: the wipe on the new refusal branch of
Encrypted::owned_encryptionhas no test.packages/stack-encrypt/src/target/index.rs:724(inline) - F2 minor:
op::DefaultIndexandIndexes::select/Select/At/Wholelost their only consumer, and their docs still describe one.packages/stack-encrypt/src/op/mod.rs:1221(inline) - F3 nit:
indexed((Ore, Ope))still produces theEncrypted<(OreTerm, OpeTerm)>that the target now refuses.packages/stack-encrypt/src/target/index.rs:191(inline) - F4 minor: public rustdoc and error text in the remaining source still describe "the query chain". Several files (below)
- F5 minor: the lifecycle conformance contract still claims coverage of the removed structural decrypt path.
conformance/stack-encrypt/lifecycle-contract.md:15(below) - F6 minor: other stale guidance outside
src/. Several files (below) - F7 minor: six pending
@cipherstash/eqlchangesets still tell users to use the API this PR removes, and they ship in the same release..changeset/eql-remove-structural-traits.md:5(inline) - F8 nit: the test comment miscounts the kinds of index.
packages/stack-encrypt/tests/combinators.rs:744(inline) - F9 nit: the CHANGELOG entry says "Three more records derive" and lists two.
packages/stack-encrypt/CHANGELOG.md:12(inline)
F4 (minor): public rustdoc and error text still describe "the query chain". Verified by reading each line at the head commit. The chain is gone. Typed queries now go through a field marker, and the data path goes through dynamic::query_term. But these docs still say a chain does the work, and two name its removed .extend(..). A reader of the published rustdoc for op::Supports, op::QueryToken, sem::MatchQuery or sem::RangeQuery is told about an API that does not exist.
packages/stack-encrypt/src/op/mod.rs:459(Arity: "the chain derives a term per plaintext"),:512(Input::derive: "Only a query chain calls this"),:779(Supports::prepare: "A query chain calls this once"),:783(check_label: "(.extend(..))"),:800and:807(Supports::query),:817(QueryToken: "Proof thatSupports::queryis called by a query chain"),:1251(test comment).packages/stack-encrypt/src/sem/mod.rs:285,:868,:976(MatchQuery"comes from a query chain"),:1089(RangeQuery"comes only from a query chain"),:1124,:1146,:1251,:2271.packages/stack-encrypt/src/sem/tokenize.rs:32;packages/stack-encrypt/src/dynamic/query.rs:1978;packages/stack-encrypt/src/dynamic/record.rs:2053("Someis refused as the chain refuses it"),:3347(.fields()chain),:5322.packages/stack-encrypt/src/plan/error.rs:150and:153: theMixedCiphersmessage and help still say "chains" ("Start every chain in one batch on the same cipher."). Review note 3 keeps the variant. The note does not say that its user-facing text names the removed API.
Fix: replace "query chain" with "a field marker's query" or "the typed query", and replace .extend(..) with the marker's .scope(..). Reword the MixedCiphers message, or say in note 3 that it is left as is.
F5 (minor): the lifecycle contract claims coverage that was deleted. Verified. conformance/stack-encrypt/lifecycle-contract.md:15 (the lifecycle-partial-output-v1 row) still says "The same holds for a structural Vec<T> that decrypts through DecryptInto, and for a column field of a structural record." This PR deletes DecryptInto and the test that covered it (collections::a_late_element_failure_wipes_the_recovered_prefix_on_the_structural_path_too), and leaves this file unchanged. The contract now lists a guarantee that no test runs. Fix: delete that sentence, or restate it as the owned path that a_late_element_failure_wipes_the_plaintext_recovered_before_it covers.
F6 (minor): other stale guidance outside src/. Verified with git grep. Each of these describes the removed API as current:
packages/stack-encrypt/CONTEXT.md:24("a hand-writtenEncryptFromtype") and:71(a target "through its ownEncryptFrom"; "A plan field is either plain verbs ... or one target"). This is the crate's domain glossary.packages/stack-encrypt/Cargo.toml:24: the comment says the derive dependency provides "#[derive(EncryptFrom)]/#[derive(DecryptInto)]".packages/eql/AGENTS.md:97: "The same derives also implement the olderEncryptFromandDecryptInto."packages/eql/crates/eql-codegen/src/targets.rs:17: "an arm runs<T as EncryptFrom<S>>::encryption()". The PR edits this file's next paragraph but not this line.docs/target-directed-encryption.md:3says "The snippets below are the API as it ships". They are allEncryptFrom. Add a status line saying the traits were later replaced, or mark it historical.- Go:
languages/golang/encrypt/context_field_test.go:153("the typed chain and the data-plan lowering each sealed a Note"), andlanguages/golang/encrypt/query_refusal_conformance_test.go:124and:128(failure messages "the typed chain ..."). The first is tied to the fixture key that review note 7 keeps. The two failure messages are not, and can say "the typed query" now.
Fix: update each line. None of them needs a fixture change.
Checked and found clean:
check_indexesand both call sites inEncrypted<Terms>: the borrowed and owned forms both refuse before any key request. The owned form zeroizes the moved plaintext. The derivedplaintext = Trecord checks inencryption(), and its defaultowned_encryptionguards the value withProtected, so its refusal is wiped too. Testscombinators::a_target_that_names_an_index_twice_is_refused_before_any_key_requestandderive_plan::two_outputs_*assert zero key requests.- The three new
Fromimpls insrc/target/input.rs:derive::a_plaintext_record_field_may_be_a_record_with_declared_contextspins the inner field's term under the extended context, and the round trip. - JSON:
JsonExtracted,JsonSubtree,JsonDocument,JsonTreeDocumentandJsonTreeExtractedkeep their refusals before any key (selector check, stored-column check).opened_frameanddecrypt_documentnow wipe by drop. I found no path where aserde_json::Valueplaintext escapes unwiped. eql-bindingsopen_target/decrypt_document: the expected-column check still runs before the opener is chosen.- Derive: the
DecryptFieldimpl now goes throughDecryptSource, with aZeroizebound. The removed bridges leave no dangling generics. - Spot-checked deleted-test claims:
every_operation_honours_the_keyset_it_runs_through(keyset id on every leaf, andForeignKeysetrefusal),encrypt_and_check_source_refuse_it_alike_with_no_key_request,decrypt_and_check_record_refuse_it_before_any_key_is_retrieved,an_unscoped_pending_merged_with_a_scoped_one_takes_the_scope, and the derive_plan refusal tests. Each exists and asserts what the table says. - Upgrade table in
packages/stack-encrypt/CHANGELOG.md: every replacement named exists with that spelling (encrypt_into_with_context,decrypt_from_with_context,KeysetCipher::encrypt_treewith&Label: IntoContext,dynamic::record::plan,dynamic::Planned::new,Pending::zip/Pending::all,StackCipher::keyset, the marker methods). - Repo rules: no new
ignore/no_rundoc blocks; no#NNNNin code comments, doc comments or READMEs (issue numbers appear only in CHANGELOGs, plans and the ADR); both crates have an Unreleased "Breaking" entry; noskills/*/SKILL.mdmentions the removed API. - The
plan_buildfuzz target, its task, its corpus and bothfuzz.ymlmatrix rows are gone. Nothing else references it. - No code still uses
EncryptFrom,DecryptInto,encrypt_as,decrypt_as,Plan::,.using(,cipher.query(,cipher.open(,plan::all,Layout,JoinContext,ValuePlanorFieldsBuilder. Only the comments and docs listed above remain. - Not run: I did not run cargo. Every finding above comes from reading the code and from
git grep.
| { | ||
| if let Err(error) = check_indexes(&Terms::INDEXES.specs()) { | ||
| // The refused value is the description's to wipe. | ||
| return Encryption::from_build(Vec::new(), move |mut source: S, cipher, _| { |
There was a problem hiding this comment.
F1 (minor): this wipe has no test. The branch is new. It is the only thing that wipes an owned plaintext when the Encrypted<Terms> target is refused. combinators::a_target_that_names_an_index_twice_is_refused_before_any_key_request drives this path with a String. That test checks the error and the key-request count. It does not check that the plaintext was wiped. No test pairs DuplicateIndex or IndexOverlap with a wipe check (git grep over tests/ finds no witness or heap scan near either). So if someone deletes line 725, every test still passes, and a refused String is dropped without being wiped.
Verified: I read the code path and searched the tests. I did not run mutants.
Fix: add a test that encrypts a plaintext whose Zeroize sets a flag (a newtype over String that implements Encrypt by delegating, in the style of collections.rs's Tracked) into Encrypted<(EqualityTerm, EqualityTerm)>, and assert the flag is set. Or add the case to heap_hygiene.rs.
There was a problem hiding this comment.
Fixed in ebf5a61. owned_records::a_plaintext_is_not_leaked_when_its_indexes_are_refused encrypts a plaintext that records whether it was dropped with its value still in it. It covers Encrypted<(EqualityTerm, EqualityTerm)> alone and a plaintext = T record with two equality fields, and it asserts the error and that no key was requested. I removed the zeroize call to check: the test fails with "the refused target dropped the plaintext unwiped".
| /// the field's type is not known where the query is written, so its | ||
| /// [`Answers`] impl is not consulted, and a field whose type answers | ||
| /// through another index needs `via`. | ||
| /// The index an operator is answered with where a field declares its |
There was a problem hiding this comment.
F2 (minor): DefaultIndex has no consumer now, and this doc says it has one. At the base commit, its only consumer was plan/chain.rs (impl<Op: DefaultIndex<In>, ..> Chooses<Op, In, Whole> for FieldPlan). git grep at the head commit finds no DefaultIndex bound or projection anywhere: only the trait, its impls here and in json/query.rs, and docs. The dynamic path reads dynamic::Op::answered_by and never this trait. But this line says it is "the table a plan that is data reads". dynamic/query.rs:11 says answered_by "is the table the typed path's DefaultIndex impls state". No test now holds the two tables to each other.
The same holds for Indexes::select, Select, At and Whole in target/index.rs. The chain was their only caller (chain.rs:850, self.indexes().select::<I, At>()). Only doctests and the select_duplicate_index UI case use them now. Their docs (index.rs:47-49, and the select method doc "for the query side, where a query asks one index for one term") still describe a query path that uses them.
Review note 1 lists the JSON operators' unreachable derive. It does not list these.
Verified with git grep at both commits.
Fix: remove DefaultIndex and select / Select / At / Whole, since this is already a breaking release. Or keep them, add them to note 1 and the CHANGELOG, and correct the two docs that claim a consumer.
There was a problem hiding this comment.
Fixed in ebf5a61. op::DefaultIndex and its impls are removed, and the CHANGELOG's removal entry lists it. The module docs of dynamic::query no longer claim answered_by mirrors it. I kept Indexes::select with Select, At and Whole: it is a tested public accessor (tests/index.rs, ui select_duplicate_index) that picks one index out of a set. Its docs no longer describe it as the query path, and review note 1 in the PR body now names it as API the library itself does not call.
| /// the orderings, so a query for one would have to choose between them. | ||
| /// The dynamic path refuses the same declarations when it parses a plan. | ||
| /// | ||
| /// A typed declaration calls this when it is described, so the refusal |
There was a problem hiding this comment.
F3 (nit): the bare combinator still makes what the target refuses. indexed (line 844) and indexed_beside do not call check_indexes. So keyset.run(indexed::<u32, _, Borrowed, _>((Ore, Ope)), &v, ctx) returns an Encrypted<(OreTerm<u32>, OpeTerm<u32>)>. Encrypting into that same type as a target is now refused with IndexOverlap. This paragraph says "A typed declaration calls this when it is described", and a reader can take indexed(..) to be one.
Verified by reading indexed and indexed_beside.
Fix: the simplest fix is to say here, and on indexed, that the bare combinator does not check. To refuse there too, note that indexed is generic over the source mode. In Owned mode a plain Encryption::failed drops the plaintext without wiping it, so a check in indexed would need the wiping branch you wrote at line 724 (and an S: Zeroize bound). Do not move the check into indexed_beside with Encryption::failed, because that would lose the wipe on the owned path.
There was a problem hiding this comment.
Fixed in ebf5a61, in the docs. check_indexes now says a target calls it and that indexed does not, and indexed says it does not check its set. I did not move the check into indexed_beside, for the reason you give: in Owned mode it would need the wiping branch and an S: Zeroize bound.
| '@cipherstash/eql': minor | ||
| --- | ||
|
|
||
| `eql-bindings`, the Rust crate released with this package: the EQL v4 types no longer implement Stack Encrypt's older traits `EncryptFrom` and `DecryptInto`. Stack Encrypt removed them, with `encrypt_as`, `decrypt_as`, the plan builder and the query chain. |
There was a problem hiding this comment.
F7 (minor): six pending changesets still advertise the API this one removes, and they ship in the same release. packages/eql/packages/eql/CHANGELOG.md tops out at 3.0.6 and mentions none of them, so all of these are unreleased. They will appear in the same @cipherstash/eql CHANGELOG section as this entry:
.changeset/eql-v4-owned-targets.md:9-10: "The same derives still implement the olderEncryptFromandDecryptInto. Soencrypt_as, plans ... work as before" and "To build a plan for a type, usePlan::value::<P>().encrypt_into::<T>().".changeset/eql-v4-bindings.md:5: "A Rust caller queries those types through Stack Encrypt's typed query chain:cipher.query(op::Eq, v).on(&plan)", and "throughencrypt_as"..changeset/eql-json-tree-walk-and-projection.md:15: "stack-encryptgainstarget::LayoutandPlan::value().encrypt_with_layout(layout)"..changeset/eql-plan-field-targets.md:5,.changeset/eql-json-tree-domain.md:5,.changeset/eql-bindings-optional-storage-only.md:5: each namesEncryptFrom/DecryptIntoas what the types implement.
A reader of the release notes is told to use Plan::value and cipher.query in the same section that says they are gone.
Verified with git grep over .changeset/ and the EQL CHANGELOG.
Fix: since these are unreleased, edit them to name the owned API, or drop the sentences about the removed API.
There was a problem hiding this comment.
Fixed in ebf5a61. The six pending changesets name the owned API now: eql-v4-owned-targets, eql-v4-bindings, eql-json-tree-walk-and-projection, eql-plan-field-targets, eql-json-tree-domain and eql-bindings-optional-storage-only. The unreleased entries of eql-bindings/CHANGELOG.md had the same problem and are corrected too. Two of its sentences said OpeCllw and OreBlock256 implement an encryption trait; neither does any more, so I deleted them.
| /// A target holds one term for each kind of index, so a target that names | ||
| /// an index twice is refused when it runs, before any key is requested: | ||
| /// alone, or as a field of a derived record. Five indexes over text always | ||
| /// repeat one. There are four kinds of index, so the fifth is a second |
There was a problem hiding this comment.
F8 (nit): there are five kinds of index over text, not four. OreBlock is Index<S> for every S: Orderable (index.rs:436), the same bound as Ore and Ope, and String meets it (this test uses OreTerm<String>). So (Equality, Match, Ore, Ope, OreBlock) repeats no index. It is refused for IndexOverlap, not DuplicateIndex. "Five indexes over text always repeat one" is therefore false. The same goes for FiveTerms' doc at line 730, "One term of each kind over text": it leaves out OreBlockTerm.
Verified by reading the bounds.
Fix: say "five kinds, and three of them answer the orderings, so no valid set has more than three", or simply that the set repeats the match index.
There was a problem hiding this comment.
Fixed in ebf5a61. The comment on FiveTerms now lists what it holds (equality, two match terms, ORE and OPE), and the test's doc says only that two match indexes under different options are the match index twice.
|
|
||
| ### Added | ||
|
|
||
| - **Three more records derive** (#1661). The owned derives did not accept |
There was a problem hiding this comment.
F9 (nit): "Three more records derive" lists two. The entry has two bullets (a struct record as a field of a plaintext record, and a ciphertext or term under context_type = NonEmpty<T>). The PR body's "Added" section has a third bullet, the duplicate-index refusal, which this CHANGELOG puts under "Breaking" instead.
Verified.
Fix: write "Two more records derive".
There was a problem hiding this comment.
Fixed in ebf5a61: "Two more records derive". The same pass corrected other unreleased entries of this CHANGELOG that gave decrypt_as or ValuePlan as the way to upgrade, described the old derives as current, or listed them as deprecated.
|
Replies to the findings in the review body. The inline findings (1, 2, 3, 7, 8 and 9) have their own replies. Finding 4 (rustdoc and error text still describe "the query chain"): fixed in ebf5a61 and e418a3d. Every line listed now says "a field marker's query" or "a query", and the two Finding 5 (the lifecycle contract claims coverage that was deleted): fixed in ebf5a61. The sentence about the structural Finding 6 (other stale guidance): fixed in ebf5a61. |
…lder and the old traits Stack Encrypt had two ways to encrypt a record and two ways to query one. The owned API (EncryptTarget, DecryptSource, encrypt_into, decrypt_from, field markers and dynamic::Planned) now does everything the older one did, so the older one goes: the traits EncryptFrom and DecryptInto with their derives, encrypt_as and decrypt_as, the plan builder, the chain, Layout, tuples of targets and JoinContext. The two removals depend on each other. The chain ran its fields through the old traits, and the old traits were the only path left into the builder, so neither compiles without the other going too. To keep what the old derives accepted, the owned derives now take a `struct` record as a field of a `plaintext` record, and a ciphertext or a term as a field of a `context_type = NonEmpty<T>` record. The builder was the only thing that refused a set of terms that names one index twice, or two indexes that answer one operator. That refusal moves to `target::check_indexes`, which runs before any key request. As a result `Encrypted<Terms>` used directly as a target refuses such a set now, where it accepted it before. The JSON targets open only to JsonPlaintext. A typed JSON query has no replacement yet: a JSON query goes through a data plan. The stored bytes do not change, and no fixture changed. Closes #1661 Closes #1662 Part of #1488
A review of the removal found one untested wipe, public API with no caller left, and docs that still described the removed API. CI found two conformance enrolments that named tests the removal had deleted or renamed. - `Encrypted<Terms>` wipes a plaintext whose set of indexes it refuses. No test checked that: deleting the wipe passed every test, and cargo-mutants does not delete statements. A test now fails without it, for the target alone and for a `plaintext = T` record. - `op::DefaultIndex` is removed. Only the chain read it, and its docs claimed the data plan did. - `indexed(..)` composed by hand does not check its set of indexes, as the target does. Its docs now say so. - The lifecycle and EQL JSON enrolments named a deleted test and a renamed one, so `test:conformance:lifecycle` failed in CI. - Rustdoc, an error message, the conformance contract, CONTEXT.md, the unreleased CHANGELOG entries and six pending changesets described the chain, `EncryptFrom`, `DecryptInto` or `encrypt_as` as current. They name the owned API now, so the release notes do not point a reader at something that is gone. Part of #1661, #1662
The plan's tracking lines were last checked on 8 November 2026, and a week of merges made many of them wrong: it listed merged pull requests as open, closed issues as open, and said that #1524 and this plan both claim envelope version 0x02, although #1524 has since been closed. Each work item now names the issues that track it as they stand on 15 November 2026, against next at 3605302 and the open stack #1613, #1659, #1664 and #1666. Items that had no issue now name the issues filed for them. Items that are finished say so and keep their decisions on record. #1663, the removal of the context path type `Label`, is on hold until the context naming work settles what that path is called. Refs #1488, #1489
e418a3d to
24def60
Compare
Summary
Stack Encrypt (
packages/stack-encrypt) is the Rust engine that every CipherStash language SDK runs on. It had two ways to encrypt a record and two ways to query one, so a developer could not tell which to use. This pull request removes the old way.What stays is the owned API:
EncryptTargetandDecryptSourcewith their derives,encrypt_intoanddecrypt_from, field markers such asEncryptedUser::EMAIL.eq(..), anddynamic::Plannedfor a declaration that is data. The stored bytes do not change.It is stacked on #1664. It closes #1661 and #1662, which depend on each other, so they are one change. The second commit addresses the automated review posted below.
Changes
Removed from
stack-encryptandstack-encrypt-derive(about 6,900 lines of source)EncryptFromandDecryptInto, the derives of the same names, and the callsencrypt_asanddecrypt_as.plan/chain.rs,plan/build.rs,plan/value.rsandplan/field_ref.rs. That isPlan::context(..).fields()..build(),pick,cipher.encrypt(&v)..using(&plan),cipher.query(op, v).on(&plan),cipher.open(record),plan::all, and every chain state.target::Layout, tuples of targets,target::JoinContextandop::DefaultIndex. Only the builder and the chain used them.plan_buildfuzz target, with its task, corpus and workflow entries. It fuzzed the builder.Changed
#[derive(EncryptTarget)]and#[derive(DecryptSource)]no longer also emit the old traits.DecryptField, which lets a record be a field of another record, stays. It now goes throughDecryptSource.JsonPlaintext, which is wiped when it is dropped. They no longer open to aserde_json::Value.eql-bindings: the EQL v4 types implement the owned traits only.JsonTreekeepsencryption(),index()andspecs()as its own methods.Added, so that the owned derives accept what the old derives accepted
plaintext = Trecord can have astruct = ..record as a field.context_type = NonEmpty<T>can have a ciphertext or a term as a field.target::check_indexes). The builder was the only thing that refused it.Tests (about 27,000 lines touched)
stack_encrypt::target, or to a data plan.tests/plan.rsis deleted.tests/plan_grammar.rsnow covers the same context boundaries through the data plan, and keeps its name because the conformance script runs it.tests/combinators.rsis new and holds what survived of both.Docs
@cipherstash/eql, the crate and module docs, the derive's attribute reference, the conformance contracts, and the API plan's tracking lines.@cipherstash/eqlchangesets described the removed API as current. They now name the owned API, so the release notes do not tell a reader to call something that is gone.Verification
Run on this branch:
mise x --env test -- cargo nextest run --workspace --all-features: 1970 passed, 2 skipped. This includes the frozen stored-bytes tests and the fixture tests, with no fixture changed.--test ui): 89 cases pass. No snapshot contains a$RUST/snippet.cargo test -p stack-encrypt --no-default-features: 894 passed. Rustdoc is clean with and without default features.mise run test:doc: 112 passed.mise run doc: clean.cargo clippy --workspace --all-features --all-targets -- -D warningsandcargo fmt --all --check: clean.cargo crap, threshold 30): 1,591 functions, none above the threshold.cargo mutants --in-diff: 32 mutants in the changed lines; 9 caught, 23 did not compile, none missed. cargo-mutants does not delete statements, so it did not see that the wipe of a refused plaintext was untested. The review found that, andowned_records::a_plaintext_is_not_leaked_when_its_indexes_are_refusednow fails when the wipe is removed.packages/eql, with--locked:eql-bindings218 passed (stack-encryptfeature) and 235 (targetsfeature);eql-encryption-tests166 passed, 61 ignored;eql-codegen202 passed, so the generated files have no drift. Clippy and fmt are clean.mise run go:testpasses in all 13 packages;go:lintreports 0 issues;go generate ./...changes nothing;wasm:guest:testpasses (74 and 76).scripts/stack-conformance.shpass, andmise run test:conformance:lifecyclepasses. CI caught that its enrolment still named a test this PR deleted; the second commit fixes that, and a renamed test in the EQL JSON enrolment. Intest:conformance:exchangethe Rust and Go halves pass. Its EQL step, andtest:conformance:eql, call mise insidepackages/eql, whose config is not trusted in my worktree, so CI is the evidence for those two. I ran the EQL test binary that holds the renamed enrolled test directly: 12 passed.Line coverage of the code that remains. I recorded coverage before and after. It is 97.34% before and 97.17% after. 75 lines that were covered are now not covered. 72 of them are in
src/json/query.rs: the typed JSON query inputs (Input<Pathed<..>>::derive), which only the chain could reach. See the first review note.Not run locally: the 52 PostgreSQL tests of the EQL test crate, which need a database with the EQL v4 bundle. They compile, and no line of SQL in them changed. The repository script tests (
pnpm run test:scripts) are not run locally either; my worktree has nonode_modules.Related
Closes #1661. Closes #1662. Part of #1488. Stacked on #1664.
Review notes
Decisions I took that you may want to change:
cipher.query(op::Contains, ..).on(&plan)andop::AtPath(..)were the only typed way to query a JSON document. A JSON query now goes through a data plan (dynamic::query_term, orPlannedwith a query plan) until stack-encrypt: element match and jsonpath match have no typed Rust API — only the binding layer, with no compile-time checks #1459 gives the document's marker its methods. This is what the API plan says. The operator typesop::Contains,op::ContainedByandop::AtPathand the JSON index'sSupportsimpls are still there, because the data plan uses their checks, but nothing public reaches theirderiveany more. stack-encrypt: element match and jsonpath match have no typed Rust API — only the binding layer, with no compile-time checks #1459 should either use them or delete them.Indexes::select(withSelect,AtandWhole) is in the same position: the chain was its only caller in the library. I kept it, because it is a tested public accessor that picks one index out of a set, and corrected its docs.(Equality, Equality)is refused at compile time once the builder goes. I restored the builder's refusal as a run-time check. This changes one behaviour of the API that remains:Encrypted<(OreTerm<u32>, OpeTerm<u32>)>used directly as a target was accepted before and is refused now. With today's indexes a set of four or five terms can no longer be valid, sotests/index.rsnow checks sets of two and three. To drop the check, remove the two calls tocheck_indexes. The combinatorindexed(..)composed by hand does not check, and its docs now say so.PlanErrorkeeps variants that nothing raises now (IdentityWithoutField,MixedCiphers,TargetWithVerbs,NoContext,IndexNotDeclared,IndexOptionsand others). Each has an error code that the Go status table and the error-code tests list, so removing them is a separate change..via(index)), choosing a keyset by name on a call (.keyset("tenant"), which loaded the keyset lazily), and a batch in which an opening with no keyset reads any keyset (plan::all).Pending::ziptakes the scope of the write.plaintext = Trecord has no marker method. The old plan answeredop::Eqfor it throughindexes(). The derive writes noop::Answersfor a record, by design, so a hand-written impl is needed to query it.dynamic::record::encrypt_withanddecrypt_withstill run the lowering themselves. stack-encrypt: the plan chain and builder are still a second way to run a plan — remove them, and run their tests throughPlanned#1662 asked for them to build aPlanned.Plannedcalls them today, and both sharedynamic::planned::compose, so nothing observable changes either way. I left it.the_chain_and_the_lowering_are_one_engine_todayand one more), and the fixture key istyped_chain. Renaming them needs a change toconformance/stack-encrypt/v1/fixture-enrollment.jsonand to the fixture.Where to look first:
packages/stack-encrypt/src/target/operations.rsandsrc/target/index.rsfor the removal and the new check.packages/stack-encrypt-derive/src/owned_encrypt.rsandowned_decrypt.rsfor what the derives emit now.packages/stack-encrypt/tests/combinators.rsandtests/plan_grammar.rs, which replacetests/plan.rs.packages/stack-encrypt/CHANGELOG.mdfor the upgrade table.Every deleted test, with the reason and what covers it now
"None" means the feature the test covered is removed and has no replacement.
tests/plan.rs(file deleted; 16 of its 31 tests moved totests/combinators.rs)a_saved_plan_is_the_same_chain_and_decrypts_backcombinators::a_derived_record_is_the_hand_composed_combinators_field_for_field;planned::round_trips_with_no_caller_contexta_query_against_an_undeclared_index_or_type_is_refusedmarker_operator_not_answered,marker_query_type;query_refusal_conformance::shared_query_refusal_profile;planned::is_refused_where_no_index_answers_the_operatordecrypt_returns_what_can_come_back_and_refuses_a_mismatched_rowsrc/dynamic/record.rsopens_back_to_the_fields_that_come_back_in_plan_order,decrypt_and_check_record_refuse_it_before_any_key_is_retrieved; uidecrypt_struct_field_not_recoverablea_value_that_does_not_match_the_plan_is_refused_without_a_key_requestsrc/dynamic/record.rsencrypt_and_check_source_refuse_it_alike_with_no_key_requesta_refused_chain_never_loads_the_keyset_it_namesa_passthrough_name_need_not_be_a_label_segmentplan_grammar::a_pinned_identity_keys_the_field_for_the_write_the_query_and_the_read; uiderive_query_on_a_passthrough_fieldevery_build_error_is_its_ownsrc/dynamic/record.rsevery_plan_refusal_maps_to_its_field_and_reasonand four more;combinators::a_target_that_names_an_index_twice_is_refused_before_any_key_requestmatch_options_must_agree_for_a_query_to_be_declared.via).all_refuses_chains_from_two_ciphersPending::ziphas the opposite rule:target::pendings_from_two_ciphers_over_the_same_keyset_merge.all_lets_an_unnamed_opening_read_any_keysetsrc/target/pending.rsan_unscoped_pending_merged_with_a_scoped_one_takes_the_scope;keysets::a_mixed_keyset_column_opens_through_the_client_in_one_call_per_keysetkeyset_selects_the_minting_keyset_and_scopes_an_openingcombinators::every_operation_honours_the_keyset_it_runs_througha_plan_is_data_its_fields_say_what_they_declareplan_grammar::a_plan_says_where_its_context_comes_froma_chain_shows_no_valuemarkersprints_its_label_and_not_the_value;plannedprints_neither_the_value_nor_the_plana_keyset_choice_names_what_it_holdsKeysetChoice).a_borrowed_string_names_a_context_and_a_keysetplan_grammar::a_context_is_a_label_or_text_that_parses_as_onetests/plan_grammar.rs(36 tests before, 16 now; 8 moved totests/combinators.rs)both_starts_build_the_plan_the_context_start_buildsa_context_refused_at_the_call_never_loads_the_keyset_it_namesplan_grammar. None for the lazy load.an_empty_batch_description_fails_on_the_context_a_chain_refusesa_typed_field_answers_the_queries_its_target_declares_and_no_othercombinators::a_marker_query_derives_under_the_label_the_write_used; uimarker_operator_not_answereda_typed_field_of_terms_alone_does_not_come_backcombinators::a_field_of_terms_alone_matches_its_query_and_mints_no_data_keya_field_is_a_target_or_data_verbs_never_bothsrc/dynamic/record.rsthe_target_and_output_forms_are_exclusivea_one_value_typed_plan_answers_its_targets_queries_and_no_othercombinators::a_plaintext_record_is_the_combinators_bytesa_one_value_plan_quotes_no_call_context_in_its_refusalsValuePlan).a_one_value_plan_names_the_value_in_its_errors_never_its_contextcombinators::a_target_that_names_an_index_twice_is_refused_before_any_key_requesta_one_value_plan_builds_over_a_target_of_any_contextValuePlan).a_vec_written_through_a_one_value_plan_opens_through_it_in_one_requestcombinators::a_collection_runs_in_one_requesta_picker_plan_needs_no_fields_impl_and_is_the_by_name_plans_bytespick).a_plan_naming_any_field_by_name_checks_the_value_and_one_of_pickers_does_notpick).a_pinned_identity_keys_a_call_context_plan_for_query_and_openplan_grammar::a_pinned_identity_keys_the_field_for_the_write_the_query_and_the_reada_tuple_is_decryptable_when_one_element_isa_terms_only_tuple_yields_to_the_ciphertext_beside_itcombinators::a_record_opens_through_its_ciphertext_wherever_it_sitstests/op.rs(33 before, 26 now)an_operator_the_field_has_no_index_for_is_refused_when_awaitedmarker_operator_not_answered. Data plan:plannedis_refused_where_no_index_answers_the_operatora_one_value_plan_answers_from_its_indexes_by_typemarkersanswers_each_operator_through_the_index_that_answers_ita_query_chain_debugs_without_its_valuemarkersprints_its_label_and_not_the_valuean_undeclared_index_is_refused_before_the_query_is_prepared.via).a_query_names_its_context_before_or_after_the_indexa_refused_operator_query_loads_no_keysetmarker_operator_not_answered. The lazy load is gone.a_field_declares_one_index_per_operatorderive_plan::two_outputs_that_answer_one_operator_are_refused_before_any_request;src/dynamic/record.rstwo_indexes_for_one_operator_are_refusedtests/json_index.rs(50 before, 46 now) andtests/json_owned.rs(13 before, 12 now)json_ownedthe_owned_api_and_the_structural_api_open_each_others_documentsit_seals_from_a_json_plaintext_and_opens_to_onegiven_a_plan::refuses_a_query_under_other_options.via). The side assertions moved tojson_ownedit_declares_the_json_indexand the newgiven_the_json_index::names_its_options_in_its_specgiven_a_plan::opens_what_it_wrote_as_the_typed_open_doesjson_indexseals_one_entry_per_node_and_opens_againgiven_a_laid_out_plan::reports_its_layoutgiven_the_json_index::names_its_options_in_its_specgiven_a_laid_out_plan::refuses_an_open_that_names_another_contextsrc/dynamic/record.rsan_expected_context_on_a_plan_with_its_own_is_two_sourcesgiven_a_laid_out_plan::answers_a_named_index_with_its_own_options.via).tests/derive.rs(20 before, 17 now),tests/derive_plan.rs(20 before, 13 now),tests/contexts.rs(24 before, 23 now),tests/collections.rs(25 before, 24 now)derivea_record_typed_field_is_the_derives_nested_structderive::a_struct_nests_in_a_struct_as_an_ordinary_field;owned_equivalence::typed_generated_and_dynamic_declarations_are_one_enginederivean_extension_reaches_a_record_typed_field_as_the_derivesderive::a_struct_nests_in_a_struct_as_an_ordinary_fieldderivea_struct_of_targets_is_the_derives_structderive_plan::a_struct_record_writes_what_the_hand_written_combinators_writederive_plana_derived_record_as_a_typed_field_answers_its_queriesderive::a_struct_is_one_batched_call_and_rebuilds_its_plaintext. See review note 5.derive_plana_context_field_plan_carries_and_checks_the_contextowned_records::a_stored_context_opens_as_stored_and_checks_an_expectationderive_plana_context_field_plan_has_one_context_sourcederive_plana_one_value_plan_with_its_own_context_refuses_the_callersValuePlan).derive_plana_struct_record_plan_is_the_hand_written_chaina_struct_record_writes_what_the_hand_written_combinators_writederive_planevery_derived_plan_buildsLayout).derive_plana_context_field_plan_debugs_its_stored_context_and_targetPlan'sDebug).derive_plana_struct_record_and_its_plan_written_by_hand_open_each_otherowned_equivalence::typed_generated_and_dynamic_declarations_are_one_enginederive_plana_context_field_record_and_its_plan_open_each_otherderive_plan::a_context_field_record_writes_what_the_hand_written_combinators_writecontextsa_plan::built_without_a_context_shows_the_label_the_call_namescollectionsa_late_element_failure_wipes_the_recovered_prefix_on_the_structural_path_toocollections::a_late_element_failure_wipes_the_plaintext_recovered_before_itCompile-fail cases (
tests/ui): 88 fail cases before, 78 now; 13 pass cases before, 11 now.aead_context_with_termowned_aead_record_with_term(it was a duplicate)bind_in_querymarker_query_has_no_bindjoin_context_is_sealedJoinContext).op_json_not_answeredquery_refusal_conformanceop_matches_on_integer,op_target_does_not_answer,plan_query_undeclared_indexmarker_operator_not_answered,derive_match_on_integerpart_on_structural_deriveplan_picker_wrong_typepick).plan_empty_index_setindexes_empty_setpass/plan_chain,pass/plan_grammarAssertions that changed in a kept test
record_lowering: the typed side, opened under another tenant, now fails withError::Kms, notError::ContextMismatch. A combinator holds no expected context to compare. The data plan still answersContextMismatch.context_conformance::shared_context_boundaries: the typed opening leg ran 11 cases and now runs 4. The query cases moved todynamic::query_term.json_index: abetweenon a path with no ordered index wasError::Term(JsonPathNotOrdered). The data plan refuses it earlier, withReason::OperatorNotAnswered.json_leakagecorpus: the sampled equality path$.tags[0]is now$.tags[*]. The typed operator read a position as a wildcard; the data plan refuses a position, andjson_operators::malformed_inputs_are_refused_with_their_reasoncovers that refusal.tests/index.rs:a_tuple_of_terms_lowers_to_the_tuple_of_their_indexes_in_orderchecks sets of two and three terms. See review note 2.json_leakageand 12 tests ofjson_indexnow need thedynamicfeature, because their queries go through the data plan.The EQL test crate lost no test. Two
compile_faildoctests for.viaineql-bindingsare deleted.