Repository navigation
Conversation
…, compiled without the dynamic feature The JSON operators' inputs, their checks and the derivation of their operands lived in the dynamic interface, behind its feature, so a typed query could not share them. They move to json::inputs and json::predicate, which build typed operands (Derived) instead of bound bytes. The dynamic interface reads a binding's frames into the same inputs, in the same order, and writes the typed operand as the bytes it always bound. dynamic::Error and Reason move to a module compiled without the feature, so the core refuses an input with the reason a binding has always seen. JsonElementsQuery and JsonPredicateQuery are the typed forms of the element and predicate operands. No operand byte, refusal or reason changes. Refs #1459
A Rust program could reach the JSON operators only through the dynamic interface, naming the operator as text and passing its inputs as JSON, since the query chain that was the last typed way is gone. A field whose type stores a JSON document now has one marker method for each kind of JSON query: contains, contained_by, at(path) with a comparison or its selector, elements(path) with member conditions, predicate(text) and children(path). Each builds the input the dynamic parser reads into, so it derives the operand a data plan derives for the same declaration, and returns the field type's own query type (JsonQueries): the engine's for JsonDocument, EQL's for JsonTreeDocument. A document field names its JSON index options with #[stash(json(ordered = [..], sealed_root = ..))], as Go's struct tag does. The derive seals the field under them through JsonTarget, and the field's marker queries under them. A path is text, checked before any key is requested; the values are typed, and an element predicate needs a condition to compile. The JSON index's Supports and Answers impls for the typed operators, and AtPathTerms, are removed: only the removed chain reached them. Closes #1459
🦋 Changeset detectedLatest commit: 7ca15a4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true |
coderdan
left a comment
There was a problem hiding this comment.
Automated review by a Claude Opus subagent, requested by the author.
The refactor in 0579ee5e3 keeps the dynamic path's checks, their order, their refusal classes and the operand bytes, apart from one unreachable internal error (finding 8). The new marker code wipes what it holds, refuses before any key request, and derives under the same context and options the field is sealed under; I confirmed the scoped, nested and sealed_root = false cases with a throwaway program. The one correctness bug is how a float becomes JSON: at(path).eq with a non-finite float silently queries for null, and an f32 never matches the value a caller wrote (finding 1). The rest is stale docs, test gaps, and two API points.
Findings
- F1, major:
at(path).eqwithf64::NANor an infinity queries fornull, and anf32widens, atpackages/stack-encrypt/src/marker/json.rs:259. Verified (code read and run). - F2, minor:
at(path).eqcannot ask fornull, whicheq_ataccepts, atpackages/stack-encrypt/src/marker/json.rs:365. Verified. - F3, minor: docs and help text still name the removed typed JSON queries (
op::AtPath,Input<Pathed<..>>), atpackages/stack-encrypt/src/op/mod.rs:492,packages/stack-encrypt/src/sem/mod.rs:359and:386, andpackages/stack-encrypt/src/json/mod.rs:211. Verified. Details below. - F4, minor: the plan says
ElementMatch,PredicateandChildrenare retired, but this PR adds them as public type tags, atdocs/plans/2026-10-24-stack-encrypt-api-plan.md:533and:1565. Verified. - F5, minor: no test checks that the stored document was sealed under the attribute's options, and none uses
sealed_root = false, atpackages/stack-encrypt/tests/json_marker.rs:89. Verified. - F6, minor: an input refusal reaches the caller as
Error::Other, whose docs say a third-party target failed and whose payload is empty, atpackages/stack-encrypt/src/marker/json.rs:550. Verified. - F7, minor: a field with no
json(..)is queried underJsonOptions::default()whatever options its type seals under, atpackages/stack-encrypt/src/json/target.rs:144. Plausible (no type in the tree hits it). - F8, nit: the refactor moves one unreachable refusal from a synchronous
dynamic::Error::Internalto an asynchronouscrate::Error::Other, atpackages/stack-encrypt/src/dynamic/json_query.rs:571. Verified. - F9, nit: the module docs read as if a position in an
eqorselectorpath is aTermError, atpackages/stack-encrypt/src/marker/json.rs:96. Verified.
F3 in full (lines outside the diff)
F3, minor, verified. This PR removes the Input impls for (P, V) and (P, (V, V)) and the JSON index's Supports impls. Several docs and two user-facing help messages still describe them:
packages/stack-encrypt/src/op/mod.rs:492-500: theInputtrait docs say "A(P, V)is an input ofPathed<One>over aserde_json::Value", and theon_unimplementednote says "op::AtPath(arityPathed) takes a path and the leaf's input,&("$.a", value)". After this PR nothing implementsInput<Pathed<_>>(rg 'Input<Pathed'finds no impl).packages/stack-encrypt/src/sem/mod.rs:359:TermError::JsonLeafNotOrdered's help says "queryop::AtPath(op::Eq)for a boolean or a null". This is the text a marker caller now sees forat(path).gt(..)with a bad value. No such query exists, and the marker cannot ask fornull(finding 2).packages/stack-encrypt/src/sem/mod.rs:386:TermError::JsonValueNotScalar's help namesop::AtPath(op::Eq)andop::Contains.packages/stack-encrypt/src/json/mod.rs:211:JsonOptions::orderedsays leaves "answerop::AtPath(op::Lt)".
Scenario: a Rust caller gets JsonLeafNotOrdered from PROFILE.at("$.age").gt(..) and is told to call an API that does not exist. Suggested fix: name the marker methods (at(path).eq(v), contains(doc)) and the dynamic operator names (eq_at, contains) in the help text. Drop the Pathed input sentence from the Input docs and the diagnostic note. If Pathed and AtPath's arity now only serve the dynamic Op enum, say so there.
Also stale, same cause: the plan's "Names. The method names above are proposals" at docs/plans/2026-10-24-stack-encrypt-api-plan.md:1565. The methods are built now.
Checked and found clean
- Refactor:
read_input,read_eq_at,read_ordering,read_between,read_elements,read_predicateandread_matchmake the same checks in the same order, with the same refusal classes.Input::elementsandInput::predicatere-check the bound, the empty list and the vars object after the dynamic reader has already checked them, so the order does not change. - Refactor:
Shape::ofmaps the same leaves to the same shapes.operand_valuenow usesserde_json::to_valuewhere it usedto_vecthenfrom_slice. These give the sameValue, because serde_json is built withoutpreserve_orderorarbitrary_precisionhere.JsonElementsMemberandJsonElementsQueryserialize like the oldMemberOperandandElementsOperand.Derived→Operandwrites the same bytes. dynamic::ErrorandReasonmove torefusal.rsunchanged and are re-exported at the same path.- Wiping: every
Bodyvariant holdsParsedorZeroizing<String>. Refused inputs, the remaining members after an early refusal, replacedvars, a failed options function and a dropped query are all wiped.JsonPlaintext::into_valueleavesnullbehind, and its value goes straight intoParsed.seal_field_owned's refusal path zeroizes the plaintext.json_owned_encryptionand theOptionimpl follow the existing guarded pattern. - Refusal before any key: options, input checks and derivation refusals all return
Pending::failedbefore any request. - Context and options: a probe program sealed records and checked the marker's selectors, tokens and ordering terms against the stored entries. It covered a scoped record, a field reached through
Marker::field, anOption<JsonDocument>field and asealed_root = falsefield. All matched.Marker::fieldcarries the inner field's options. - Derive: the
__stash_json_options___field_Nnames come from field indexes, so they cannot clash with each other or break on raw identifiers. Generics pass throughsplit_for_impl. Tuple fields get no marker but are still sealed under their options. Misuse is refused with a clear error (tests/ui). - CHANGELOG: the removals are under "Breaking". The changeset and the
eql-bindingsCHANGELOG match the code. There are no issue numbers in code comments and no newignoreorno_runblocks. tests/json_marker.rspasses locally (16 of 16).
| }; | ||
| } | ||
| json_values! { | ||
| ordered: i8, i16, i32, i64, u8, u16, u32, u64, f32, f64, String, &str; |
There was a problem hiding this comment.
F1, major: a non-finite float in at(path).eq silently becomes null, and an f32 widens so it never matches.
JsonScalar::into_json is Value::from(self). serde_json's From<f64> and From<f32> return Value::Null for NaN and the infinities (Number::from_f64(f).map_or(Value::Null, Value::Number)). Input::eq_at accepts null as a scalar, so the query derives the needle for {"score": null}. From<f32> also widens: 0.1f32 becomes 0.10000000149011612.
Verified with a throwaway program against this branch. The record seals {"score": null} and {"score": 0.1} under json(ordered = ["$.score"]):
eq(NaN) accepted; matches {score: null}: true; matches {score: 0.1}: false
eq(INFINITY) matches {score: null}: true
eq(0.1f32) matches {score: 0.1}: false; eq(0.1f64): true
gt(NaN): Err(Term(JsonLeafNotOrdered))
Scenario: PROFILE.at("$.score").eq(x), where x is a computed f64 that turns out to be NaN, returns the rows whose score is null. There is no error. A caller with an f32 gets no rows for a value they can see in the document. On the ordering side NaN is refused, but the message says "takes a number or a string", which is confusing for an f64.
Suggested fix: make the conversion fallible. Refuse a non-finite float before any key is requested, as an input refusal or a new TermError. For f32, either drop it from JsonScalar and JsonOrdered, or convert it through its shortest decimal form (f32::to_string then parse as f64) so 0.1f32 means 0.1. Add tests for NaN, the infinities and an f32 in eq, the orderings and the elements conditions.
There was a problem hiding this comment.
Fixed in cf40c37. A value that is not a JSON number is now refused before any key is requested, rather than becoming null:
JsonScalar::into_jsonreturnsNonefor a NaN or an infinity. The query holds that as a refusal, andencryptreturns the newTermError::JsonNumberNotFinite(stack_encrypt::json_number_not_finite). This coverseq, the orderings,betweenand theelementsconditions.- An
f32converts through its shortest decimal form, so0.1f32is0.1. - Tests in
tests/json_marker.rs, modulevalues: a NaN or an infinity ineq,gtandbetweenis refused with no key request, andeq(0.1f32)andeq(0.1f64)both find a stored0.1.
| /// document that holds it there, so exact. At a path with a wildcard, | ||
| /// rows where any element holds it. A position (`[2]`) is refused: a | ||
| /// selector reads it as the wildcard. | ||
| pub fn eq(self, value: impl JsonScalar) -> MarkedJson<T, AtPath<Eq>, C::Start> { |
There was a problem hiding this comment.
F2, minor: at(path).eq cannot ask whether the value at a path is null.
JsonScalar is implemented for booleans, numbers and strings only, so a null cannot be passed. The dynamic eq_at accepts null, since scalar allows it. The predicate compiler builds EqAt(path, null) itself for P == null. And TermError::JsonLeafNotOrdered's help tells callers to use equality "for a boolean or a null" (finding 3). The removed typed op::AtPath(op::Eq) took any Into<Value>, so this is a narrowing too.
Scenario: a Rust caller who wants profile @> '{"deleted_at": null}' must write predicate("$.deleted_at == null"). That compiles to an operand evaluated by json_tree_match, which uses no index. A Go caller sends the plain containment needle.
Verified by reading json_values! and Input::eq_at.
Suggested fix: add ValueAt::is_null(), or implement JsonScalar for Option<V: JsonScalar> with None as null, or for a unit JsonNull type. Keep JsonOrdered off it.
There was a problem hiding this comment.
Fixed in cf40c37: at(path).is_null() asks for a JSON null at the path. It is equality with null, which the dynamic eq_at already accepts. It is not on JsonOrdered.
values::is_null_finds_a_json_null_and_not_an_absent_path checks that it matches a stored null, and matches neither an absent path nor a number. The JsonLeafNotOrdered help text now points at it.
| The marker methods build the same inputs the dynamic interface reads a | ||
| binding's frames into (`json::inputs`, compiled without the `dynamic` | ||
| feature), so the typed form and the dynamic form share one set of input | ||
| rules and derive the same operands. The operand types `Contains`, `AtPath`, |
There was a problem hiding this comment.
F4, minor: the plan says these operand types are retired, but this PR adds them back as public types.
This paragraph, which the PR edits, still ends: "The operand types Contains, AtPath, ElementMatch, Predicate, and Children that the plan proposed earlier are retired, replaced by these methods, on 7 November 2026." The same claim is at line 1565-1567 ("Names. ... are retired").
This PR adds marker::json::{Selector, ElementMatch, Predicate, Children} as public type tags. MarkedJson<T, Op, S> takes op::Contains, op::ContainedBy and op::AtPath<..> as Op (Review note 3). A reader of the plan will look for types that the plan says do not exist, and find them.
Verified.
Suggested fix: say that these names now exist as the kind parameter of MarkedJson, that they carry no values, and that the methods are still the API. Give the date of that decision. Also update "Names. The method names above are proposals" at line 1565.
There was a problem hiding this comment.
Fixed in cf40c37. Both places in the plan now say that those names were retired as operands a caller builds. They also say that MarkedJson<T, Op, S> names the query's kind with value-free type tags: op::Contains, op::ContainedBy or op::AtPath of a comparison, plus Selector, ElementMatch, Predicate and Children, recorded on 16 November 2026. The "proposals" line now says the method names were confirmed in the naming review.
| use super::*; | ||
|
|
||
| #[tokio::test] | ||
| async fn it_round_trips_under_its_options() { |
There was a problem hiding this comment.
F5, minor: no test checks that the stored document was sealed under the attribute's options, and no test uses sealed_root = false.
it_round_trips_under_its_optionsonly round-trips.options_that_name_a_position_refuse_the_seal_and_the_queryshows the options function runs during sealing. But nothing checks that the stored document carries an ordering term at an ordered path. The same is true ofjson_tree_marker.rs. Ifseal_fieldwere changed to call the options function and then seal under the defaults, every test in both files would still pass. The marker's orderings would then match no stored row.sealed_root = falseis parsed and documented, but no passing test uses it. It changes the read tokens (json/derive.rs:2148), sochildrenandelementson such a field depend on the marker receiving it.- The scoped and nested tests compare only the containment needle with the stored document. They do not compare a selector, an ordering or a token. The data-plan comparisons cover only the unscoped top-level field.
Verified by reading. I also ran a throwaway program to check that the code is right today. A sealed_root = false record's header says false. Its children("$.obj") token is found in the stored document. The $.score entry is found by the marker's selector and carries an ordering term, top level and nested through Marker::field, and under a scope.
Suggested fix: add those checks as tests. For example: doc.entries().iter().find(|e| e.selector() == sel.as_bytes()).and_then(|e| e.ordering_term()).is_some(); a sealed_root = false record with doc.header().sealed_root() == false and doc.find(&key.token_bytes()[0]).is_some(); and the same selector and token checks in under_a_scope and in_a_nested_record.
There was a problem hiding this comment.
Fixed in cf40c37, in a new module the_fields_options in tests/json_marker.rs:
an_ordered_paths_entry_carries_an_ordering_term_at_the_markers_selector: the entry the marker'schildren("$.age")token finds has an ordering term and the marker's selector.$.name's entry has no ordering term. Aseal_fieldthat ignored the options now fails this test.an_open_root_is_sealed_and_queried_open:sealed_root = falsereaches the header, and the marker's root token finds the root.under_a_scope_the_tokens_and_selectors_are_the_scopes: under the right scope, the token and selector find the ordered entry; under another scope they find nothing.through_an_outer_field_the_tokens_and_selectors_are_the_inner_fields: the same checks throughMarker::field.
| /// A refusal of the JSON input, as the crate reports it: a cipher error, | ||
| /// such as a term refusal, passes through; the input's own refusals are | ||
| /// boxed with their reason, as `dynamic::Planned` boxes them. | ||
| fn wrap(error: crate::refusal::Error) -> crate::Error { |
There was a problem hiding this comment.
F6, minor: input refusals are reported through a variant whose docs and payload say something else.
Review note 2 covers the choice of Error::Other. It misses two consequences.
crate::Error::Otheris documented (cipher.rs:280-290) as "A third-partyEncryptTarget/DecryptSourceimplementation failed for a reason of its own". Forelements("$.visits")with no wildcard it now means "your input is wrong".ErrorPayloadforOtheris an empty map (cipher.rs:465), and the diagnostic code isstack_encrypt::other. TheReason(JsonPathNoWildcard,JsonPredicateSyntax, and the rest) is lost to any caller that reads the code or the payload. Without thedynamicfeature the boxed type is private, so the reason survives only in the message text.
Scenario: a service maps stack_encrypt::other to a 500 "internal" response. A user's bad jsonpath predicate then shows up as a server fault.
Verified by reading wrap, Error::Other and its payload.
Suggested fix: at the least, update Error::Other's docs to include this use, and forward the boxed refusal's payload (downcast to the refusal type in payload()). Better: add a variant for a refused query input that carries the reason, as Error::Term does for term refusals.
There was a problem hiding this comment.
Partly fixed in cf40c37:
Error::Other's docs now say it also carries a JSON query's refusal of its input, and that this is the caller's fault, not the engine's.- Its
ErrorPayloadforwards the boxed refusal's payload, soreason,field,constructandatreach anything that reads the payload, with or without thedynamicfeature. Tested byrefusals::a_boxed_refusals_payload_names_its_reason.
Not done: the code is still stack_encrypt::other. A dedicated variant needs a public type for the refusal, which is dynamic::Error today and private without the feature. It would also change what dynamic::Planned returns for the same refusals, since Planned boxes them the same way. I have left that for Dan to decide in the PR thread.
There was a problem hiding this comment.
Done in bf1511f, as Dan asked: a refusal of the caller's input is now the new Error::Input.
- It holds the dynamic interface's error, public without the
dynamicfeature asstack_encrypt::refusal::Errorandrefusal::Reason(the same types asdynamic::Erroranddynamic::Reason). - Its message, code and payload show through. An
elementspath with no wildcard isstack_encrypt::dynamic_sourcewithreason: json_path_no_wildcard, and a predicate that does not parse isstack_encrypt::dynamic_predicatewith itsreason,constructandat. These are the codes the dynamic interface reports for the same input. dynamic::Planned, and anIndexSpecrun directly over aValueFrame, report a refused input the same way. The second shipped in 0.3.0, so it has a Breaking entry in the CHANGELOG.Internaland a target resolver's own failure stayError::Other.Other's docs and payload are back to what they were before cf40c37.- The Go guest classifies
Error::Inputas it classifies the dynamic refusal inside it, so these refusals areSTATUS_ENCODINGthere, notSTATUS_INTERNAL.
Tested by refusals::an_input_refusal_has_the_dynamic_interfaces_code_and_payload, the reason checks in refusals::the_input_shapes_a_marker_cannot_build_are_refused_with_their_reason, the error-code registry in codes.rs, and the guest's status::tests::an_input_refusal_is_the_callers_input.
| label = "this field's type is `{Self}`", | ||
| note = "the JSON queries (`contains`, `at`, `elements`, `predicate`, `children`) belong to a field whose type stores a JSON document, such as `JsonDocument` or `eql_bindings::v4::json_tree::JsonTreeDocument`" | ||
| )] | ||
| pub trait JsonQueries { |
There was a problem hiding this comment.
F7, minor (plausible): a field with no json(..) is queried under JsonOptions::default(), whatever options its type seals under.
Marker::new sets json: crate::json::default_options. The derive seals a field with no attribute through EncryptTarget::encryption(), not through JsonTarget. The marker's JSON methods need only T: JsonQueries, which is public, can be implemented outside the crate, and does not depend on JsonTarget. So the two agree only because JsonDocument and JsonTreeDocument both seal under the defaults.
Scenario: a crate defines a document type whose EncryptTarget::encryption() seals under sealed_root = false, or with ordered paths, and implements JsonQueries for it. A field of that type with no attribute gets a marker that refuses every ordering. Worse, its children and elements derive tokens under the sealed root, so the operands silently match nothing.
Plausible: no type in the tree does this today.
Suggested fix: give JsonTarget an associated fn default_options() -> JsonOptions (default JsonOptions::default()). Make JsonQueries: JsonTarget<_>, or give JsonQueries the same function. Have the marker and the derive use it when there is no attribute. Or, smaller: document on JsonQueries that the type's EncryptTarget::encryption must seal under JsonOptions::default().
There was a problem hiding this comment.
Fixed in cf40c37. JsonQueries has a new default_options(), which returns JsonOptions::default() unless a type overrides it, and documents that it must equal what the type's EncryptTarget::encryption seals under. A marker whose field names no json(..) now holds None and queries under T::default_options(), instead of the hard-coded defaults.
the_fields_options::a_field_that_names_no_options_queries_under_its_types_own tests a custom document type that seals with $.age ordered. Its marker orders at $.age, and its token finds the stored entry.
| hi: eql(&hi)?, | ||
| }, | ||
| Derived::Selector(selector) => Operand::Text(selector.to_string()), | ||
| Derived::Node(key) => { |
There was a problem hiding this comment.
F8, nit: one refusal changed class in a commit described as changing none.
Before (24def60db, dynamic/json_query.rs:846), children and child_count whose key held other than one token failed derive synchronously with dynamic::Error::Internal. Now it fails the pending asynchronously with crate::Error::Other("a path with no wildcard names one node"). This is unreachable for a path that node_path accepted, so no binding sees it. But the PR says no refusal or refusal class changes, and Internal is the class a binding must not blame on its caller.
Verified by reading both versions.
Suggested fix: check the token count in inputs::derive's Input::Node arm and return Error::Internal there, before the pending, as before. Or carry the one token in Derived::Node, so both this code and eql-bindings' token() stop re-checking it.
There was a problem hiding this comment.
Fixed in cf40c37. json::inputs::derive checks that a node's key holds exactly one token in the Input::Node arm, and returns dynamic::Error::Internal synchronously, as before the refactor. The dynamic Operand conversion relies on that check.
| //! operators apply, through the same code: an ordering at a path the field | ||
| //! does not order is refused | ||
| //! ([`TermError::JsonPathNotOrdered`](crate::sem::TermError::JsonPathNotOrdered)), and so is | ||
| //! a position (`[2]`) where the operand is a selector. A refusal that is |
There was a problem hiding this comment.
F9, nit: the docs make the position refusal sound like a TermError.
"an ordering at a path the field does not order is refused (TermError::JsonPathNotOrdered), and so is a position ([2]) where the operand is a selector". For at(path).eq and at(path).selector(), a position is refused as Error::Other with Reason::JsonPathPosition. For an ordering it is TermError::JsonOrderingAtPosition. refusals::equality_at_a_position_is_refused asserts Error::Other.
Verified.
Suggested fix: give each refusal its class, for example: "A position ([2]) is refused too: in an ordering as TermError::JsonOrderingAtPosition, and in eq or selector as an input refusal (below)."
There was a problem hiding this comment.
Fixed in cf40c37. The paragraph now gives each refusal its class:
Error::Termfor the index's own refusals: an unordered path, an ordering at a position, a value an ordering cannot take, and the new non-finite number.Error::Otherfor the input's shape: a position where the operand is a selector (JsonPathPosition), anelementspath with no wildcard, achildrenpath with one, and a predicate the grammar does not read. It also says the payload carries the reason.
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 85 | 0 | 103 | 0 |
Every mutant in the changed lines was caught by a test.
A review of the marker's JSON queries found that a NaN or an infinity became JSON null, so at(path).eq(f64::NAN) matched the documents holding null, and that an f32 widened to the f64 its bits name, so eq(0.1f32) missed a stored 0.1. Such a number is now refused before any key is requested, as TermError::JsonNumberNotFinite, and an f32 is the number its shortest decimal form names. at(path).is_null() asks for a JSON null, which equality at a path took before and the marker could not say. A field that names no json(..) options is queried under the options its type seals under (JsonQueries::default_options), not the engine's defaults whatever the type. Error::Other's payload carries a JSON input refusal's reason, and its docs say it can be one. A node key that does not hold one token is refused synchronously, as before the refactor. Docs and help text that named the removed typed operators name the marker and dynamic operators. Tests cover each of these, and the field's options end to end: an ordered path's stored entry carries an ordering term at the marker's selector, sealed_root = false reaches the header, and the tokens and selectors match under a scope and through an outer field. Refs #1459
|
Replies to the review. Each inline finding has its own reply naming the fixing commit. F3 (docs and help text that still named the removed typed JSON queries): fixed in cf40c37.
All nine findings are addressed. F6 is only partly addressed: a dedicated error variant is a decision for Dan, because it adds a public type and changes what |
A field marker's JSON query whose input had the wrong shape, and a data
plan run through dynamic::Planned whose source did not fit it, failed
with Error::Other, whose code (stack_encrypt::other) a caller cannot
tell from an engine fault. A service that maps that code to a server
error would blame itself for a user's bad jsonpath predicate.
They now fail with Error::Input, which holds the dynamic interface's
error. Its message, code and payload show through, so one refusal
reports the same way on both paths. The error type is public without
the dynamic feature as stack_encrypt::refusal::{Error, Reason}, the same
types as dynamic::{Error, Reason}. The module's own bug (Internal) and a
target resolver's own failure stay Error::Other. One conversion,
refusal::Error::into_crate, serves the marker, Planned and an IndexSpec
run over a ValueFrame.
Breaking: an IndexSpec run directly over a ValueFrame it has no term for
failed with the dynamic Error::Term in Error::Other; it is now in
Error::Input, with the code stack_encrypt::dynamic_term.
The Go guest classifies Error::Input as it classifies the dynamic
refusal it holds, so these are STATUS_ENCODING rather than
STATUS_INTERNAL.
Refs #1459
… unasserted The in-diff mutation run over the typed JSON queries (#1459) left five mutants alive. Four are live at HEAD and now fail a test: - a `&String` value at a path derives the operand an owned one does; - a marked JSON query prints its field and not its values; - an element predicate reports its member count and that it is not empty, and a constructed one with no member reports that it is; - a nullable field of a type with its own options queries under them, which pins `Option<T>` forwarding `default_options`. The fifth replaced the trait's default `default_options` body with `Default::default()`, which is the body it already has.
… equivalent mutant `Input::variant`, which a compiled predicate's `Debug` prints for each leaf, had four of its twelve arms covered and scored 79.6 against the CRAP threshold of 30 (#1459). A unit test now builds one input of each kind and checks both of its names. The second in-diff mutation pass left one survivor: the trait's default `JsonQueries::default_options` replaced with `Default::default()`, which is the body it has. It joins the documented equivalent exclusions, anchored on that replacement, so `Option<T>`'s forwarding impl is still mutated and pinned by its test.
Summary
Stack Encrypt (
packages/stack-encrypt) is the Rust engine every CipherStash language SDK runs on. It can store a JSON document encrypted, as EQL'spublic.eql_v4_json_searchcolumn, and answer queries over it: containment (@>), comparisons at a path, element predicates, jsonpath predicates and child reads. Until now a Rust program could only reach those queries through the dynamic interface, the layer built for language bindings, by naming the operator as text and passing its inputs as JSON. #1666 removed the last typed way.This pull request gives a document field of a derived record one marker method for each kind of JSON query, as the API plan sets out:
Each query returns the EQL type a SQL parameter binds as, and is the same operand the Go SDK sends for the same declaration. It is stacked on #1666 and closes #1459.
Changes
One definition of each JSON input (first commit, no behaviour change)
dynamicintojson::inputsandjson::predicate. They no longer need thedynamicfeature, and they produce typed operands instead of bytes.dynamic::ErrorandReasonmove to a module that always compiles, so the core can refuse with the reasons a binding has always seen.json::JsonElementsQueryandjson::JsonPredicateQuery, the typed element and predicate operands.Marker methods (
marker::json)contains,contained_by,at(path)witheq,lt,lte,gt,gte,betweenandselector,elements(path)with member conditions andwildcard_bound,predicate(text)withvarsandwildcard_bound, andchildren(path).json::JsonQueries. That trait converts each engine operand into the field type's query type: the engine's own types forJsonDocument, and the EQL types forJsonTreeDocument.MarkedJson. It takes a scope with.scope(..)and ends with.encrypt(&keyset), like a scalar field's query.Field options (
stack-encrypt-derive)#[stash(json(ordered = [..], sealed_root = false))]names a document field's JSON index options, as Go'sencrypt_into=JsonSearch(ordered=..)tag does. You chose this over options carried in the field's type.json::JsonTargetunder those options, and gives the field's marker the same options. Without the attribute a field takes the default options, which order no path.EQL (
eql-bindings)JsonTreeDocumentimplementsJsonTargetandJsonQueries, so its marker returnsJsonTreeQuery,JsonTreeLeafQuery,JsonTreeElementsQuery,JsonTreePredicateQuery,JsonTreeSelectorandJsonTreeToken.Removed
Jsonindex'sSupportsimpls forop::Contains,op::ContainedByandop::AtPath,JsonDocument'sAnswersandLabelledTargetimpls, andjson::AtPathTerms. Only the query chain refactor(stack-encrypt,eql-bindings)!: remove the plan chain, the builder and the old traits #1666 removed reached them. The operator types stay, because the dynamicOpenum is generated from them.Docs
@cipherstash/eql, and corrections to the refactor(stack-encrypt,eql-bindings)!: remove the plan chain, the builder and the old traits #1666 changeset and the API plan, which now shows the field attribute.Verification
Run on this branch:
mise x --env test -- cargo nextest run --workspace --all-features: 2030 passed, 2 skipped (re-run afterError::Input: two tests that assertedError::Otherwere updated, then pass), including the conformance and frozen stored-bytes suites, with no fixture changed. A first run hit one failure instack-auth'stest_poll_for_token_unknown_error. That crate is untouched here; the test passed alone and in a full re-run.tests/json_marker.rs: 24 tests. Each kind of marker query is checked against a sealed document, and against the data plan's operand for the same declaration. Containment needles are compared as sets, and every other operand byte for byte.packages/eql/tests/encryption/tests/json_tree_marker.rs: 3 tests, including the plan's "Query a JSON document" example run as a program.$RUST/snippet. One existing snapshot,op_target_not_labelled, changed:JsonDocumentno longer appears among the types that implementLabelledTarget.cargo nextest run -p stack-encrypt --no-default-features: 885 passed; doctests 113 passed. Clippy, and rustdoc with-D warnings, are clean with and without default features.mise run test:doc,mise run doc, workspace clippy with-D warnings, andcargo fmt --all --check: clean.packages/eql, with--locked:eql-bindings219 passed (stack-encryptfeature) and 236 (targets);eql-encryption-tests169 passed, 61 ignored (PostgreSQL);eql-codegen205 passed. AfterError::Input,eql-bindings(targets) andeql-encryption-teststogether: 405 passed, 0 failed. Clippy and fmt are clean.wasm:guest:test74 and 76 passed (75 and 77 afterError::Input, with its status test);mise run go:testpasses in 13 packages (again afterError::Input, against all five guests rebuilt);go:lintreports 0 issues; lifecycle conformance passes.cargo mutants --in-diff, as CI runs it), in two passes:&Stringvalue, a query'sDebug,JsonElementsQuery::is_emptyboth ways, andOption<T>forwarding its type's options). The fifth no longer exists: the review fixes rewrote that function.Error::Input: 28 mutants. 10 were caught, 17 did not compile and 1 survived. The survivor replacesJsonQueries::default_options's body withDefault::default(), which is the same call. It is now excluded as equivalent in.cargo/mutants.toml.mise run crap:stack-encrypt): first failed onInput::variant(score 79.6 against a threshold of 30, 4 of 12 arms covered). A unit test now covers every arm, and all 1655 functions are under the threshold.Not run locally: the PostgreSQL tests, which need a database with the EQL v4 bundle, and the EQL step of the conformance script, which my worktree cannot run.
Review. A review by a Claude Opus subagent is posted on this PR with nine findings, all answered in
cf40c376b. The one real bug: a NaN or an infinity became JSONnullinat(path).eq, and anf32widened. Both are fixed and tested. The last finding, a dedicated error variant for a refused input, isError::Input, added in a later commit.Related
Closes #1459. Stacked on #1666. Part of the API plan merged in #1365. #1677 (open one child bound to its parent) and #1668 (marker queries for the ordered EQL scalar types) stay open.
Review notes
Decisions you may want to change:
stashgenwrites; that is a new issue if you want it.Error::Input. For example, anelementspath with no wildcard, or a predicate the grammar does not read. You asked for a dedicated variant after the review. It holds the dynamic interface's error, now public without thedynamicfeature asstack_encrypt::refusal::Error(the same type asdynamic::Error). Its message, code (stack_encrypt::dynamic_source,stack_encrypt::dynamic_predicateand the rest) and payload are that error's, so one refusal reports the same way on both paths.dynamic::Planned, and anIndexSpecrun directly over aValueFrame, now report a refusal of the input the same way, where they returnedError::Other. The second shipped in 0.3.0, so it has a Breaking entry in the CHANGELOG. The Go guest mapsError::Inputto the status its dynamic refusals get (STATUS_ENCODING), where it fell through toSTATUS_INTERNAL. A term refusal, such asJsonPathNotOrdered, is stillError::Term; this module's own bug (Internal) and a target resolver's own failure stayError::Other.MarkedJson<T, op::AtPath<op::Gt>, S>reuses the operator types for containment and the comparisons, so a marker method, the dynamicOpand the operator type name the same operator. Selectors, element match, predicates and children have no operator type, somarker::jsonadds four tags:Selector,ElementMatch,PredicateandChildren. JSON queries are a separate type fromMarkedQuerybecause Rust refuses twoencryptmethods it cannot prove disjoint.JsonTarget,JsonQueries(withdefault_options),Error::Input, therefusalmodule,TermError::JsonNumberNotFinite,ValueAt::is_null,JsonElementsQuery,JsonPredicateQuery,MarkedJson,ValueAt,Elements,JsonScalarandJsonOrdered. I renamed my firstAtandMatchtoValueAtandPredicate, becausetarget::Atand theMatchindex already use those names.children(path)onJsonDocumentreturns aJsonSubtreeKey. It holds the node's one token, so the engine needs no new token type. OnJsonTreeDocumentit returnsJsonTreeToken.jsonb. I added no new database tests.Where to look first:
packages/stack-encrypt/src/marker/json.rs: the marker methods andMarkedJson.packages/stack-encrypt/src/json/inputs.rs: the shared inputs, the checks andderive.packages/stack-encrypt/src/dynamic/json_query.rs: the dynamic reader, now thin. Its checks are in the same order as before.packages/stack-encrypt-derive/src/owned_encrypt.rs:json_options_fnsand the two sealing calls.packages/stack-encrypt/tests/json_marker.rsandpackages/eql/tests/encryption/tests/json_tree_marker.rs.