Skip to content

test: sign derivation oracle and rule tables (tests only) - #1331

Draft
zzylol wants to merge 1 commit into
mainfrom
test/sign-derivation
Draft

zzylol wants to merge 1 commit into
mainfrom
test/sign-derivation

Conversation

@zzylol

@zzylol zzylol commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: the memo's value signs (GroupProperties::column_signs) are checked only against hand-picked expectations, and several rules are unsound under the sign-bit definition. Example: sum_over_time(neg[5m]) over a window of only -0.0 is labelled NonPositive, but the executor returns +0.0.
  • Change: tests only, the Phase 3 of the test-first plan for PR S1 (sign derivation by construction):
    • rule tables, one row per transfer rule;
    • an execution oracle that checks every known sign against values the executor actually computes;
    • exhaustiveness checks;
    • consumer tests;
    • the S2 tests, ignored.
  • Effect: 55 tests. 14 pass, 40 fail, and 1 cannot run (an S3 placeholder). The oracle's CI run finds 67,911 values whose sign bit contradicts the derived sign.

Review guide

Plan summary

The spec defines column_signs[c] as a claim about every non-NaN, non-NULL exact value of column c:

  • NonNegative: the sign bit is clear (+0.0, positives, +inf).
  • NonPositive: the sign bit is set (-0.0, negatives, -inf).
  • An Int64 value counts as v as f64, so the int 0 is N.

The invariants:

  • I1 soundness;
  • I2 an explicit rule for every variant;
  • I3 one derivation per group;
  • I4 consumers read the sign and never recompute it;
  • I5 one group, one sign;
  • I6 the sign does not depend on the evaluation context.

The test sections:

  • T1: rule tables.
  • T2: exhaustiveness (names to typed keys, and the module lint).
  • T3: execution oracle.
  • T4: regressions B1–B15 and F1–F5.
  • T5: consumers (error contract, Count-Min, DDSketch, selected plans).
  • T6: cost and the planned PhysicalASAPDAGNode.column_signs, which are PR S2 and ignored here.

Provisional answers (the user reviews them)

  • Q1 Executor fix: sums start at -0.0 and add the Kahan compensation only when it is nonzero. So Sum and Avg keep P.
  • Q2 Executor fix: sgn(±0) = x. So Sgn keeps its input's sign.
  • Q3 The executor resolves signed zeros as Go does: Max(+0,-0) = +0 and Min(+0,-0) = -0. clamp(x, +0, -0) is U.
  • Q4 Year is N only over N input.
  • Q5 Type-aware rules: an Int64 result is U wherever the float rule says P and a zero is possible.
  • Q6 NULL is excluded, like NaN.
  • Q7 The DDSketch relative gate is in S1, at the logical edge, through state_column_sign.
  • Q8 A declaration wins over the counter default. "Declared" means the family's value entry. Only the Scan rule changes for an undeclared _sum.
  • Q9 arithmetic_sign matches op first, and a source-text check catches tuple wildcards. Every FieldDataType and Option variant is listed.
  • Q10 Extra precision only for atan2, which gets l's sign, and for Case, which takes what its branches agree on.
  • Q11 A test-only helper derives a candidate expression's signs. add_candidate gets no debug assert.
  • Q12 The oracle prints its skip list. It fails if the covered-node count drops below the recorded 1683. CI runs 50 fuzz seeds × 3 variants, and the 200-seed run is #[ignore = "nightly"].

Behavior: before → after

None. This PR adds tests and todo!() stubs only, and no production path calls the stubs.

Interfaces

column_properties.rs

+/// Whether every non-NaN value has one sign bit: `NonNegative` or `NonPositive`.
+impl ValueSign {
+    pub fn is_one_sign(self) -> bool;
+}

group/mod.rs

 impl GroupProperties {
+    /// The sign of a state group's state column (its first non-plain field); `None` without one.
+    pub fn state_column_sign(&self) -> Option<ValueSign>;
 }

scalar_type_rules.rs

+/// `promql_clamp_min(value, 0)` → `Some(MathFunc::ClampMin { min: 0.0 })`; `None` for other calls or a non-literal bound.
+pub fn math_function_of_call<C>(name: &str, args: &[ScalarExpr<C>]) -> Option<MathFunc>;
+/// `promql_hour(value)` → `Some(TimeFunc::Hour)`.
+pub fn time_function_of_call(name: &str) -> Option<TimeFunc>;

test_support/forwarders.rs (test-support feature only, Q11)

+/// The column signs a candidate expression would derive on its own, without adding it to the memo.
+pub fn column_signs_of_candidate_expression(memo: &Memo, expression: &Operator<MemoGroupId>) -> Result<Vec<ValueSign>, MemoGroupError>;

Every body is todo!(). Usage: the tests in this PR are the only callers.

test_support/memo_fixtures.rs (test-support feature only)

-fn binary_op(kind: BinaryOpKind, lhs: MemoGroupId, rhs: MemoGroupId) -> Operator<MemoGroupId>;
+/// `neg > bool neg` is `binary_op(BinaryOpKind::Compare(CompareOpKind::Gt), None, true, neg, neg)`.
+pub fn binary_op(
+    kind: BinaryOpKind,
+    vector_match: Option<VectorMatch>,
+    return_bool: bool,
+    lhs: MemoGroupId,
+    rhs: MemoGroupId,
+) -> Operator<MemoGroupId>;

The rule tables build every BinaryOp through this fixture (arithmetic and set_operator call it with None, false); value_sign_rules.rs has no local builder.

Design scope

  • New test files:
    • types/tests/value_sign_rules.rs and value_sign_derivation_sites.rs: Stage 0 group properties.
    • integration-tests/tests/value_sign_{oracle,consumers,function_keys,executor_zeros}.rs: end to end, from frontend through the executor.
  • No new types.
  • Dev-dependencies metricsql_parser and strum for integration-tests. They provide the independent function-name list for T2.1.

Algorithm changes

None.

Verification

Each row was run locally with cargo test -p <crate> <filter>; ignored tests were run with --ignored.

Re-run on main 863153e (after the retarget): this head (f5d364f) and the previous head (03cd542) were run with the commands below, with --include-ignored, and compared by test name. The result is unchanged: 14 pass, 40 fail and 1 cannot run, each test with the same status as before. Two adaptations were needed for the new main:

  • crates/cost/src/node/usage/tests.rs conflicted with main's new DDSketch magnitude tests; both sets are kept.
  • main now reads a DDSketch's sign from the one source column the build folds, when its producer passes that column on unchanged. The fixture priced_state_bytes therefore gives its producer the schema [ts, value] and lists both as unchanged for a time range (a rate computes value), and sets the new value_magnitude_range: None. Without this, ddsketch_over_a_declared_raw_scan_is_unchanged failed and an_undeclared_histogram_sum_prices_two_stores passed, for the wrong reason (no sign was read at all); with it, both have their previous statuses. The cost filter ddsketch_over_ now also matches 4 of main's own tests, which pass and are not counted.

Earlier re-run on the rebased base (bc48168, on main 3e434c7): this head (03cd542) and the previous head (004cb3b on 7c0e508) were run with the commands below, with --include-ignored, and compared by test name. The result is unchanged: 14 pass, 40 fail and 1 cannot run, and the 82 panic lines are identical. The oracle numbers are the same: 1683 nodes covered, 235 skipped, 67,911 counterexample values, and the 200-seed run (4026 covered, 510 skipped) still has 1,185 values. One fix was needed for the new base: the cost test priced_state_bytes sets ProducerOutput::chain_tumbling_window_width_ms: None, a field main added.

Earlier re-run (base 7c0e508): every test was run with --include-ignored, on both this head and the previous head (22c7ef7 on d099b0b), and the outputs were compared by test name. The result is unchanged: 14 pass, 40 fail and 1 cannot run, and every panic message is the same. The oracle numbers are the same too: 1683 nodes covered, 235 skipped and 67,911 counterexample values, and the same 122 counterexample lines. The 200-seed run still has 1,185 values. One line changed, and only in the nightly skip list. Twelve topk(3, sum_over_time(m_gauge[1m])) Aggregates are still skipped, but the reason now reads aggregate intent has no native implementation instead of aggregate output names differ from measures, because the base now names and validates every aggregate measure.

Commands (from the workspace root):

cargo test -p asap-types --test types value_sign_ -- --include-ignored
cargo test -p asap-integration-tests --test integration value_sign_ -- --include-ignored
cargo test -p asap-accuracy --lib -- relative_sum_and_avg_contracts_need_one_signed_inputs error_contract_and_sign_read_one_function_key ddsketch_relative_guarantee_needs_a_one_signed_state_column b12_ddsketch_relative_guarantee_needs_one_signed_weights --include-ignored
cargo test -p asap-cost --lib -- b08_ddsketch ddsketch_stores_by_sign an_undeclared_histogram_sum cost_derives_no_sign --include-ignored
cargo test -p asap-frontend-promql --test promql -- histogram_metadata::histogram_metadata --exact --include-ignored
cargo test -p asap-planner --lib -- physical_nodes_carry_their_groups_column_signs column_signs_survive_plan_serialization --include-ignored
cargo test -p asap-stage2-physical-optimization --lib count_min_is_offered_only_over_non_negative_weights -- --include-ignored
Section Test What it catches Today
T1.1 scan_signs_follow_declarations_and_metric_types Scan signs: declarations, the counter default, histogram _sum/_count/_bucket fail: h_sum is N (B14)
T1.2 row_operators_copy_their_input_sign Filter, Sort, Limit, Dedup, TimeRange, TimeShift(offset, @ 100, @ end()), Subquery, SeriesSample, InfoEnrich, MaintainPopulation, Relabel, Values, window column, Join, SetOp, Concat pass
T1.3 vector_from_scalar_has_its_literals_sign vector(-0) is P; vector(NaN) and vector(time()) are U pass
T1.4 every_measure_has_a_sign_rule an exhaustive row for every AggIntent over N, P and U fail: 93 of 294 rows. Entropy is N; every Math(f) except Abs is U; every TimeFn is U
T1.5 arithmetic_signs_for_every_operator_and_operand_sign 7 ops × 9 sign pairs, run as vector, Float64 Project and Int64 Project fail: 52 of 189. -, %, ^ and atan2 are U; Int N*P, P*N, N/P and P/N are P (F5)
T1.6 every_math_function_has_a_sign_rule every MathFunc as promql_<f>(value, literals); SQL abs/ABS; a non-literal bound; another function fail: 68 of 127, all U where a sign is known
T1.7 every_time_function_has_a_sign_rule date functions; year is N over N only fail: 22 of 24
T1.8 every_scalar_expression_has_a_sign_rule every ScalarExpr variant fail: 7 of 55. Int negation of N gives P; a cast to Utf8 keeps the sign; a cast of P to Int64 gives P (B15); the Case rows (Q10)
T1.9 binary_op_values_and_labels filters, set operators, bool, group_right fail, new finding: a vector/vector a > bool b keeps the left side's sign. neg > bool -1 is labelled P, but its value is 0 or 1
T1.10 summary_states_and_readouts state weights, readouts, finalize per ExactKind, merges fail: the entropy readout is N
T1.11 value_sign_one_sign_and_state_column_sign is_one_sign; state_column_sign of a DDSketch build, a Scan and a chain fail: stub
T2.1 every_promql_function_name_parses_to_a_typed_key every lowered parser name maps to its exact key fail: stub. The name-set check passes: there are 35 names
T2.2 every_typed_key_is_reachable_from_a_name each key except Timestamp comes from some name fail: stub
T2.2 a_non_literal_bound_has_no_typed_key, parse_functions_reject_other_calls non-literal bounds and other names give None fail: stub
T2.3 the_sign_module_denies_wildcard_enum_arms the lint attribute, and no tuple match (…) ending in _ => fail: no attribute; tuple wildcard at arithmetic_sign
T3.1 every_known_sign_holds_on_evaluated_values oracle: 50 seeds × 3 sign variants plus 144 fixed queries, every subtree executed fail: 67,911 counterexample values (below); 1683 nodes covered, 235 skipped
T3.1 every_known_sign_holds_on_evaluated_values_over_200_fuzz_seeds (ignore = "nightly") the same over 200 seeds fail: 1,185 values, all F1
T3.2 exact_accumulator_realizations_keep_the_group_sign chain → merge → finalize over pos, neg and -0.0 panes fail: neg Sum gives +0.0 against P (F1)
T3.3 a_groups_sign_does_not_depend_on_its_queries I5 across [q1,q2], [q2,q1], [q2], [q3,q4], [q4] fail: stub. The first half passes: 31 expressions, 0 disagree
T4 b01, b03, b04, b05, b09, b10 (sign), b11 (sign), f4 regressions already fixed pass
T4 b06, b07, b14, b15, f5 N - P, entropy, h_sum, an int cast, int negation fail, as predicted
T4 b10_abs_spellings_parse_to_one_key abs, ABS and promql_abs all give MathFunc::Abs fail: stub
T4 b11_estimated_zero_divisor_needs_a_checked_division (ignore = "S3") placeholder for S3 cannot run (todo!)
T4 b12_…, b13_… DDSketch over mixed; no derivation outside the sign module fail
T4 f1_a_sum_of_negative_zeros_keeps_its_sign_bit sum_over_time, avg_over_time, sum and avg over -0.0 fail: all +0.0
T4 f2_sgn_of_negative_zero_is_negative_zero sgn(-0.0) = -0.0 fail: +0.0
T4 f3_clamp_resolves_signed_zeros_as_prometheus Go's signed zeros in clamp pass (Rust leaves this unspecified; it passes on this build)
T5.1 relative_sum_and_avg_contracts_need_one_signed_inputs relative contract iff is_one_sign, over declared and derived child signs fail: only the q(pos) - q(neg) row
T5.2 count_min_is_offered_only_over_non_negative_weights Cms and CmsWithHeap over 7 weights fail: the clamp_min(mixed, 0) row. HydraCms cannot run: it has no build parameters
T5.3 ddsketch_relative_guarantee_needs_a_one_signed_state_column a guarantee over pos/neg, none over mixed fail: mixed is granted (#807)
T5.4 error_contract_and_sign_read_one_function_key keeps_error per MathFunc fail: stub (the contract checks pass first)
T5.5 selected_sketches_satisfy_their_sign_conditions 72 workloads through the facade fail: stub. A temporary inline check found exactly B12, and no Count-Min was selected
T6 b08_…, ddsketch_over_non_positive_values_prices_one_store, an_undeclared_histogram_sum_prices_two_stores, cost_derives_no_sign, histogram_metadata, physical_nodes_carry_their_groups_column_signs, column_signs_survive_plan_serialization (ignore = "PR S2: …") S2 cost and field tests fail (run with --ignored); the two field tests end in todo!
T6 ddsketch_over_mixed_values_prices_two_stores, ddsketch_over_a_declared_raw_scan_is_unchanged unchanged pricing pass

Counterexamples the oracle found

Each line gives the query, the data, the node and column, the derived sign, and the evaluated values. Evaluation times:

  • only-zeros: the window holds only -0.0;

  • ordinary: an ordinary window;

  • non-finite: the window holds ±inf and NaN.

  • F1, a sum of -0.0 is +0.0. Data: neg declared P, 12 series. Derived P, evaluated +0.0:

    • sum_over_time(neg[5m]), Aggregate value: 12 values at only-zeros, 1 at ordinary, 1 at non-finite.
    • avg_over_time(neg[5m]): the same.
    • sum by (job) (neg), sum: 3 at only-zeros.
    • The same through review queries s11, s16, s17, s21, s36, b06, b07c, e08, e09, e11 and s24m. The averages and sums above them also give +0.0.
    • SQL SUM(-p), with p declared N: 6 at only-zeros.
    • T3.2: an exact Sum chain over neg, finalized, gives +0.0 for job j1 at 20 min, and for j0–j2 at 30 min.
    • Fuzz, non_positive variant: sum_over_time, avg_over_time, sum/max/count/quantile by over them, sum(m_gauge), and SQL SUM/AVG. Seeds 0, 1, 6, 7, 10, 12, 22, 24, 28, 29, 40, 41, 43, 44, 47.
    • The nightly run adds a Sort and a Limit under topk(3, avg_over_time(..)).
  • F5, an int zero labelled P. Data: flows.bytes Int64 declared N, bytes ∈ {0, 1, 5000}. Derived P, evaluated int 0 (non-finite / only-zeros / ordinary):

    • SELECT -bytes: 1200 / 3600 / 1200.
    • SELECT bytes * -1: 1200 / 3600 / 1200.
    • SELECT bytes / -2: 2400 / 3600 / 2400. Here 1 / -2 is also 0.
    • x03 SUM(bytes * -1): the same in its Project, plus 6 in its Aggregate and its output column.
  • B14, the _sum of a histogram with negative observations. Query rate(foo_sum[5m]), data foo_sum decreasing.

    • Scan value, derived N: 30,240 negative values, for example -8.07.
    • TimeRange value, derived N: 3,600 negative values at each time, for example -4788.3.
  • New: vector/vector bool compare. Derived P:

    • neg > bool neg: 12 values of +0.0 at each of three times.
    • neg > bool (neg * 2): +0.0 (3 / 12 / 1) and +1.0 (9 / 11).

Precision list (Unknown columns whose values all had one sign; reported, not failed): 24 entries. Examples:

  • clamp_min(neg, 0) is all clear;
  • clamp_max(pos, -0) is all set;
  • s10's subtraction;
  • x05 AVG(bytes - p).

Skip list: 235 nodes, 25 kinds that the executor cannot run or lower. Examples:

  • per-entity Rate, Increase, IRate, Deriv, Delta, StdDev;
  • Cast to Int64, which is why B15 is not observed;
  • SQL abs;
  • subqueries and @ 100;
  • without;
  • and with a matcher on the right side.

T2.3 by hand

Done in a throwaway copy, not committed. Adding #![deny(clippy::wildcard_enum_match_arm)] to column_properties.rs makes cargo clippy -p asap-types fail at four arms:

  • the Finalize _ => Unknown;
  • project_lineage's _ => None, a lineage function in the same module;
  • expression_sign's _ => Unknown;
  • measure_sign's _ => Unknown.

It does not flag the tuple match in arithmetic_sign (F11 confirmed).

Mismatches with the plan's T7 predictions

  • T1.9: predicted to pass; it fails on the new vector/vector bool finding. neg > bool -1 is labelled P.
  • T1.8: besides the predicted cast and int-negation rows, CAST(x AS VARCHAR) keeps the numeric sign (predicted U), and the three Case rows fail (Q10 precision).
  • T1.10: the plan's "SummaryMerge of N with P gives U" row cannot be built. The memo rejects a merge of two definitions (Coverage(DefinitionMismatch)), so it was dropped.
  • T1.6: the plan's "unknown name foo(value)" cannot be built, because the memo rejects unregistered functions. It is replaced by nullif(value, 1).
  • T2.1: the lowered names number 35, not 31. The plan counted only the one-argument functions.
  • T3.1:
    • B14 shows at the Scan and TimeRange, not at the rate, because the executor cannot run a per-entity Rate.
    • B15 is not observed, because the executor cannot run Cast.
    • F2, F3 and F4 produce no counterexamples today (they are U).
  • T3.3: the sign half passes as predicted; the candidate half stops at the Q11 stub.
  • T4 f3: predicted "unspecified; may fail"; it passes.
  • T5.2: the HydraCms row cannot run, since it has no build parameters.
  • T5.5: confirms only B12; there was no Count-Min selection to check.
  • T6 b08: the test feeds the DDSketch directly from a per-series rate. The result depends on that shape: over a TimeRange of a subquery of c_total, today's raw-producer check would already price one store.

Decisions and open questions

  • Needs reviewer decision: the provisional answers Q1–Q12 above, and whether the oracle's 2-minute debug runtime fits CI.
  • Decided: none.

Follow-ups

Next: the implementation, after review.

@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation ir Pre-ASAP or post-ASAP intermediate representation testing Corpus, regression, differential, or benchmark tests labels Nov 15, 2026
@zzylol
zzylol force-pushed the fix/1307-relative-error-through-negation branch from d099b0b to 5923dce Compare November 15, 2026 06:40
@zzylol
zzylol force-pushed the test/sign-derivation branch from 22c7ef7 to 004cb3b Compare November 15, 2026 15:32
@zzylol
zzylol force-pushed the fix/1307-relative-error-through-negation branch from 7c0e508 to bc48168 Compare November 16, 2026 00:57
@zzylol
zzylol force-pushed the test/sign-derivation branch from 004cb3b to 03cd542 Compare November 16, 2026 01:54
Tests for sign derivation by construction, written before the
implementation. They fail today where the rules or the executor are
unsound or incomplete.

- types/tests/value_sign_rules.rs: one row per transfer rule (Scan,
  row operators, measures, arithmetic for Float64 and Int64, math and
  date functions, scalar expressions, binary operators, summary states,
  readouts and finalize), plus the regression rows and a source check
  for the module lint.
- integration-tests value_sign_oracle.rs: every subtree of the fuzz and
  adversarial queries is planned and executed exactly, and every known
  sign is checked against the evaluated values.
- integration-tests value_sign_function_keys.rs: every PromQL math or
  date function name from the parser parses to one typed key.
- integration-tests value_sign_executor_zeros.rs: signed zeros through
  sum, sgn and clamp.
- Consumer tests: error contract, Count-Min in Stage 2, the DDSketch
  relative guarantee, and the selected-sketch check. S2 cost tests are
  ignored.

Non-test code only declares the functions the tests call, with todo!()
bodies: ValueSign::is_one_sign, GroupProperties::state_column_sign,
math_function_of_call, time_function_of_call and the test-only
column_signs_of_candidate_expression.

The test-support fixture memo_fixtures::binary_op is public and takes
vector_match and return_bool, so the rule tables build every BinaryOp
through it instead of a local copy.
@zzylol
zzylol force-pushed the test/sign-derivation branch from 03cd542 to f5d364f Compare November 16, 2026 04:46
@zzylol
zzylol changed the base branch from fix/1307-relative-error-through-negation to main November 16, 2026 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation ir Pre-ASAP or post-ASAP intermediate representation testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant