Repository navigation
refactor(types, frontends): column lineage replaces table qualifiers (#1207 PR 2b) - #1233
Merged
Merged
Conversation
zzylol
force-pushed
the
refactor/1207-memo
branch
3 times, most recently
from
November 9, 2026 04:40
fe0ff33 to
8095a51
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 9, 2026 06:16
e2b5456 to
58549f2
Compare
zzylol
force-pushed
the
refactor/1207-memo
branch
from
November 9, 2026 16:33
8095a51 to
49a145c
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 9, 2026 16:50
58549f2 to
b3655e1
Compare
zzylol
force-pushed
the
refactor/1207-memo
branch
from
November 10, 2026 01:26
49a145c to
43f81fe
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 10, 2026 01:26
b3655e1 to
0479df3
Compare
zzylol
force-pushed
the
refactor/1207-memo
branch
from
November 10, 2026 13:37
43f81fe to
36312cb
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 10, 2026 13:37
0479df3 to
50686ae
Compare
zzylol
force-pushed
the
refactor/1207-memo
branch
from
November 10, 2026 16:28
36312cb to
77fc121
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 10, 2026 21:37
50686ae to
a6eca31
Compare
…1207 PR 2b) GroupProperties gains column_lineage (one ColumnLineage per output column, derived once from the input groups'). Lineage and column_signs are derived together in stages/logical/column_properties.rs by main's one per-operator column layout (aggregate_output / summary_agg_layout and OutputColumnSource): a copied column keeps its lineage and sign, a computed one is Computed with the operator's sign rule. Chains compute window_end and their state and keep their keys' lineage; a TimeRange or TimeShift over a chain, and TimeShift { at: End | Start } above a selector, pass lineage through; a merge of a chain's selected windows has the chain build's layout; a label of `or` or group_right keeps what both sides agree on. SourceColumn { source: Source, column } is in ir/operator/arguments.rs beside Source. ColumnLineage is a memo-scope property beside ValueSign in stages/logical/column_properties.rs; the memo's coverage derivation reads each input group's stored lineage through MemoGroupAsInput, with no OperatorInput accessor, and carries it up through the liftable operators above the aggregate's input (lineage_at_top). Aggregate coverage names a selection's column by its SourceColumn lineage; ColumnIdentity is deleted. A predicate on a Computed column, or on one of two columns of one lineage, is not lifted; the regrouping key check compares lineage. Schema fields and the IR Project lose their table qualifier; the front ends bind qualified names with their own scope while lowering. Declared signs stay keyed by source name and column (SourceFacts).
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 10, 2026 22:08
a6eca31 to
03163fd
Compare
zzylol
marked this pull request as ready for review
November 11, 2026 02:28
…join scans, one-side match labels
- ColumnLineage::Computed { group, column }: a computed column is named by
the memo group and output column that computed it, so two computed
columns are equal only when they are the same computation. The new
group's id is passed to column_lineage_and_signs, derive_group_properties,
validate_expression_over_input_groups and aggregate_coverage. A
regrouping merge's state is computed by the merge.
- SourceColumn gains scan_ordinal: a Scan is 0; at a Join and a vector-match
BinaryOp the right side's columns of a source the left also scans shift
past the left's ordinals, so a self-join's two sides are different keys.
SourceColumn moves below impl Source and drops Hash; ColumnLineage drops
Hash and serde.
- project_lineage: one Project lineage rule for project_columns and
lineage_at_top.
- BinaryOp: labels named in group_left(...) take the right side's lineage
and sign, in group_right(...) the left side's, overwriting a column of
that name in place.
- Front-end resolver: a qualified name never binds a column qualified by
another table; Bound::new asserts one qualifier per column or none, and
an aggregate whose measures add label columns is made unqualified at its
call site.
- Tests: self-join scan ordinals and alias-renamed key; three match-grouping
label tests; qualified binding unit test. Design doc: scan_ordinal,
Computed { group, column }, SourceColumn location, plain concepts-doc
reference.
…ts scan ordinals
A BinaryOp does not concatenate both sides' columns: its labels are the
left side's for one-to-one and group_left, the right ("many") side's for
group_right, and a matched label has equal values on both sides, so it
takes the output side's lineage and sign (`or` keeps what both sides agree
on). No scan ordinal shifts at a BinaryOp; only a Join's right side shifts.
Labels named in group_left/group_right still come from the one side.
Tests: rate(x[5m]) / rate(x[1h]) keeps job as x scan 0; a same-source
group_right takes the right side's labels (failed before this change).
…re computed, Option lineage in coverage, resolver qualifiers per column - The memo group id reaches only column_lineage_and_signs (via derive_group_properties); project_lineage returns Option lineage (None: computed here), which lineage_at_top and source_column read, so aggregate_coverage, validate_expression_over_input_groups and the coverage helpers lose the group parameter. - ColumnProperties: Default (computed, sign Unknown) and all_computed replace the repeated computed(Unknown) forms; Debug/PartialEq dropped; ColumnLineage drops Eq. - BinaryOp `or`: a label only the right side has is computed (null on the left side's rows). - regrouped key check reads with matches!. - Deleted: resolve_column_refs (unused) and the planner's SourceColumn re-export (unused). - Front-end resolver: Bound stores one qualifier per column (None when unqualified); the padding closure, the window-function check and the empty-list sentinel go; one Scope::qualifier accessor; pass-through operators move the child node. An aggregate's qualifiers are sized by its node's schema. - Tests: memo_fixtures::series_column / table_column replace the local lineage helpers; latency_series and series use memo_fixtures::scan; new test an_or_label_only_the_right_side_has_is_computed. - Docs: drop the stale Field.table and Project qualifier from pre-asap-ir and operator-sharing; SourceColumn doc no longer names crate::stages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Field::table) that every operator rule had to keep right; aggregate coverage named a column by(table, name)(ColumnIdentity), so a rename made one source column look like two, andMemo::add_candidate's exact schema compare rejected a readout renamed to an aggregate's column because its qualifier differed.GroupProperties::column_lineage(each output column'sSourceColumn { source, scan_ordinal, column }, orComputed { group, column }, the memo group and column that computed it) replaces the qualifier, derived withcolumn_signsfrom main's one per-operator column layout, a memo-scope property besideValueSign; the memo's coverage derivation reads each input group's stored lineage and keys selections bySourceColumn; fields losetable, the IRProjectlosesqualifier; front ends bindf.srcwith their own scope while they lower.SELECT f.src, approx_distinct(f.dst) FROM flows f WHERE f.proto = 'tcp' GROUP BY f.src, the aggregate, the HLL state and its readout each have lineage[flows.src, Computed by that group at column 1], the state's selection is{ flows.proto ∈ {tcp} }, and the readout renamed to the aggregate's columns is accepted as the aggregate group's candidate (rejected before by qualifier).Review guide
mainat99356a314(feat(stage0, types): memo hash-consing, group properties (column_sign, depends_on_evaluation_time, aggregate_coverage), @ end()/@ start() above their subtree (#1207 PR 2) #1230, refactor(types): name the memo's view-coverage file for what it holds, drop two coverage forwarders, fix stale coverage docs (#1207) #1272 and refactor(types): aggregate coverage is a memo-group property: types and derivation in stages, no IR generic (#1207) #1273, aggregate coverage as a memo-group property, merged); refactor(stage1-pass1): per-computation candidates over memo groups (#1207 PR 3) #1232 (PR 3) builds on it. Four commits, heada6c1ffe51:03163fd1e(the previous PR 2b re-expressed on main's interfaces and layout),152220ff6(review fixes: computed-lineage identity, self-join scan ordinals, oneProjectlineage rule, match-grouping labels from the one side, strict qualified binding) theBinaryOpoutput-side fix158d4d5f3(no scan shift at aBinaryOp) and review round 2 (group id only where computed,Optionlineage in coverage, resolver qualifiers per column, fixtures, dead code). Earlier heads50686ae0d,a6eca314b,03163fd1e,152220ff6and158d4d5f3, backupsbackup/1233-lineage-<sha>for each. Diff vs base: 71 files, +1,884 / −711.types/src/ir/operator/arguments.rs(SourceColumn).types/src/stages/logical/column_properties.rs:ColumnLineage,column_lineage_and_signs,ColumnProperties,after_left_scans,project_lineage, theBinaryOparm,columns_from_sources;types/src/stages/logical/group.rs:GroupProperties::column_lineage,derive_group_properties;memo/mod.rs:find_or_add_group(the new group's id before derivation).types/src/stages/logical/aggregate_coverage.rs:lineage_at_top,source_column,residual,regrouped,build_input;types/src/ir/properties/selection.rs(SelectionBoxkeyed bySourceColumn).frontend-common/src/resolve.rs:Scope,Bound, qualifier rules.types/tests/column_lineage.rs.table: None/qualifier: Nonefrom everyFieldandProjectliteral (executor, frontends, devtools, tests);topk_output_namereplacing the copies of thetopk_{k}name rule; design-doc updates.BinaryOplabel rules; scan-ordinal shift atJoinonly;BinaryOplabels from the output side; a regrouping merge's state computed by the merge); which conjuncts coverage lifts (only aSourceColumnlineage no other input column shares);lineage_at_top(the aggregate input's lineage carried up the liftable chain); the resolver's qualifier propagation; deleted requalification innode.rs.Behavior: before → after
The SQL query above, through Stage 0 and the Pass 1 candidate it needs (
an_hll_readout_renamed_to_the_aggregate_column_is_its_candidate). It serves the design need of the concepts doc, section 3, "Column lineage": a column's identity across groups that survives renames and keeps a join's tworegions apart.Field::tableper field, kept by every rule plus two patchescolumn_lineageper group:[flows.src, flows.dst, flows.proto, flows.ts](scan 0) for the scan,[flows.src, Computed { group, column: 1 }]for aggregate, state and readout, each its own groupFROM flows a JOIN flows b … WHERE a.proto = 'tcp'(a, proto): keyed by alias(flows, 0, proto);b.protois(flows, 1, proto); renaming the aliases keeps the keya * on(job) group_left(team) b,awith its ownteamteamkeepsa.teamteamisb.teamwithb's sign (PromQL overwrites it);group_right(team)takesa.teamSELECT source … FROM (SELECT f.src AS source FROM flows f)vsGROUP BY f.src(none, source)vs(f, src): two columnsflows.srcrate(m[5m]) > 0)value ∈ (0, ∞)unlifted_operatorsadd_candidaterejects it (qualifier differs)"table""table"key; older plan files still deserializeInterfaces
Against main
99356a314.crates/types/src/ir/operator/arguments.rs·crates/types/src/stages/logical/column_properties.rs·crates/types/src/stages/logical/group.rs·crates/types/src/ir/properties/selection.rs·crates/types/src/ir/schema/mod.rs·crates/types/src/ir/operator/non_asap.rs·crates/planner/src/lib.rsOperatorInputis unchanged: no lineage accessor. Front ends:UnresolvedOp::Scan::qualifier: Option<String>and the resolver's privateScope/Bound(frontend-common);resolve_column_refignores a qualifier (ColumnRef::Qualified { name, .. }resolves by name; the front end binds the qualifier first).Scope::resolvebindst.cto the column qualifiedt, else to the uniquecwhose qualifier is not known, never to acqualified by another table;Boundstores one qualifier per column (Nonewhen unqualified; no empty-list sentinel) andBound::newdebug_assert!s it; an aggregate whose measures add label columns (count_values,absent) or a top-k is unqualified at its call site, sized by its node's schema.Algorithm changes
One column layout for lineage and sign
column_signs; a second layout could disagree.column_lineage_and_signsmatches the operator once and returns aColumnProperties { lineage, sign }per output column, then splits it intocolumn_lineageandcolumn_signs. It reuses main's layout:aggregate_outputandsummary_agg_layoutgive each output column'sOutputColumnSource(InputColumncopied with its lineage and sign;Measurecomputed withmeasure_sign;AddedLabelcomputed,Unknown). AScangivesSourceColumn { source: its Source, scan_ordinal: 0, column: field name }with its declared sign (still keyed by source name and column,SourceFacts);Filter,Sort,Limit,Dedup,TimeRange,TimeShiftand the row-selecting PromQL operators pass their input's columns; aProjectcolumn item copies its input column, an expression item isComputedby the group withexpression_sign(project_lineage, also used bylineage_at_top); a chain is its build's layout without the build's time column, after a computedwindow_end; theSummaryMergeof a chain's selected windows has the chain build's layout; a readout copies its state's plain columns; aJoinconcatenates, shifting the right side's scan ordinals of a source the left also scans past the left's (after_left_scans;Semi/Antikeep the left only);SetOp,Concatand aSummaryMergekeep what their inputs agree on, a column they disagree on is computed by the group (a regroupingSummaryMerge's state is always computed by it); aBinaryOptakes its labels from the output side, the left for one-to-one andgroup_left, the right ("many") side forgroup_right(a matched label has equal values on both sides, so the output side's lineage and sign), and fororwhat both sides agree on; it computes the value with main's sign rules (arithmetic_sign, …), copies the labels the match adds from the right side, and gives a labelgroup_left(...)names the right ("one") side's lineage and sign, onegroup_right(...)names the left side's, overwriting a column of that name in place. ABinaryOpdoes not concatenate both sides' columns, so it shifts no scan ordinal: inrate(x[5m]) / rate(x[1h])jobstays(x, 0, job). A column the group computes isComputed { group: its id, column: its output index }:find_or_add_grouppasses the new group's id (MemoGroupId(groups.len())) toderive_group_properties.debug_assert_eq!); two columns have equal lineage only when they hold the same values (the same column of the same scan, or the same computation); main's sign tests (memo_group_properties.rs, Stage 0'sa_scan_reads_the_declared_column_signs) pass unchanged.Aggregate coverage: which conjuncts are lifted
(table, name); a rename made one source column two, and a conjunct on a computed column (a rate value) was lifted although no source column names it.residual). 2. Before: lift one when it is a value set or interval on one column whose(table, name)is unique in the aggregate's input; after: whose lineage is aSourceColumnunique among the input's columns (source_column). 3. Key the selection by that source column. The aggregate input's lineage is the storedcolumn_lineageof the group below the liftable chain (read throughMemoGroupAsInput), carried up through the chain (lineage_at_top:Filter/TimeRange/TimeShiftkeep it, aProjectfollowsproject_lineage, its expression itemsNone: computed, never a selection key), so a rule's temporary build needs no lineage of its own.regroupedmaps each key to the build input's column of equal lineage (a computed key matches only the same group and column); the timestamp column aDisjointInTimemerge and a window selection constrain is named by its lineage.regions, a self-join's two scans). AComputedcolumn stays in the definition. Tests:types/tests/column_lineage.rs,memo_aggregate_coverage.rs.rate(m[5m]) > 0no longer liftsvalue ∈ (0, ∞). A top-k readout's and a population evaluation's columns areComputed.Front-end binding of qualified names
resolvereturns a privateBound { node, qualifiers }. Qualified references resolve in aScope { schema, qualifiers }: the column with that qualifier and name, else the unique column of that name whose qualifier is not known; a column qualified by another table never matches (before: any unique name). AScanhas its own qualifier (the SQL frontend setsUnresolvedOp::Scan::qualifierto the table or alias), a pass-through operator its input's, aProjectits alias or none, an aggregate'sbykeys their input's, aJoinleft then right, aSetOpthe left side's; PromQL operators have none.a_qualified_group_key_binds_its_join_side(frontend-sql),a_qualified_name_never_binds_another_tables_column(frontend-common).Decisions and open questions
Needs reviewer decision: None.
Decided:
ColumnLineage::Computed { group: MemoGroupId, column: ColumnId }; a passed-through column keeps its identity, a column the group computes (or its union-like inputs disagree on) isComputedby the group at its output index; the regroup key check stays an equality. Onlycolumn_lineage_and_signs(andderive_group_properties, which calls it) takes the group's id; elsewhere a computed column isNone(project_lineage,lineage_at_top,source_column), which no selection names.SourceColumn { source, scan_ordinal, column }; aScanis 0; only at aJoin(column concatenation) do the right side's source columns of a source the left also has shift by the left's largest ordinal for it plus one; union-like operators andBinaryOpdo not shift.BinaryOplabels come from the output side: the left for one-to-one andgroup_left, the right ("many") side forgroup_right; matched labels have equal values on both sides, so the output side's lineage and sign.orkeeps what both sides agree on. The namesakes check stays (Project [a AS k, a AS k2]).group_left(labels)takes them from the right side,group_right(labels)from the left, lineage and sign, overwriting an existing column of that name in place. Supersedes the R4 entry's agreed-label rule for named labels.Projectlineage rule (project_lineage) forproject_columnsandlineage_at_top;ColumnLineagedropsHash/serde,SourceColumndropsHash; the two parallelVecs onGroupPropertiesstay; a qualified name never binds another table's column;Bound::newasserts its qualifier count.column_lineage_and_signs;Optionlineage in coverage;ColumnProperties: Default;ormakes a right-only label computed; resolver stores a qualifier per column; sharedseries_column/table_columnfixtures;resolve_column_refsand the planner'sSourceColumnre-export deleted (unused); stale qualifier docs removed.OperatorInputaccessor. Aggregate coverage is a memo-group property (refactor(types): aggregate coverage is a memo-group property: types and derivation in stages, no IR generic (#1207) #1273), soColumnLineagelives besideValueSigninstages/logical/column_properties.rs(ir/properties/column_lineage.rsis not added) and the memo's coverage derivation reads each input group's storedGroupProperties::column_lineagethroughMemoGroupAsInput;OperatorInput::column_lineageis not added.SourceColumn { source: Source, column }stays inir/operator/arguments.rsbesideSource; selection keys areSourceColumnandColumnIdentityis deleted;lineage_at_topstays the rule, in the memo-scope derivation. This supersedes theColumnLineageplacement and theOperatorInput::column_lineageaccessor of the entry below.SourceColumn { source: Source, column: String }inir/operator/arguments.rsbesideSource(typedSource, not a name);ColumnLineageinir/properties/column_lineage.rswithOperatorInput::column_lineage(both superseded by the entry above). The derivation stays memo-scope instages/logical/column_properties.rs, withcolumn_signs, over main's one layout (OutputColumnSource/summary_agg_layout; no second layout).GroupProperties::column_lineage: Vec<ColumnLineage>. Declared signs stay keyed by source name and column (SourceFacts), not bySourceColumn. This supersedes the 2026-11-09 entries that putColumnLineageinstagesand gaveSourceColumnaStringsource.SourceColumnlineage;ColumnIdentitydeleted; the rules of the previous PR 2b carry over (a predicate on aComputedcolumn, or on one of two columns of one lineage, is not lifted; the regrouping key check compares lineage).UnresolvedOp::Scan::qualifier: Option<String>;ResolveError::Ambiguousmessage "matches several columns";resolve_column_refignores a qualifier.BinaryOp's match-added labels carry their sign: copied from the right side's column with its lineage; foror/group_right, a left label keeps what both sides agree on. One layout copies lineage and sign together, so they cannot disagree. Example: ina * on(job) group_left(team) b, the last columnteamhas lineageb.teamandb'steamsign.lineage_at_top) instead of deriving a temporary build's lineage; for a memo group it equals the group's stored lineage. (SourceColumn'sHashwas dropped in review: nothing hashes it.) ABinaryOp's value column isComputed(also under a comparison filter), as before.Deleted besides the interfaces above: requalification in
OperatorNode::with_new_childrenandvalidate_structure(node.rs; only names may be overridden); privatecolumn_identity(→source_column) andbuild_input_schema(→build_input, the input for its schema and lineage);column_signs' sign-only helpers (signs_of,project_signs,aggregate_signs,signs_from_sources,state_signs,readout_signs, sign-onlyagreed) → theirColumnPropertiesversions; copies of thetopk_{k}output-name rule →topk_output_name; unusedresolve_column_refsand the planner facade'sSourceColumnre-export; the test files' local lineage helpers (column_of,column_of_scan,source_column,latency_column,series_column) →memo_fixtures::series_column/table_column; testsqualified_field_serde_roundtripand the qualified case ofpartitions_over_matches_keys_by_column.Verification
cargo test --workspace --no-fail-fastata6c1ffe51and at main99356a314(temporary worktree), compared by test name: main 1140 passed / 189 failed / 9 ignored, this PR 1154 / 189 / 9. 0 newly failing; the 189 failures are the same tests on both (later PRs'todo!()s). +14 passing: the 15 new tests (typescolumn_lineage: the six concepts-doc and chain tests,a_self_join_names_its_two_scans_apart,a_group_left_label_comes_from_the_right_side,a_group_left_label_overwrites_the_left_sides_column_of_that_name,a_group_right_label_comes_from_the_left_side,a_same_source_binary_op_keeps_its_labels_scan,a_same_source_group_right_takes_the_right_sides_labels,an_or_label_only_the_right_side_has_is_computed; frontend-sqla_qualified_group_key_binds_its_join_side; frontend-commona_qualified_name_never_binds_another_tables_column) minus the deletedqualified_field_serde_roundtrip.cargo fmt --all --checkandcargo clippy --workspace --all-targets -- -D warningsclean;cargo doc --no-deps -p asap-types -p asap-frontend-common -p asap-plannerreports the same six warnings as main (none in files this PR touches). Nothing undertypes/src/ir/importscrate::stages.scan_predicates_lift_and_their_states_merge(example 4,memo_aggregate_coverage.rs) also assertslatency.job.Wire format: the exported
SchemaJSON has no"table"key on fields; a selection key serializes as{source, scan_ordinal, column}withsourcetheSourceenum (only in a materialized view's coverage).view_coverage's content id hashes the expanded definition, which holds no selection keys, so lineage does not enter it; the selection is aBTreeMapserialized in key order.Design docs:
pre-asap-ir.mdandoperator-sharing.mdlose the staleField.table/Projectqualifier;asap-primitive-schema.md§2's table, §4.2.2 (worked example and "Which column a condition is on") and theField,Schema,SelectionBoxandSourceColumnlistings use lineage; §6.6 namesColumnLineageinstages/logical/column_properties.rsandSourceColumninir/operator/arguments.rs; scan ordinals andComputed { group, column }are described in §2, §4.2.2 and §6.6; the concepts-doc reference is plain text (the file is not on main).Follow-ups
add_candidateand names columns bySourceColumn.Computed;rate(m[5m]) > 0is not lifted.ColumnRefs inside the IR (an extension intent's payload columns, a SQL temporal aggregate's arguments) still resolve by unique name inresolve_column_ref; making it reject a qualified name needs the front ends to bind those through their scope first: SQL: qualified column refs in extension intents and temporal aggregates bind by name only #1280.SourceColumn { source: String }, unitComputed); it is synced (scan ordinal,Computed { group, column }) after this PR merges.