Memo example memo-ratio-of-quantiles.json is 2.9 MB: store each schema once and write the memo section compact #1033

Problem, by example

tools/dag-viewer/examples/memo-ratio-of-quantiles.json, the memo example for

avg by (job) (quantile_over_time(0.5, data[5m]) / quantile_over_time(0.5, data[5m]) + quantile_over_time(0.9, data[5m]))

is 2,895,266 bytes on main. #1031 shrank it from 13.8 MB by storing each memo node once in memo.nodes. The target in #1027 was under 1 MB, and the example still misses it.

Measured on main (f49900b):

Over a third of the compact size is the 1,099 copies of only 15 distinct schemas. The rest of the pretty-printed size is indentation.

Why

• stage_pipeline writes the document with serde_json::to_writer_pretty.
• Each entry of memo.nodes carries its full output schema, although almost every node shares its schema with many others. Typical cases: a candidate's estimate and its finalize, or the 336 PaneChain state groups.

Proposal

1. Schema table. Store each distinct schema once as memo.schemas, and have each node in memo.nodes hold schema: <index>. memoNodeDag in tools/dag-viewer/stages.js resolves the index when it rebuilds a DAG, so memo.js and enumeration.js don't change.
2. Compact JSON. Write the committed memo examples compact.
   • Alternative: pretty-print everything except memo.
   • Estimated with both: about 1,154,941 − 391,263 + ~30,000 ≈ 0.8 MB.

Only the memo section changes format. Documents without memo stay as they are.

Scope and acceptance

• The committed memo-ratio-of-quantiles.json is under 1 MB.
• A script comparison against the current example finds the same groups, candidates, group boundaries and enumerated choices. perf(devtools): store each memo node once in the stage-pipeline document #1031 used the same check.
• node --test tools/dag-viewer/*.test.mjs, python3 -m unittest test_viewer test_server and the stage_pipeline tests pass.
• A document with memo in the perf(devtools): store each memo node once in the stage-pipeline document #1031 format is rejected with the "regenerate it with stage_pipeline" message, as perf(devtools): store each memo node once in the stage-pipeline document #1031 does for the older format.

Related

• Follows stage_pipeline memo: write each node once per step instead of a full subtree per group and candidate #1027 / perf(devtools): store each memo node once in the stage-pipeline document #1031.
• Part of Tracking: DAG viewer shows the memo after each step and every candidate DAG it stands for #1008 (DAG Viewer: memo and enumerated candidate DAGs).