Skip to content

Generated format sweep: every pass × method × depth × subsampling × field × size (task #47) - #212

Merged
StuartCameronCode merged 5 commits into
mainfrom
stuart/47-generated-format-sweep-every-pass-method
Nov 16, 2026
Merged

StuartCameronCode merged 5 commits into
mainfrom
stuart/47-generated-format-sweep-every-pass-method

Conversation

@StuartCameronCode

Copy link
Copy Markdown
Owner

Adds a format sweep that is generated from the code, not from hand-written lists. It checks that every pass and method runs on every source format, field flag and frame size. This is R2a of task #42's plan. Tasks #23, #29, #31, #33, #34, #35 and #37 were each caused by one untested value on one of these axes.

What it sweeps

  • Variants (111). They are built through the app's own ParameterConverter, from:
    • every PassType, classified by a switch with no default, so a new pass doesn't compile until it is classified
    • every schema methods entry and implementation activeWhen entry
    • three enum axes: QTGMC presets, rotation, pitch mode
  • Rows:
    • depth {8, 9, 10, 12, 14, 16} × {4:2:0, 4:2:2, 4:4:4}
    • plus every other NATIVE_FORMATS entry, parsed from pixel_format.rs (4:1:1 and the yuvj formats)
    • × {progressive, TFF}
    • × {96x64, 98x64 (odd chroma width), 720x480}
  • Fixtures: one degraded 8-bit master per size, with grain, MPEG-2 blocking, scratches, spots, shake, dot crawl and rainbow. The deeper fixtures are derived from it with real sub-8-bit content (chainSample), so a precision bug can't hide behind empty low bits.

Checks per row

  1. Runs: the preview exits with a PNG. Since task docs: README macOS minimum versions (Intel 12 / Apple Silicon 15) #42 there is one template, so this covers the encode's pass code too.
  2. Depth: the PNG keeps the source's depth under Color Format "original".
  3. Acts: the variant changes the picture on at least one row, so the other checks prove something.
  4. Parity (nightly): each deeper row agrees with its 8-bit row, above the passthrough's own difference between the two.

Passes declared delta-only also get an FFV1 encode. The samples they promise to leave alone must be bit-exact: the low-depth guards (_low_depth_change, DeScratch), the luma-only and chroma-only passes, rotation, crop, Edge Repair and borders.

Where it runs

  • Push gate (untagged, ~500 tiny previews): four rotating rows per CPU variant plus one extra native format. Every variant meets every subsampling, both field flags and both tiny sizes.
  • Nightly (sweep + heavy): the full product, as its own job beside the heavy suite on every bundle. Per-row results are uploaded as an artifact.
  • Add-on and GPU variants: run only where the add-ons or a GPU are available. The GPU variants are tagged gpu, which no CI runner runs.
  • format_sweep_cases_test.dart (push gate, no worker) checks:
    • the variant count per filter
    • every *Method enum against its schema, both ways
    • that every hand-written table key names something that exists
    • that the gate rows cover every axis value

Findings

Filed together as task #56 (F1–F8):

  • width-2-mod-4 failures in QTGMC Super/Ultra Fast and in MCTemporalDenoise (an ordinary 854x480 source hits them)
  • a quarter turn keeps _FieldBased
  • Deband turns an 8-bit source's output 16-bit
  • Bifrost is non-deterministic when the width is 2 mod 4
  • CLAHE's parity at 16-bit

Each failing row is pinned in knownFailures with the error text it must fail with. A pinned row that starts passing fails the test, so the entry is removed with its fix.

Groups 1–2 of integration_high_bit_depth_filters_test.dart (six hand-picked formats) are removed because the sweep covers them. Its hand-tuned parity group stays. CLAUDE.md and ENGINEERING_NOTES are updated.

Review fixes

  • The row pool could run one row too many at once.
  • A run killed mid-write could leave a partial fixture that the next run reused. Fixtures are now written under a temporary name and renamed.
  • Entries in the "inert on this fixture" allowlist must now name a variant that exists.
  • The bit-exact group now marks a GPU variant skipped instead of passing it.

Testing (macOS arm64, deps 1.17.0-rc1)

  • flutter analyze: clean.
  • format_sweep_cases_test.dart: 33/33.
  • Implement stage:
    • full push gate and full nightly product: all rows pass or match their pinned failure
    • add-on variants: 25 rows ok
    • GPU variants (RIFE and the six AI upscalers): all pass
  • Review: gate variants re-run from a cleared fixture cache, 38/38. DeScratch's bit-exact check passes.
  • Not run: the Windows and Linux nightly legs. They run on the next nightly or a workflow_dispatch of nightly.yml.

🤖 Generated with Claude Code

@StuartCameronCode
StuartCameronCode force-pushed the stuart/47-generated-format-sweep-every-pass-method branch from 7e63700 to 2b9c7bc Compare November 16, 2026 20:39
StuartCameronCode and others added 3 commits November 17, 2026 07:42
… field × size (task #47)

Cases come from PassType, the schemas' methods and implementation entries,
three enum axes and the worker's NATIVE_FORMATS, built through the app's
ParameterConverter. Per row: the preview runs, keeps the source's depth,
the variant acts, and (nightly) 8-bit vs deeper parity above the
passthrough's own difference; delta-only passes get an FFV1 bit-exact
check. Failing rows are pinned in knownFailures (task #56).

- support/format_sweep.dart: generator, axes, fixture chain, tables
- format_sweep_cases_test.dart: push gate, no worker (counts, enum <-> schema)
- integration_format_sweep_test.dart: gate rows untagged, full product
  tagged sweep+heavy, GPU variants tagged gpu
- nightly.yml: a sweep leg per bundle, results uploaded; heavy leg
  excludes sweep
- high-bit-depth test: groups 1-2 removed (the sweep covers them)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- _Pool hands a finishing slot to the next waiter, so it can't run size+1
- fixtures written to a temp name and renamed, so a killed run can't leave
  a partial file the cache check accepts
- inertOnFixture keys must name an existing variant, not just its filter
- the delta-only group marks a GPU variant skipped instead of passing it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in in WorkDepth's doc

Since task #46 the IVTC and Soft Telecine preview shows the encode's frame
for its source frame; the fixture has no pulldown, so that is the source
frame unchanged and the 'acts' check is waived for both, as for Frame
Rate's duplicate. WorkDepth's comment names task #48's run_within, which
replaced _low_depth_change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@StuartCameronCode
StuartCameronCode force-pushed the stuart/47-generated-format-sweep-every-pass-method branch from 2b9c7bc to 3b53abc Compare November 16, 2026 20:42
StuartCameronCode and others added 2 commits November 17, 2026 07:58
… (task #47)

The Linux push gate saw vspipe die with no Python error on every row of
the four presets that alone share Overlap = blocksize/4 and SearchParam 1;
they pass on macOS arm64. Pinned Linux-only and flaky (the hosted fleet is
mixed for AVX-512), after F2 so F2's rows keep their own match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebased push gate failed the same four QTGMC presets on macOS x64 and
Windows (no AVX-512) as on Linux x64; both arm64 runs pass them. Pin on
every non-arm64 platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant