Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 64 additions & 18 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ name: Nightly Integration Tests
# end-to-end through `vspipe | ffmpeg` for every pipeline pass and verify real
# output properties (codecs, field order, audio streams, pixel format, trimming,
# subtitles). Same 4-platform matrix and deps/whisper provisioning as ci-test.yml.
# Every bundle runs two legs: `heavy` (minus `sweep`) and the generated format
# sweep (`sweep`, integration_format_sweep_test.dart, task #47), whose
# per-row results are uploaded as an artifact.

on:
schedule:
Expand Down Expand Up @@ -47,17 +50,18 @@ jobs:
# x64 runs both CPU-tier bundles (issue #92). The runner is a v3 CPU, so
# the v2 run proves the v2 bundle is functionally equivalent — not that
# it runs on an older CPU; that is build-deps-macos.yml's static gate.
# Each bundle runs twice, side by side: the heavy suite and the
# generated format sweep (task #47, tag `sweep`), so neither lengthens
# the other.
include:
- arch: arm64
runner: macos-15
asset: macos-arm64
- arch: x64
runner: macos-15-intel # last native Intel image, available until ~Aug 2027
asset: macos-x64
- arch: x64
runner: macos-15-intel
asset: macos-x64-v2
name: macos (${{ matrix.asset }})
- {arch: arm64, runner: macos-15, asset: macos-arm64, suite: heavy}
- {arch: arm64, runner: macos-15, asset: macos-arm64, suite: sweep}
# macos-15-intel: last native Intel image, available until ~Aug 2027
- {arch: x64, runner: macos-15-intel, asset: macos-x64, suite: heavy}
- {arch: x64, runner: macos-15-intel, asset: macos-x64, suite: sweep}
- {arch: x64, runner: macos-15-intel, asset: macos-x64-v2, suite: heavy}
- {arch: x64, runner: macos-15-intel, asset: macos-x64-v2, suite: sweep}
name: macos (${{ matrix.asset }}, ${{ matrix.suite }})
runs-on: ${{ matrix.runner }}
env:
VAPOURBOX_DEPS_DIR: ${{ github.workspace }}/deps/macos-${{ matrix.arch }}
Expand Down Expand Up @@ -122,7 +126,7 @@ jobs:
flutter pub get
dart run build_runner build --delete-conflicting-outputs

- name: Heavy integration tests
- name: Heavy integration tests (${{ matrix.suite }})
# ToolLocator resolves the worker relative to the running executable,
# which under `flutter test` is the test runner, not the app bundle. The
# tests that drive WorkerManager for real therefore need this pointed at
Expand All @@ -131,7 +135,20 @@ jobs:
env:
VAPOURBOX_WORKER: ${{ github.workspace }}/worker/target/debug/vapourbox-worker
# Not @Tags(['gpu']): the hosted runners have no GPU (dart_test.yaml).
run: cd app && flutter test --tags heavy --exclude-tags gpu
# `sweep` tests are also `heavy`; each leg takes its own.
run: >-
cd app && flutter test
${{ matrix.suite == 'sweep' && '--tags sweep --exclude-tags gpu' || '--tags heavy --exclude-tags gpu,sweep' }}

- name: Format sweep results
# Every row's outcome, time and differences (integration_format_sweep_test.dart).
if: ${{ always() && matrix.suite == 'sweep' }}
uses: actions/upload-artifact@v5
with:
name: format-sweep-${{ matrix.asset }}
path: Tests/TestOutput/format_sweep/results-*.json
if-no-files-found: ignore


- name: Crash reports (on failure)
# A plugin that segfaults in vspipe shows only "signal 11" in the test
Expand All @@ -146,7 +163,9 @@ jobs:
fail-fast: false
matrix:
asset: [windows-x64, windows-x64-v2]
name: windows (${{ matrix.asset }})
# The heavy suite and the format sweep (task #47), side by side.
suite: [heavy, sweep]
name: windows (${{ matrix.asset }}, ${{ matrix.suite }})
runs-on: windows-latest
env:
VAPOURBOX_DEPS_DIR: ${{ github.workspace }}/deps/windows-x64
Expand Down Expand Up @@ -209,13 +228,26 @@ jobs:
flutter pub get
dart run build_runner build --delete-conflicting-outputs

- name: Heavy integration tests
- name: Heavy integration tests (${{ matrix.suite }})
# See the macOS job: ToolLocator needs the worker path under
# `flutter test`, where the running executable is the test runner.
env:
VAPOURBOX_WORKER: ${{ github.workspace }}\worker\target\debug\vapourbox-worker.exe
# Not @Tags(['gpu']): the hosted runners have no GPU (dart_test.yaml).
run: cd app && flutter test --tags heavy --exclude-tags gpu
# `sweep` tests are also `heavy`; each leg takes its own.
run: >-
cd app && flutter test
${{ matrix.suite == 'sweep' && '--tags sweep --exclude-tags gpu' || '--tags heavy --exclude-tags gpu,sweep' }}

- name: Format sweep results
# Every row's outcome, time and differences (integration_format_sweep_test.dart).
if: ${{ always() && matrix.suite == 'sweep' }}
uses: actions/upload-artifact@v5
with:
name: format-sweep-${{ matrix.asset }}
path: Tests/TestOutput/format_sweep/results-*.json
if-no-files-found: ignore


linux:
if: ${{ github.event_name == 'schedule' || inputs.platform == 'all' || inputs.platform == 'linux' }}
Expand All @@ -224,7 +256,9 @@ jobs:
fail-fast: false
matrix:
asset: [linux-x64, linux-x64-v2]
name: linux (${{ matrix.asset }})
# The heavy suite and the format sweep (task #47), side by side.
suite: [heavy, sweep]
name: linux (${{ matrix.asset }}, ${{ matrix.suite }})
runs-on: ubuntu-24.04 # must match the deps-build runner (glibc)
env:
VAPOURBOX_DEPS_DIR: ${{ github.workspace }}/deps/linux-x64
Expand Down Expand Up @@ -294,7 +328,7 @@ jobs:
flutter pub get
dart run build_runner build --delete-conflicting-outputs

- name: Heavy integration tests
- name: Heavy integration tests (${{ matrix.suite }})
# ToolLocator resolves the worker relative to the running executable,
# which under `flutter test` is the test runner, not the app bundle. The
# tests that drive WorkerManager for real therefore need this pointed at
Expand All @@ -303,4 +337,16 @@ jobs:
env:
VAPOURBOX_WORKER: ${{ github.workspace }}/worker/target/debug/vapourbox-worker
# Not @Tags(['gpu']): the hosted runners have no GPU (dart_test.yaml).
run: cd app && flutter test --tags heavy --exclude-tags gpu
# `sweep` tests are also `heavy`; each leg takes its own.
run: >-
cd app && flutter test
${{ matrix.suite == 'sweep' && '--tags sweep --exclude-tags gpu' || '--tags heavy --exclude-tags gpu,sweep' }}

- name: Format sweep results
# Every row's outcome, time and differences (integration_format_sweep_test.dart).
if: ${{ always() && matrix.suite == 'sweep' }}
uses: actions/upload-artifact@v5
with:
name: format-sweep-${{ matrix.asset }}
path: Tests/TestOutput/format_sweep/results-*.json
if-no-files-found: ignore
56 changes: 46 additions & 10 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -447,6 +447,14 @@ Adding a filter touches many files. Missing any step causes silent failures (fil
(assert the new pass's call + key params appear in the generated `.vpy`), and a
full-encode smoke test for the new pass in `integration_new_passes_test.dart`
(or another `@Tags(['heavy'])` file). These now run in CI.
- The generated format sweep (`integration_format_sweep_test.dart`, see
"High Bit Depth Sources") picks a new pass, method or `implementation`
entry up by itself: update its count in `expectedVariantCounts`
(`format_sweep_cases_test.dart`), classify a new `PassType` in
`sweepClassOf`, add an `actuationOverrides` entry only if the pass is inert
at its defaults (the sweep fails "changed nothing" otherwise), and a
`deltaOnly` row if it wraps a low-depth plugin (`WorkDepth`) or promises
untouched planes.

**10. Plugin Binaries**
- Add a download/build block to each `Scripts/download-deps-*` script (deps
Expand Down Expand Up @@ -1511,11 +1519,34 @@ A >8-bit source is read natively off the pipe. Two ways this breaks after that:
**Test this class of bug with a reference, not an expected value** — a
correctly-scaled filter produces the same picture at 10-bit as at 8-bit, to
within rounding. `app/test/integration_high_bit_depth_filters_test.dart` (heavy)
runs every pass on identical 8/10-bit fixtures and diffs the preview frames; it
also runs 9/12/14/16-bit to check the depth-rejection guards, and compares a
9-bit source with the 10-bit one and a 14-bit source with the 16-bit one they
are decoded as. A >8-bit preview PNG
comes out `rgb48be`, not `rgb24` — that's expected, don't force 8-bit.
compares every pass at 8 and 10-bit with hand-tuned parameters at 352x288,
and compares a 9-bit source with the 10-bit one and a 14-bit source with the
16-bit one they are decoded as. A >8-bit preview PNG comes out `rgb48be`, not
`rgb24` — that's expected, don't force 8-bit.

**The generated format sweep (task #47) is what checks every pass runs on
every format.** `app/test/support/format_sweep.dart` builds the cases from
the code, never from a list: every `PassType` (classified by a switch with
no `default`), every schema `methods` entry or `implementation` `activeWhen`,
three enum axes (QTGMC presets, rotation, pitch mode), through the app's own
`ParameterConverter`; × depth {8, 9, 10, 12, 14, 16} × {4:2:0, 4:2:2, 4:4:4}
plus every other `NATIVE_FORMATS` entry (parsed from `pixel_format.rs`) ×
{progressive, TFF} × {96x64, 98x64 (chroma width odd), 720x480}. Fixtures are
one degraded 8-bit master per size, the deeper depths derived from it by a
chain with real sub-8-bit content (`chainSample`). Per row
`integration_format_sweep_test.dart` checks the preview **runs**, keeps the
source's **depth** (PNG bit depth), the variant **acts** on some row, and
(nightly) 8-bit vs deeper **parity** above the passthrough's own difference
(swscale converts 8-bit 4:2:0 to RGB without chroma interpolation: ~4.5
levels at passthrough). Passes declared in `deltaOnly` also get an FFV1
encode whose untouched samples must be bit-exact. A failing row is either
fixed or listed in `knownFailures` with the text it must fail with and its
task; a listed row that starts passing fails the test.
`format_sweep_cases_test.dart` (push gate, no worker) pins the variant count
per filter, each `*Method` enum against its schema both ways, and the hand
tables' keys. Rows run in parallel (`VAPOURBOX_SWEEP_JOBS`, default half the
cores) with the cache capped at 512 MB a job: uncapped, 7 concurrent SD
previews took 26.5 s each instead of 1.4.

### Output Colour Format

Expand Down Expand Up @@ -2948,9 +2979,13 @@ Headless Dart-VM tests. Three groups:
end-to-end via `app/test/support/worker_harness.dart`. The encode-heavy ones
are `@Tags(['heavy'])`: the **push gate** runs `--exclude-tags heavy,gpu`
(script-only: `integration_filter_parameters_test`,
`integration_qtgmc_parameters_test`); **nightly.yml** runs `--tags heavy
--exclude-tags gpu`. Tests that need a Vulkan device are tagged `gpu` and
run on no CI runner (see "GPU Plugins: RIFE and AI Upscaling").
`integration_qtgmc_parameters_test`, and the format sweep's untagged
rotating rows, ~500 tiny previews); **nightly.yml** runs two legs per
bundle: `--tags heavy --exclude-tags gpu,sweep` and the format sweep's full
product, `--tags sweep --exclude-tags gpu` (every `sweep` test is also
`heavy`; its per-row results are uploaded as an artifact). Tests that need
a Vulkan device are tagged `gpu` and run on no CI runner (see "GPU
Plugins: RIFE and AI Upscaling").
`grep -l "Tags(\['heavy'\])" app/test/*.dart` is the authority for which files
those are.

Expand Down Expand Up @@ -3002,8 +3037,9 @@ including the subtitle integration test, excluding the heavy full-encode
integration tests. Matrix: macOS **arm64** (`macos-15`), macOS **x64**
(`macos-15-intel`), **Windows x64**, **Linux x64** (`ubuntu-24.04`). The heavy
full-encode integration tests run separately in `.github/workflows/nightly.yml`
(cron + `workflow_dispatch`) via `flutter test --tags heavy --exclude-tags gpu` on the same
4-platform matrix, running each x64 platform against **both** CPU-tier bundles.
(cron + `workflow_dispatch`) via `flutter test --tags heavy --exclude-tags gpu,sweep` on the same
4-platform matrix, running each x64 platform against **both** CPU-tier bundles,
with the format sweep (`--tags sweep`) as a parallel leg of each.
Fixtures (`small_clip.mp4`, telecine/interlaced clips) are
committed under `Tests/TestResources/`.

Expand Down
5 changes: 5 additions & 0 deletions app/dart_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,8 @@ tags:
# Full-encode integration tests — run nightly, excluded from the push gate.
gpu:
# Needs a Vulkan device (RIFE, AI upscaling) — excluded on the GPU-less CI runners.
sweep:
# The generated format sweep's full product (integration_format_sweep_test.dart,
# task #47). Every sweep test is also `heavy`; nightly runs it as its own
# job (`--tags sweep --exclude-tags gpu`) and the heavy job excludes it.
# Its push-gate rows are untagged.
Loading
Loading