Add visual regression testing for RNTester by robhogan · Pull Request #59115 · react/react-native

robhogan · 2026-10-15T14:52:51Z

RNTester's rendering coverage, particularly text, currently lives in Meta's internal jest-e2e suite. To move it into public CI we need screenshot comparison that works for fork PRs and doesn't add PNGs to the repo. This adds that, as a separate set of visual flows alongside the existing Maestro E2E.

How it works

Visual tests are Maestro flows in packages/rn-tester/visual-tests/flows/. Each one deep-links to an RNTester example, waits for it to be ready and captures a screenshot cropped to a testID. Maestro only captures - it never decides pass or fail.

The work is split across a trust boundary, so a fork PR never runs with credentials:

Visual capture runs as two jobs in Test All, reusing its RNTester Release builds: test_visual_android_rntester after build_android and test_visual_ios_rntester after test_ios_rntester. Each runs every flow in one Maestro invocation, on a Pixel 7 API 35 emulator (1080x2400, 420dpi) or an iPhone 17 Pro on iOS 26.2, and uploads the PNGs and a manifest recording the commit and device environment as an artifact. They have no secrets.
Visual Compare RNTester (workflow_run on Test All, from main) treats that artifact as data - it's validated before any image is decoded and nothing in it is executed. It finds the baseline, compares with pixelmatch, posts a check run per platform and a single PR comment with expected, captured and diff images, and on pushes to main publishes the captures as a new baseline. It runs once Test All completes, including when unrelated jobs fail, so the report arrives with the rest of CI rather than as soon as the captures are ready.
Visual Approve RNTester (pull_request_target, on the visual-change-approved label) records a maintainer's approval of an intentional change. The approval is tied to the exact set of differences, so a later push that changes them removes the label and needs a new review.

Comparison is strict by default: any changed pixel is a difference. A test can be loosened in visual-tests/config.json, with evidence. Captures have been byte-identical across separate builds and runners so far, so I'd expect that to be rare.

What defines the baseline

A baseline is the full set of captures from a commit on main, keyed by commit and by device environment. Every push to main publishes one, and nothing is ever overwritten.

A PR is compared with the baseline for the commit it's based on (the first parent of the merge commit CI tests), or the nearest earlier commit on main that has one, and the report says which. So a PR based on last week's main is compared with last week's rendering, not with whatever has landed since. Approving a PR doesn't promote its images - main is captured again after merge and that becomes the baseline.

The environment key covers everything that affects rendering (OS runtime, device, screen size, density, font scale, locale, appearance, animations, system image and, on Android, the emulator version, which is pinned). If the CI image drifts, the key changes and a fresh baseline is published rather than every PR showing a spurious diff.

Settings

Storage is any GCS bucket, configured through repository variables. Writes use GitHub's OIDC token, so there's no stored secret, and only the two trusted workflows on main can authenticate. They need create-only access (roles/storage.objectCreator). Reads are public.

Variable
`VISUAL_BUCKET`	Bucket to upload to
`VISUAL_PUBLIC_BASE_URL`	Public URL for reads and report images - the bucket, or a CDN in front of it
`VISUAL_GCP_PROJECT`	GCP project for authentication
`VISUAL_WORKLOAD_IDENTITY_PROVIDER`	Workload identity provider trusting this repo's `visual-compare-rntester.yml` and `visual-approve-rntester.yml` on `main`

Expo can host the bucket and a CDN for this. Reports and approvals expire after 30 days by lifecycle rule. Baselines are small (8-175 KB per image), and at the current rate of pushes to main the cost is cents a month.

packages/rn-tester/visual-tests/README.md covers adding a test and handling a failure. This starts with two text tests and runs non-blocking. The plan is to port high-value jest-e2e text and rendering tests, then make the checks required once they've proven reliable.

Changelog:

[INTERNAL] [ADDED] - Visual regression testing for RNTester in CI

Test Plan:

Ran the full flow in a fork (https://github.com/robhogan/react-native), with Test All trimmed to the jobs this depends on, against a personal GCS bucket:

A PR with no visual change passes on both platforms (Visual spike: chain-no-change robhogan/react-native#3).
A deliberate text layout change fails both platforms, with expected, captured and diff images on the checks page and in the PR comment (Visual spike: chain-text-regression robhogan/react-native#4).
Applying visual-change-approved passes the approval check. Pushing a further, different change removed the label with a comment and failed the check again.
Captures were byte-identical across 5 Android and 4 iOS runs, from 3 separate builds on fresh runners.
When an unrelated Test All job fails, the comparison still runs. Run against a Test All run whose Android capture had failed, it reports Android as having no capture and compares iOS as usual.
Pixel Launcher raised an ANR dialog during emulator boot in 3 of 8 probe runs. After the capture's setup step, none of the 8 had a dialog showing.
Running all flows in one Maestro invocation brought the iOS capture step to 2m20s for two tests, from 3m49s for one test when each flow started Maestro separately. Failed flows are retried individually.
New unit tests for artifact validation (path traversal, symlinks, hash and size checks), comparison, storage keys, approval and report rendering.

Not yet tested: a PR from a third-party fork, to confirm it can't reach storage credentials.

cortinico

Thats' great - can we add some markdown in the repos to explain how to use this?

cortinico · 2026-10-15T17:03:06Z

+  VISUAL_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
+
+jobs:
+  build_android:


We already build rn_tester in previous workflows, no? Can't we reuse the APK from that jobs?

cortinico · 2026-10-15T17:03:56Z

+          api-level: 35
+          target: google_apis
+          arch: x86_64
+          ram-size: '8192M'
+          heap-size: '4096M'
+          disk-size: '10G'
+          cores: '4'
+          disable-animations: true
+          avd-name: visual_emulator


I'm a little concerned that we don't have an explicit resolution configured here and this could make the screenshots drift quite a lot once we bump this rule. Potentially we should add like profile: 'pixel' or so?

cortinico · 2026-10-16T16:08:34Z

@robhogan I'm having a harder time in reviewing this. Can you ask your agent to split it in smaller meaningful commits (doens't need to ne separate PRs), so it would be easier for me to do a pass?

Also my browser is unhappy with the diff size 😅

Adds the first two visual tests as Maestro flows under `packages/rn-tester/visual-tests/flows/`. Each flow deep-links to an RNTester example through `helpers/open-example.yml`, waits for it to render and takes a screenshot cropped to the example's `testID`. The flows only capture, and never decide whether a test passed. The last line clipping example gains a `testID` to crop to. Changelog: [Internal]

Adds two Test All jobs that run the visual flows against the RNTester Release builds Test All already makes: `test_visual_android_rntester` after `build_android` and `test_visual_ios_rntester` after `test_ios_rntester`. They call the reusable `visual-capture-rntester.yml`, which runs every flow in one Maestro invocation on one canonical device per platform (a Pixel 7 API 35 emulator and an iPhone 17 Pro on iOS 26.2), retrying only failed flows. Each job uploads the screenshots plus a manifest as a `visual-capture-<platform>` artifact. The manifest records the commit, the base commit and the device environment, hashed into an environment key so that baselines from a different device setup are never compared. The emulator version is pinned and part of the key, and the capture dismisses a Pixel Launcher ANR dialog that can appear at boot. These jobs run PR code and have no secrets. Nothing consumes the artifacts yet. Changelog: [Internal]

Adds the comparison core, as pure functions over a capture directory and a baseline. `validate.js` treats the capture artifact as untrusted: it rejects path traversal, symlinks, unlisted or duplicate files, hash mismatches and oversized images, reading PNG dimensions from the header before any image library decodes them. `compare.js` compares each image with pixelmatch under the per-test policy in `visual-tests/config.json`, which is strict by default. Adds `pixelmatch` and `pngjs` as dev dependencies. Changelog: [Internal]

Adds the remaining building blocks for the trusted workflow. `storage.js` defines the bucket layout (write-once baselines keyed by environment and commit, reports per PR head) and finds the nearest baseline in a commit's ancestry. `report.js` renders the check run output and the PR comment. `github.js` is a minimal REST client. `approval.js` fingerprints the set of differences, so an approval label only covers the differences it was applied to. Changelog: [Internal]

Adds Visual Compare RNTester, which runs from the default branch on `workflow_run` when Test All completes, including when unrelated jobs fail. It downloads each platform's capture, validates it, compares it with the baseline for the PR's base commit and uploads the report through GitHub OIDC, so no secret is stored. It posts a check run per platform, the approval gate check and a single PR comment. On pushes to `main`, it publishes the captures as a new baseline. The bucket and identity provider are configured through the `VISUAL_*` repository variables. Changelog: [Internal]

Adds Visual Approve RNTester, which runs on `pull_request_target` when a maintainer applies `visual-change-approved`. It records an approval for the current fingerprint of differences and re-evaluates the gate check. A later push with different differences removes the label and needs a new review. Changelog: [Internal]

Adds `packages/rn-tester/visual-tests/README.md` on how the tests run in CI, adding a test, running flows locally, handling a failed check and tolerances, and links it from RNTester's README. Changelog: [Internal]

robhogan · 2026-10-19T09:56:15Z

@robhogan I'm having a harder time in reviewing this. Can you ask your agent to split it in smaller meaningful commits (doens't need to ne separate PRs), so it would be easier for me to do a pass?

Also my browser is unhappy with the diff size 😅

Sorry 😅. Rolling it into test-all so it could re-use the main pipeline's builds did inflate things a bit. Split into 7 commits now.

cortinico

I've done a full pass and this is great!

cortinico · 2026-10-21T18:23:42Z

+    "threshold": 0,
+    "includeAA": true,
+    "maxDiffPixels": 0


q: Probably that's a bit too strict to start with?

Up to you but I'd keep it strict by default - in testing this I found everything to be completely deterministic and reproducible as long as we're locked to a consistent environment. If we had a threshold we'd want to put it just above observed run-to-run noise, and there's no evidence of any yet. If we see noise, we can always loosen it.

If we want to test anything with known fuzzy output I think the test author should probably expect that (and this is overridable per test). Otherwise this catches the subtle stuff we might miss by eye.

cortinico · 2026-10-21T18:32:20Z

+          project_id: ${{ vars.VISUAL_GCP_PROJECT }}
+          workload_identity_provider: ${{ vars.VISUAL_WORKLOAD_IDENTITY_PROVIDER }}


Are those already configured for the repo?

I have no access to that, will pick up in chat.

cortinico · 2026-10-21T18:36:33Z

+      - name: Record approval
+        if: steps.staged.outputs.upload == 'true'
+        run: gcloud storage cp --recursive --no-clobber "$OUT_DIR/upload/*" "gs://$VISUAL_BUCKET/"


If I understand correclty: you record the approval of the baseline here once a label is applied.

I think it's ok to go ahead with this for now, but ideally, the mutation on GCP should be done only once the commit land on main

The reason is that the label can be added/removed. Also sometimes you fat finger the wrong label, so I would be cautious on doing too much side effects on such a simple event.

The write here isn't the baseline, it's writing the metadata that the PR change is approved, and the hash of the approved visual change.

If we only had the label itself as the source of truth, we wouldn't know when a visual had changed since it was approved (because labels don't have metadata), so we'd have to remove the label on any push to the branch. That's doable but it's more friction.

This way the approval is effectively associated with the new visual, and stable over other pushes to the PR.

cortinico · 2026-10-21T18:38:01Z

+
+1. Test All builds RNTester Release for Android and iOS. Once a build is ready, the `test_visual_android_rntester` and `test_visual_ios_rntester` jobs (`.github/workflows/visual-capture-rntester.yml`) run every flow in this directory on one canonical device per platform: an API 35 Pixel 7 emulator and an iPhone 17 Pro simulator on iOS 26.2. They upload the screenshots and a manifest of the device environment as `visual-capture-android` and `visual-capture-ios` artifacts.
+2. When Test All completes, Visual Compare RNTester (`.github/workflows/visual-compare-rntester.yml`) compares the screenshots with the baseline and reports a `Visual regression (android)` and `Visual regression (ios)` check and a `Visual regression approval` gate, plus a single PR comment with the expected, captured and diff images for anything that changed.
+3. On pushes to `main`, the captures become the baseline for that commit.


I was expecting to find a workflow that does this but I couldn't find it 🤔

This is kind: 'main' under the compare workflow, rather than a separate workflow. Clarifying the readme...

meta-codesync · 2026-10-21T18:48:33Z

@cortinico has imported this pull request. If you are a Meta employee, you can view this in D126719236.

…main publishes a new baseline, so rebasing a PR invalidated its approval even when every expected and captured image was byte-identical. The expected image hash already pins what each capture is compared with, so the fingerprint now covers only the images, test IDs, statuses and environment. Changelog: [Internal]

Logs from a passing capture help explain a screenshot that rendered differently, which a passing flow doesn't otherwise flag. Changelog: [Internal]

Visual Compare ran after every Test All, including pushes to stable branches, where there's no baseline to compare with or publish. It now skips pushes to other branches before starting a runner, and skips pull requests into other branches once it has looked up the pull request, because workflow_run doesn't carry the base branch of a pull request from a fork. Changelog: [Internal]

Changelog: [Internal]

Both workflows need the repository's GCP project and workload identity provider to read and write visual results. Until those variables are configured, every run failed, so both workflows now skip instead, and the visual tests can land ahead of the storage setup. Changelog: [Internal]

Summary: Add screenshot comparison for RNTester in public CI, including fork pull requests, without committing baseline PNGs. Visual flows run alongside the existing Maestro E2E coverage. ### How it works Visual tests are Maestro flows in `packages/rn-tester/visual-tests/flows/`. Each one deep-links to an RNTester example, waits for it to be ready and captures a screenshot cropped to a `testID`. Maestro only captures - it never decides pass or fail. The work is split across a trust boundary, so a fork PR never runs with credentials: - **Visual capture** runs as two jobs in Test All, reusing its RNTester Release builds: `test_visual_android_rntester` after `build_android` and `test_visual_ios_rntester` after `test_ios_rntester`. Each runs every flow in one Maestro invocation, on a Pixel 7 API 35 emulator (1080x2400, 420dpi) or an iPhone 17 Pro on iOS 26.2, and uploads the PNGs and a manifest recording the commit and device environment as an artifact. They have no secrets. - **Visual Compare RNTester** (`workflow_run` on Test All, from `main`) treats that artifact as data - it's validated before any image is decoded and nothing in it is executed. It finds the baseline, compares with pixelmatch, posts a check run per platform and a single PR comment with expected, captured and diff images, and on pushes to `main` publishes the captures as a new baseline. It runs once Test All completes, including when unrelated jobs fail, so the report arrives with the rest of CI rather than as soon as the captures are ready. - **Visual Approve RNTester** (`pull_request_target`, on the `visual-change-approved` label) records a maintainer's approval of an intentional change. The approval is tied to the exact set of differences, so a later push that changes them removes the label and needs a new review. Comparison is strict by default: any changed pixel is a difference. A test can be loosened in `visual-tests/config.json`, with evidence. Captures have been byte-identical across separate builds and runners so far, so I'd expect that to be rare. ### What defines the baseline A baseline is the full set of captures from a commit on `main`, keyed by commit and by device environment. Every push to `main` publishes one, and nothing is ever overwritten. A PR is compared with the baseline for the commit it's based on (the first parent of the merge commit CI tests), or the nearest earlier commit on `main` that has one, and the report says which. So a PR based on last week's `main` is compared with last week's rendering, not with whatever has landed since. Approving a PR doesn't promote its images - `main` is captured again after merge and that becomes the baseline. The environment key covers everything that affects rendering (OS runtime, device, screen size, density, font scale, locale, appearance, animations, system image and, on Android, the emulator version, which is pinned). If the CI image drifts, the key changes and a fresh baseline is published rather than every PR showing a spurious diff. ### Settings Storage is any GCS bucket, configured through repository variables. Writes use GitHub's OIDC token, so there's no stored secret, and only the two trusted workflows on `main` can authenticate. They need create-only access (`roles/storage.objectCreator`). Reads are public. | Variable | | |---|---| | `VISUAL_BUCKET` | Bucket to upload to | | `VISUAL_PUBLIC_BASE_URL` | Public URL for reads and report images - the bucket, or a CDN in front of it | | `VISUAL_GCP_PROJECT` | GCP project for authentication | | `VISUAL_WORKLOAD_IDENTITY_PROVIDER` | Workload identity provider trusting this repo's `visual-compare-rntester.yml` and `visual-approve-rntester.yml` on `main` | Expo can host the bucket and a CDN for this. Reports and approvals expire after 30 days by lifecycle rule. Baselines are small (8-175 KB per image), and at the current rate of pushes to `main` the cost is cents a month. `packages/rn-tester/visual-tests/README.md` covers adding a test and handling a failure. This starts with two text tests and runs non-blocking. The plan is to expand high-value text and rendering coverage, then make the checks required once they've proven reliable. Changelog: [Internal] - Visual regression testing for RNTester in CI Pull Request resolved: react#59115 Test Plan: Ran the full flow in a fork (https://github.com/robhogan/react-native), with Test All trimmed to the jobs this depends on, against a personal GCS bucket: - A PR with no visual change passes on both platforms (robhogan#3). - A deliberate text layout change fails both platforms, with expected, captured and diff images on the checks page and in the PR comment (robhogan#4). - Applying `visual-change-approved` passes the approval check. Pushing a further, different change removed the label with a comment and failed the check again. - Captures were byte-identical across 5 Android and 4 iOS runs, from 3 separate builds on fresh runners. - When an unrelated Test All job fails, the comparison still runs. Run against a Test All run whose Android capture had failed, it reports Android as having no capture and compares iOS as usual. - Pixel Launcher raised an ANR dialog during emulator boot in 3 of 8 probe runs. After the capture's setup step, none of the 8 had a dialog showing. - Running all flows in one Maestro invocation brought the iOS capture step to 2m20s for two tests, from 3m49s for one test when each flow started Maestro separately. Failed flows are retried individually. - New unit tests for artifact validation (path traversal, symlinks, hash and size checks), comparison, storage keys, approval and report rendering. Not yet tested: a PR from a third-party fork, to confirm it can't reach storage credentials. Local validation after fixes: - `arc lint` passes for all 28 changed paths. - Workflow Node unit tests: 3 suites, 46 tests pass. - Yarn validation passes for `xplat/js`, `xplat/js/public`, and `xplat/js/react-native-github`; offline mirror verified. - Reformatted new workflow scripts with the OSS Prettier/ESLint configuration and rebased onto current stable. CI review follow-up: - Make approval-label removal idempotent when another workflow already removed it; retain errors for permission failures. - Reject invalid `VISUAL_CAPTURE_ATTEMPTS` before app installation. - `js1 test` with a focused Node workflow configuration: 3 suites, 54 tests pass, including both regression cases. - `arc lint` passes after these fixes. Additional CI follow-up: - Fix the RN Babel/regenerator failure by reading the HTTP error body before interpolating it. - Distinguish a failed GCS listing or missing listed report from a successful empty listing; storage errors preserve the approval label. - Standard `js1 test` in the `rn-oss` project: all 3 workflow suites and 57 tests pass, using the same RN transformation pipeline as CI. - OSS Prettier and `arc lint` pass. Dependency alignment follow-up: - Align `arvr/js` pngjs with the new xplat dependency and regenerate its lockfile and checked-in package. - Existing `yarn test -p xplat-js-versions-aligned-test` gate: 186 tests pass. - `arc lint` passes for the dependency changes. - Yarn validator verifies the offline mirror; `arvr/js` is excluded by the validator configuration, so the ARVR alignment gate above provides its package-sync check. Capture environment follow-up: - Reject invalid or missing required environment properties before constructing a baseline key, including zero/NaN API levels and densities; retain optional native-bridge/emulator metadata. - Standard `js1 test` in `rn-oss`: all 3 workflow suites and 73 tests pass, including Android/iOS environment regression cases. - OSS Prettier and `arc lint` pass. Review disposition: `latestResults` intentionally stops approval when the newest listed report has no `results.json`. A partial newer upload must not authorize an older fingerprint. A successful empty listing is the only missing-report case treated as no report; storage/read failures remain errors. Complete ARVR install validation: - Remove 34 legacy PNG files not shipped by `pngjs` 7 and regenerate package files/metadata produced by the CI installer. - All 25 tracked PNG files match the reused centralized archive byte for byte (`/tmp/rn-visual-pngjs-upgrade-proof.json`). - Exact CI command `fbpython arvr/js/.castle/scripts/check_yarn_install.py --quiet` passes all four package roots with `Success. No changes detected.` - `arc lint` passes for all applicable regenerated files. - V6 CI confirms `yarn-validation`, `fbsource-yarn-package-sync-validation`, and all three workflow unit-test suites pass. The 3PL gate classified the existing in-place upgrade as new vendoring; provenance is documented above. Its manual retry was denied by the `llms_ci_claude_code` scheduling ACL. Flow compatibility follow-up: - Bound `defineRecordClass` defaults with `T extends interface {}` to satisfy the regenerated Immutable `Record` declarations. - `yarn flow focus-check --max-workers=8 --show-all-errors libraries/flux/records/Record.js` reports zero errors, checking the helper and its dependents (162 files). Full-root Flow validation remains a CI gate. - `arc f` passes. `arc lint` reports only the existing `local/testing-no-untested-method` warning for this pre-existing helper; all new type-bound lint findings are fixed. V8 CI and release-boundary follow-up: - Both `3pl-duplicate-detector` checks pass in V8. The earlier manual retry request is resolved. - Split the Gauntlet-only generated metadata into prerequisite `D126750229`; the base and prerequisite both pass `validate_jest_e2e_and_non_jest_e2e_changes.sh` locally. - The aggregate patch of the prerequisite, base, and ten suite drafts is byte-identical to the pre-split stack. - Gauntlet policy requires the prerequisite to land and enter a promoted release before this dependent diff can land; peer acceptance and final CI remain required. V9 validation results: - All three ARVR installer CI gates pass: Linux, Windows, and macOS. Both `flow-check-fbs` checks, `yarn-validation`, and `fbsource-yarn-package-sync-validation` pass. - Reverified all ten suite diffs remain drafts in the dependency chain above this base; each contains a Maestro flow and scenario inventory. All ten current YAML flows parse, and all 50 new capture IDs are unique and valid. - The prerequisite `D126750229` is blocked by a proven ScanCode name collision: its private Meta Gauntlet workspace is incorrectly identified as public npm `gauntlet`. The exact log evidence and documented TPMS first-party remediation are attached to that prerequisite. A maintainer must correct the TPMS classification, then land and release the framework prerequisite. Base peer acceptance and any remaining CI are still required. October 22 CI follow-up: - Update the Helix invalid-PNG assertions to the error emitted by `pngjs` 7. The tests continue verifying invalid data rejection, capture cleanup, preservation of a decode error when cleanup fails, and no retry. - `yarn test -p "HelixHTestImages-test|HelixHTest-test" --runInBand`: both affected suites pass, 300 tests total. - `arc f` and `arc lint` pass for both changed test files. - The later export job fails because the linked GitHub PR belongs to Rob Hogan; a PR ownership decision is pending. The Gauntlet TPMS classification correction remains pending. Reviewed By: cipolleschi Differential Revision: D126719236 Pulled By: cortinico

Summary: RNTester's rendering coverage, particularly text, currently lives in Meta's internal jest-e2e suite. To move it into public CI we need screenshot comparison that works for fork PRs and doesn't add PNGs to the repo. This adds that, as a separate set of visual flows alongside the existing Maestro E2E. ### How it works Visual tests are Maestro flows in `packages/rn-tester/visual-tests/flows/`. Each one deep-links to an RNTester example, waits for it to be ready and captures a screenshot cropped to a `testID`. Maestro only captures - it never decides pass or fail. The work is split across a trust boundary, so a fork PR never runs with credentials: - **Visual capture** runs as two jobs in Test All, reusing its RNTester Release builds: `test_visual_android_rntester` after `build_android` and `test_visual_ios_rntester` after `test_ios_rntester`. Each runs every flow in one Maestro invocation, on a Pixel 7 API 35 emulator (1080x2400, 420dpi) or an iPhone 17 Pro on iOS 26.2, and uploads the PNGs and a manifest recording the commit and device environment as an artifact. They have no secrets. - **Visual Compare RNTester** (`workflow_run` on Test All, from `main`) treats that artifact as data - it's validated before any image is decoded and nothing in it is executed. It finds the baseline, compares with pixelmatch, posts a check run per platform and a single PR comment with expected, captured and diff images, and on pushes to `main` publishes the captures as a new baseline. It runs once Test All completes, including when unrelated jobs fail, so the report arrives with the rest of CI rather than as soon as the captures are ready. - **Visual Approve RNTester** (`pull_request_target`, on the `visual-change-approved` label) records a maintainer's approval of an intentional change. The approval is tied to the exact set of differences, so a later push that changes them removes the label and needs a new review. Comparison is strict by default: any changed pixel is a difference. A test can be loosened in `visual-tests/config.json`, with evidence. Captures have been byte-identical across separate builds and runners so far, so I'd expect that to be rare. ### What defines the baseline A baseline is the full set of captures from a commit on `main`, keyed by commit and by device environment. Every push to `main` publishes one, and nothing is ever overwritten. A PR is compared with the baseline for the commit it's based on (the first parent of the merge commit CI tests), or the nearest earlier commit on `main` that has one, and the report says which. So a PR based on last week's `main` is compared with last week's rendering, not with whatever has landed since. Approving a PR doesn't promote its images - `main` is captured again after merge and that becomes the baseline. The environment key covers everything that affects rendering (OS runtime, device, screen size, density, font scale, locale, appearance, animations, system image and, on Android, the emulator version, which is pinned). If the CI image drifts, the key changes and a fresh baseline is published rather than every PR showing a spurious diff. ### Settings Storage is any GCS bucket, configured through repository variables. Writes use GitHub's OIDC token, so there's no stored secret, and only the two trusted workflows on `main` can authenticate. They need create-only access (`roles/storage.objectCreator`). Reads are public. | Variable | | |---|---| | `VISUAL_BUCKET` | Bucket to upload to | | `VISUAL_PUBLIC_BASE_URL` | Public URL for reads and report images - the bucket, or a CDN in front of it | | `VISUAL_GCP_PROJECT` | GCP project for authentication | | `VISUAL_WORKLOAD_IDENTITY_PROVIDER` | Workload identity provider trusting this repo's `visual-compare-rntester.yml` and `visual-approve-rntester.yml` on `main` | Expo can host the bucket and a CDN for this. Reports and approvals expire after 30 days by lifecycle rule. Baselines are small (8-175 KB per image), and at the current rate of pushes to `main` the cost is cents a month. `packages/rn-tester/visual-tests/README.md` covers adding a test and handling a failure. This starts with two text tests and runs non-blocking. The plan is to port high-value jest-e2e text and rendering tests, then make the checks required once they've proven reliable. ## Changelog: [INTERNAL] [ADDED] - Visual regression testing for RNTester in CI Pull Request resolved: react#59115 Test Plan: Ran the full flow in a fork (https://github.com/robhogan/react-native), with Test All trimmed to the jobs this depends on, against a personal GCS bucket: - A PR with no visual change passes on both platforms (robhogan#3). - A deliberate text layout change fails both platforms, with expected, captured and diff images on the checks page and in the PR comment (robhogan#4). - Applying `visual-change-approved` passes the approval check. Pushing a further, different change removed the label with a comment and failed the check again. - Captures were byte-identical across 5 Android and 4 iOS runs, from 3 separate builds on fresh runners. - When an unrelated Test All job fails, the comparison still runs. Run against a Test All run whose Android capture had failed, it reports Android as having no capture and compares iOS as usual. - Pixel Launcher raised an ANR dialog during emulator boot in 3 of 8 probe runs. After the capture's setup step, none of the 8 had a dialog showing. - Running all flows in one Maestro invocation brought the iOS capture step to 2m20s for two tests, from 3m49s for one test when each flow started Maestro separately. Failed flows are retried individually. - New unit tests for artifact validation (path traversal, symlinks, hash and size checks), comparison, storage keys, approval and report rendering. Not yet tested: a PR from a third-party fork, to confirm it can't reach storage credentials. ### Internal CI repair after the V11 import - Rebased on `remote/master` (`de108a6bc86ad106de94b84aa819e58b23cfc797`). Removed the dependency on abandoned D126750229; no Gauntlet framework files are included. All ten child suite patches are unchanged. Rob's PR react#59115 remains linked. - `xplat/yarn-validation/bin/yarn-validation`: passed for the root and RN OSS workspace, including the offline mirror check. - ARVR `HelixHTestImages-test`, `HelixHTest-test`, and `xplat-js-versions-aligned-test`: 3 suites and 486 tests passed. - `js1 test` selecting `rn-oss`: all 3 visual workflow suites and 51 tests passed. - OSS ESLint with `--no-ignore --max-warnings=0` on all 15 workflow JavaScript files: passed. `arc f` applied formatting; `arc lint` found no new issues. The Record helper missing-test warning also occurs on the unchanged `remote/master` file. - The exact four-root ARVR installer check exits 1 solely for existing Gauntlet `electron-chromedriver/METADATA.bzl` package URL drift. Restored that unrelated file and excluded the abandoned prerequisite change. - Device/Maestro captures were not repeated for this dependency and lint repair; Rob's Android/iOS evidence above and the child suite test plans are retained. Fresh CI results remain pending. Reviewed By: Abbondanzo, cipolleschi Differential Revision: D126719236 Pulled By: cortinico

react-native-bot · 2026-10-22T16:31:32Z

This pull request was successfully merged by @robhogan in b1755a3

^{When will my fix make it into a release? | How to file a pick request?}

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 15, 2026

robhogan marked this pull request as ready for review October 15, 2026 15:15

facebook-github-tools Bot added the Shared with Meta Applied via automation to indicate that an Issue or Pull Request has been shared with the team. label Oct 15, 2026

cortinico reviewed Oct 15, 2026

View reviewed changes

This was referenced Oct 16, 2026

Visual spike: chain-no-change robhogan/react-native#3

Open

Visual spike: chain-text-regression robhogan/react-native#4

Open

robhogan added 7 commits October 16, 2026 19:17

Document RNTester visual tests

8564417

Adds `packages/rn-tester/visual-tests/README.md` on how the tests run in CI, adding a test, running flows locally, handling a failed check and tolerances, and links it from RNTester's README. Changelog: [Internal]

robhogan force-pushed the robhogan/visual-regression-testing branch from 6fe7eed to 8564417 Compare October 17, 2026 07:10

cortinico approved these changes Oct 21, 2026

View reviewed changes

robhogan added 5 commits October 22, 2026 14:12

Upload Maestro logs for every visual capture

7353163

Logs from a passing capture help explain a screenshot that rendered differently, which a passing flow doesn't otherwise flag. Changelog: [Internal]

Clarify where visual baselines are published

d399319

Changelog: [Internal]

meta-codesync Bot closed this in b1755a3 Oct 22, 2026

react-native-bot added the Merged This PR has been merged. label Oct 22, 2026

This was referenced Oct 22, 2026

[Do not merge] Visual test check: no visual change #59369

Draft

[Do not merge] Visual test check: intended visual change #59370

Draft

		project_id: ${{ vars.VISUAL_GCP_PROJECT }}
		workload_identity_provider: ${{ vars.VISUAL_WORKLOAD_IDENTITY_PROVIDER }}

Conversation

robhogan commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

What defines the baseline

Settings

Changelog:

Test Plan:

Uh oh!

cortinico left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cortinico commented Oct 16, 2026

Uh oh!

robhogan commented Oct 19, 2026

Uh oh!

cortinico left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robhogan Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meta-codesync Bot commented Oct 21, 2026

Uh oh!

react-native-bot commented Oct 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

robhogan commented Oct 15, 2026 •

edited

Loading

robhogan Oct 22, 2026 •

edited

Loading