Repository navigation
test(zero-swift): fuzz the client with chaos against zero-cache - #111
Merged
Merged
Conversation
ChaosFuzzTests takes two clients through 40 random steps behind a ChaosProxy: mutations, cuts, outages, stalls, cuts partway through a message, Postgres writes behind their back, and relaunches on the same store. Then the network heals, and the run passes when Postgres holds each mutation once, each client's last mutation ID counts its mutations, and both clients see what Postgres holds. The E2E jobs run 2 seeds; the nightly workflow runs 150 new ones. Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859 Co-authored-by: tmm <tmm@tmm.dev>
It syncs every open issue and expected exactly the fixture's. The chaos fuzz, running beside it, keeps dozens of its own open for each seed, and one run saw them. Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859 Co-authored-by: tmm <tmm@tmm.dev>
The main test step ran them a second time beside the conformance suites, whose long synchronous loops (SweepIVMTests.sweepFuzz, the IVM vectors) hold every thread of the cooperative pool for tens of seconds on a 4-core runner. Nothing else on the pool runs meanwhile: a watchdog saw it unresponsive for 23 s and then over 30 s, and gdb showed all four pool threads in those tests. clientReconnectsAfterTheServerGoesAwayMidFrame spent its 15 s waiting for its connect to be scheduled. With E2ETests alone on 4 CPUs it passes; beside the conformance suites it failed. Amp-Thread-ID: https://ampcode.com/threads/T-c513a037-a257-45e0-b9a9-15c15edfe859 Co-authored-by: tmm <tmm@tmm.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chaos fuzz of the client against a real zero-cache. Test-only; no changeset.
ChaosFuzzTests(inE2ETests, so the PR end-to-end jobs run it) takes two clients through 40 random steps behind aChaosProxy: mutations, cuts, outages, stalls, cuts partway through a frame after N bytes, Postgres writes from outside, and client relaunches. After the run it checks that everything converges, in this order:lastMutationIDequals the number of mutations it made.The order matters. Checking views before the server had confirmed every mutation was a race in the test itself (seed 1013), not a client bug.
chaos-fuzzjob that runs 150 seeds from<run_number>000.ZERO_CHAOS_SEED=<seed>;ZERO_CHAOS_SEEDSandZERO_CHAOS_FIRST_SEEDset the range.The first CI run found a zero-cache crash: a client reset during a WebSocket handoff killed the server (fixed in #112). This run includes that fix.
Two more fixes for flakes the second run showed:
E2ETests. The end-to-end jobs run them on their own. Beside the conformance suites, the cooperative pool was unresponsive for 23 s and then over 30 s on 4 CPUs, measured with a watchdog. gdb showed all four pool threads running the long synchronous conformance loops (SweepIVMTests.sweepFuzz, the IVM vectors). In that window,clientReconnectsAfterTheServerGoesAwayMidFramespent its whole 15 s waiting for its connect to be scheduled. Locally on 4 CPUs it fails in the full suite and passed 3 of 3 runs withE2ETestsalone. This also corrects test(zero-swift): nightly fuzzing: ZeroVirtual, and Swift against JS on new seeds #110, which blamed the ZeroVirtual fuzz's main-actor work.chaos-issues. It expected exactly the fixture's open issues, and the chaos seeds running beside it keep their own issues open.