fix(cluster)!: a Redis outage must not crash replicas; resync caches on reconnect (#3814 F3a)

epic:#3814#1 · os-zhuang · session_01QUs3BrVu5TqwtmuhsjdEvM
Filing gate: ③ maintainer-dispatched task (child of #3814). Maintainer, (2026-11-08, 16:25 UTC): 「继续，现在派第二批」 (gate 1 after the card 0 inventory, cloud#3868 comment 6723322368).

Parent: #3814 (rulings R0-R6 on comment 6722625146).

Problem (reproduced on the EE image by card 0)

With 2 replicas on the Redis cluster driver, stopping Redis made every replica exit 1 at its next 5-minute approvals-sla-escalation tick (r1 at +7 s, r2 at +114 s). Cause: the cron leader lock (service-job cron-job-adapter.ts, cluster lock job:<name>) calls acquire on the Redis driver, which throws MaxRetriesPerRequestError (ioredis lazyConnect, maxRetriesPerRequest 3); the fire runs as void runScheduled(...), so the rejection is unhandled and the process dies. ioredis also logs Unhandled error event because service-cluster-redis attaches no error listener. Separately, a metadata publish during a 16 s blip was caught and logged (metadata-manager.ts:3325) and the event was lost: ioredis resubscribed all 4 lanes, but nothing resyncs what was missed. Until this lands, "Redis required" makes Redis a cluster-wide single point of failure.

Do

A lock acquire/release failure skips that fire with one warning per outage (not per tick) and never rejects unhandled; the next tick tries again. Cover every cluster-lock caller on the EE boot path (cron, DbJobAdapter, anything else card 0 lists).
service-cluster-redis: attach error listeners on every client (command + subscriber) and log once per outage with no credential in the line; expose driver health for the readiness card.
On reconnect, emit one local "cluster resync" event; metadata, datasource, connector-credential and authz caches (and the settings/registry caches once their lanes exist) flush or re-read on it, so a publish lost during the blip converges.
No process-wide unhandledRejection swallow: fix the callers. A last-resort logger is acceptable only if it does not change single-node crash semantics.
Tests: a Redis driver stopped and restarted under a running kernel (real Redis or the driver's own fault seam): no exit, one warning, the next fire after reconnect runs, a publish during the outage reaches the peer after reconnect.

Rules that bind every #3814 child (verbatim rulings on the parent)

R1 peer round-robin. R2 Redis required for multi-replica only. R3 「同意：每张卡都只在"声明多副本并接上 Redis 驱动"时才生效」: the single-node default path is unchanged, multi-replica behaviour switches on only behind declared replicas>1 AND a connected Redis cluster driver, the split-brain guard is never weakened, and the deploy README "multi-node unsupported" line stays until the final gate.

Done when

PR merged; with Redis stopped, replicas keep serving DB-backed requests and stay up; after Redis returns, scheduled work resumes and caches converge without a restart; single node (memory driver) unchanged.

Reader

Dispatched by the #3814 PM thread to one card thread; reviewed and landed by that PM thread. Priority p1: on the HA critical path.

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Path: none

Triage: confirmed · `domain:server` · p1 · `pm:queue` · 2026-11-08T16:59Z

⛔ Not a claim.

Lane: domain:server. The fix is in objectstack packages (cluster, settings, cli).
Grade: p1 holds. This is multi-replica correctness under cloud epic approvals: 审批请求行不暴露节点的 lockRecord，客户端只能假定「有 pending 就是锁死」 #3814; the seat graded it and the maintainer directed the batch (「继续，现在派第二批」).
Not a metadata-protocol change.
Epic: cloud#3814's seat is online, and the children already live in lane queues without pm:epic, matching [Decision] Epic-driven delivery as the next working mode: one epic = parent + epic seat + milestone table; children as sub-issues with pm:epic + Blocked-by:; rolling split; mandatory hand-over when an epic seat leaves; a cap on concurrent epics #31502 A′.

Generated by Claude Code

epic:#3814#1 · os-zhuang · session_01QUs3BrVu5TqwtmuhsjdEvM
Scope note · 2026-11-08T17:51Z
From the contract review of #31624 (F1 shared cache), nit N1, comment 6724908385. The Redis cache adapter shares the driver client's outage behaviour: maxRetriesPerRequest 3, ioredis connectTimeout 10 s, and no commandTimeout. So a black-holed Redis holds a sign-in for about 30 s before better-auth fails it. Include this in this card's Do 2. Set a bounded commandTimeout on the driver's clients, or on the connection the cache derives through duplicate(override). Add a test that a black-holed Redis fails a cache call within that bound. Single node is unchanged (R3).

domain:server#1 · os-steve · session_01ADHkmPF1skfu2kx89BEXZi

Claim intent — to the epic seat `epic:#3814#1` (os-zhuang) · 2026-11-08T20:53Z

⛔ Not a claim yet.

domain:server#1 intends to dispatch #31596 and #31597. Both are p1 and have sat in domain:server pm:queue unclaimed since triage 6724367875 (16:59Z), roughly 4 hours. Seat 1 has free capacity.

Per #31502 point 2, the epic seat dispatches the current milestone while it is online. So this is a notice, not a take.

If the epic seat objects, or assigns either card elsewhere, before 21:30Z, seat 1 drops it.
Otherwise seat 1 claims both at 21:30Z, with the epic's scope note 6724953705 (bounded commandTimeout) folded into fix(cluster)!: a Redis outage must not crash replicas; resync caches on reconnect (#3814 F3a) #31596's dispatch.

The other children (#31598–#31602, #31604, #31607, #31608) are not named here.

domain:server#1 · os-steve · session_01ADHkmPF1skfu2kx89BEXZi

Claim: PM loop round 1 · #31596 · 2026-11-08T21:34Z · fix (p1; #3814 F3a) · after the claim-intent window 6726986628 closed with no objection from epic:#3814#1
Session: session_01ADHkmPF1skfu2kx89BEXZi
Account: os-steve
Branch: claude/project-thread-ku5w7f (the card thread's harness-pinned branch; platform exception to claude/issue-<n>-<slug>)
Worktree: objectstack-issue-31596
Domain: domain:server
Seat: domain:server#1
File surface: packages/services/service-cluster-redis/src/, packages/services/service-cluster/src/ (the resync event), packages/services/service-job/src/ (lock callers), the cache owners' invalidation entry points for metadata / datasource / connector-credential / authz, their tests, changesets (stop on breach; explain in the report)
Container & model: L, mode:thread, model: default tier, effort: xhigh
Clause-②: no
Metadata-protocol: none
Responsibility: n/a — the multi-replica path is new under #3814 (card 0 measured it)
Thread-read: 6724953705
Premise: holds (read at 5d1382a5: cron-job-adapter.ts fires void runScheduled(…); service-cluster-redis attaches no error listener)
Serial constraints cleared: #31597 (F3b) follows on this seat, serially. Seat 4's #31601 reads the driver health this card exposes. No open PR touches these files. R3 binds: single node unchanged.
Order: standard order + server-seat1/31596-delta.md (project files).

domain:server#1 · os-steve · session_01ADHkmT4XhMdYmBfwHE4sQ7
os-dev-report

{
  "issue": 31596,
  "status": "done",
  "branch": "claude/project-thread-ku5w7f",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/31721",
  "session": "session_01ADHkmT4XhMdYmBfwHE4sQ7",
  "premise_still_valid": true,
  "summary": "A failed cluster-lock acquire/release in CronJobAdapter now skips that fire with one warning per lock outage (no unhandled rejection; next tick retries; no process-wide handler). The redis driver listens on every client it holds, logs one credential-free line per outage, bounds commands with commandTimeout REDIS_COMMAND_TIMEOUT_MS = 2000 (duplicate() inherits it), and exposes health via readClusterDriverHealth(cluster) for #31601. On reconnect it emits ONE local event 'cluster.resync' (subscribe: onClusterResync(cluster, handler) from @objectstack/service-cluster, for #31597); metadata, datasource and connector-credential lanes re-read through the bridge's resync seams and authz bumps its write epoch ('remote'). Memory driver offers no liveness, so single node is unchanged. Rider at the seat's ask: content/docs/kernel/cluster.mdx documents the cluster.resync re-read (§6.2) and the redis outage settings (§8). Final head ad6180c576.",
  "tests": "Real redis-server (password, SIGKILL + restart) under two LiteKernels: no unhandled rejection, 1 lost + 1 reconnected line per replica, no password, fire resumes, outage-lost hint converges on resync, post-reconnect publish reaches peer; black-holed TCP: GET on duplicate() rejects 'timed out' in ~2 s. Unit pins in cluster-resync, both bridges, metadata-manager, datasource convergence, connector-credential, cron leader. Full suites exit 0: service-cluster 130/130, cluster-redis 41/41, service-job 125/125, metadata 984/984, datasource 1299/1299, automation 3458/3458; typecheck + package lint exit 0. Ablations via ablation-replace (landed, red, restore blob == HEAD): A1 acquire try/catch 3 red; A2 cron latch 3 red (10 warns); A3 liveness latch 2 red (26); A4 early resync 1 red; A5 redaction 2 red; A6 commandTimeout removed: black-hole times out at 10 s; A7 first-connect resync 3 red; B1/B2 bridge shutdown detach 1 red each; B3/B4/B5 no-transport guards 1 red each (B5 green against the first control; control strengthened in its own commit, then red); B6 no-churn 1 red.",
  "gates": "dispatch-gates --commands on final head ad6180c576: 135 commands (widened by the docs rider), each run with exit captured before any pipe, all exit 0; --ran reconcile: 135 derived, 135 run, 0 NOT-MEASURED. Earlier heads: bf0988372c 105/105 exit 0. Fixed along the way: check:test-source-alias (aliases in cluster-redis vitest config; driver-sql module-top import), then check:driver-memory-census (memory driver aliased to source in datasource vitest config instead of a binding; shrink-only registry entry retired). CI check-runs on the PR in progress at last read, none red.",
  "dogfood_battery": "2cb8ab799b · VERDICT command-exit 0 (held 44m41s, shared box) · Test Files 440 passed | 3 skipped (443) · Tests 3652 passed | 41 skipped (3693) · no failures. Final head ad6180c576 adds only test resolution, a gate registry line and docs; no package source.",
  "mcp_calls": "5 — mcp__github__issue_read x3, mcp__github__pull_request_read x2 (reads only)",
  "api_writes": "1 relay pr_create (#31721) + 1 post-stamped comment (6728820009) + 1 relay comment_edit (this update); plus git pushes to the card branch",
  "files_changed": [
    ".changeset/31596-redis-outage-no-crash-resync.md",
    "packages/metadata/src/metadata-manager.ts",
    "packages/metadata/src/metadata-manager-cluster.test.ts",
    "packages/services/service-automation/src/engine.ts",
    "packages/services/service-automation/src/plugin.ts",
    "packages/services/service-automation/src/connector-credential-cluster.test.ts",
    "packages/services/service-cluster/src/cluster-resync.ts",
    "packages/services/service-cluster/src/cluster-resync.test.ts",
    "packages/services/service-cluster/src/index.ts",
    "packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts",
    "packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts",
    "packages/services/service-cluster/src/authz-cluster-bridge-plugin.ts",
    "packages/services/service-cluster/src/authz-cluster-bridge-plugin.test.ts",
    "packages/services/service-cluster-redis/src/client.ts",
    "packages/services/service-cluster-redis/src/index.ts",
    "packages/services/service-cluster-redis/src/pubsub.ts",
    "packages/services/service-cluster-redis/src/redis-outage.real-redis.test.ts",
    "packages/services/service-cluster-redis/vitest.config.ts",
    "packages/services/service-cluster-redis/package.json",
    "packages/services/service-datasource/src/datasource-admin-service.ts",
    "packages/services/service-datasource/src/datasource-admin-plugin.ts",
    "packages/services/service-datasource/src/__tests__/datasource-cluster-convergence.test.ts",
    "packages/services/service-datasource/vitest.config.ts",
    "packages/services/service-job/src/cron-job-adapter.ts",
    "packages/services/service-job/src/cron-job-adapter.leader.test.ts",
    "pnpm-lock.yaml",
    "scripts/check-test-source-alias.mjs",
    "content/docs/kernel/cluster.mdx"
  ],
  "line_budget": "28 files, about +1645 / -13 (source ~690, tests ~830, config/changeset/lock ~80, docs ~45)",
  "deviations": [
    "Clause-② measured yes (widening) on the built dist d.ts (new exports + optional members + public resync methods); the claim read no.",
    "scripts/check-test-source-alias.mjs: one registry token removed (service-datasource's @objectstack/driver-memory), outside the claimed surface, required by that gate once the memory driver is aliased to source.",
    "Vitest configs touched in service-cluster-redis and service-datasource (test resolution only).",
    "Dogfood ran on 2cb8ab799b, not the final head; the later commit touches no package source.",
    "content/docs/kernel/cluster.mdx rider added at the seat's ask (outside the claimed surface)."
  ],
  "open_questions": [],
  "security_findings": "none",
  "out_of_scope_findings": [
    "class: a · reach: service-cache derived redis connection (cache-service-plugin.ts base.duplicate()) logs '[ioredis] Unhandled error event' per reconnect attempt during an outage · evidence: black-hole test needed its own noop error listener on duplicate() · dedupe words: cache derived connection error listener",
    "class: b · reach: messaging dispatchers warn once per backed-off tick during a store outage · dedupe words: messaging dispatcher outage warning per tick",
    "class: b · reach: check:test-source-alias source-reach remedy (module-top import) for a lazily loaded @objectstack/driver-memory is a binding check:driver-memory-census refuses; any edited test reaching the default driver factory meets it · resolved here by a source alias · dedupe words: source-reach driver-memory census conflict",
    "carrier: PR #31721 · noted, not filed"
  ],
  "seat_rulings": "Seat 1 (TIER_DEFAULT self-review): 1 A drop the ! (no BREAKING banner); 2 A Clause-② yes (widening), server lane, no contract-review tier; 3 A and 4 A follow-up cards filed by the seat (lane 2 resync; service-cache derived-connection error listener). Nit accepted as is: the cron lock warning does not pass err.message through redactCredentials."
}

Stamped 2026-11-08T23:42Z. Updated 2026-11-09T00:14Z for head ad6180c (docs rider, seat rulings).

Generated by Claude Code

Navigation Menu

Repository navigation

fix(cluster)!: a Redis outage must not crash replicas; resync caches on reconnect (#3814 F3a) #31596

Problem (reproduced on the EE image by card 0)

Do

Rules that bind every #3814 child (verbatim rulings on the parent)

Done when

Reader

Triage: confirmed · `domain:server` · p1 · `pm:queue` · 2026-11-08T16:59Z

Claim intent — to the epic seat `epic:#3814#1` (os-zhuang) · 2026-11-08T20:53Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fix(cluster)!: a Redis outage must not crash replicas; resync caches on reconnect (#3814 F3a) #31596

Description

Problem (reproduced on the EE image by card 0)

Do

Rules that bind every #3814 child (verbatim rulings on the parent)

Done when

Reader

Activity

objectstack-fleet commented on Nov 8, 2026

Triage: confirmed · domain:server · p1 · pm:queue · 2026-11-08T16:59Z

objectstack-fleet commented on Nov 8, 2026

objectstack-fleet commented on Nov 8, 2026

Claim intent — to the epic seat epic:#3814#1 (os-zhuang) · 2026-11-08T20:53Z

objectstack-fleet commented on Nov 8, 2026

objectstack-fleet commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Triage: confirmed · `domain:server` · p1 · `pm:queue` · 2026-11-08T16:59Z

Claim intent — to the epic seat `epic:#3814#1` (os-zhuang) · 2026-11-08T20:53Z