Skip to content

fix(runtime): key a packaged job by its package coordinate and keep its catalogue key across a restart (ADR-0150 E7) - #31730

Merged
objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/project-thread-znr079
Nov 9, 2026
Merged

objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/project-thread-znr079

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Steve · project thread

Fixes #31679

Clause-②: no
Metadata-protocol: none

ADR-0150 child E7 (epic #31527). Seat: domain:server#3.

维护者速读

  • 改了什么:定时任务在任务服务上的键改由「软件包坐标(包 id + 任务名)」决定;重启时先读任务目录(sys_job)里本包自己的 包id:任务名 行,仍处于启用状态就沿用它。
  • 为什么改:以前裸任务名归「本次启动最先调度的包」,重启时包的加载顺序和安装顺序不同,第一个包的目录行和运行历史就会被第二个包的任务接走(实测见下表 S8/S9)。
  • 风险与回滚:只影响两个包声明同名任务、且用数据库任务服务的环境;没有任何请求被拒绝。回滚即还原本 PR,键回到按启动顺序分配。
  • 先看:packages/runtime/src/app-artifact-handlers.ts 的 jobKeyFor 与 catalogueHoldsScopedKey;再看 app-artifact-handlers.jobs.test.ts 新增的 ADR-0150 E7 一组。
  • 两分钟验证:pnpm --filter @objectstack/runtime exec vitest run src/app-artifact-handlers.jobs.test.ts,预期 41 个用例全绿。
  • 需要你做的:无,席位落地。

What changes

jobKeyFor (packages/runtime/src/app-artifact-handlers.ts) now takes the job's package coordinate, { packageId, name } (ADR-0150 D2/D4), and answers in this order:

  1. the key this package already holds the job under in this process (unchanged);
  2. new: the package-scoped key PACKAGE_ID:NAME, when the job catalogue holds it active. The reading is DbJobAdapter.readArming, the optional, adapter-local method the schedule trigger already reads. It is not part of IJobService, so a job service without it keeps no catalogue and gets no reading. Only the asking package's own scoped key is read: that key names its package, so the reading is about this job only. The bare row names no package and is never read as anyone's;
  3. the authored name, when no other package holds it in this process (unchanged);
  4. the package-scoped key, naming the package that holds the authored name (unchanged).

So the first holder keeps the bare key and a second keeps its PACKAGE_ID:NAME, as the card and the ADR-0150 execution plan row E7 state. What step 2 adds is that this stays true across a restart that schedules the packages in another order than they were installed in. Before, the bare key went to whichever package the boot scheduled first, so the first package's catalogue row and run history moved to the second package's job (rows S8 and S9 below).

A cancel clears active (a replace that drops the job, or the package's uninstall), and a runtime that stops does not. So a package that was uninstalled and later installed alone takes its authored name again (S10), exactly as before.

A catalogue read that throws is no reading: the job is keyed by this run's order, as on a job service without a catalogue, and that is said once at warn per job. The job still runs, so this is a functional degradation, not a durability one.

The catalogue read is awaited before the in-process holder is asked, so the window between that answer and claimJobKey is no wider than before.

No other file changes. service-job is not edited: readArming already exists there.

Step 0: the keys today, and on this head (same inputs)

The same scratch harness (never committed) ran on base b126c130 and head fc99b124. It has one catalogue that outlives a process and one job service per process. Packages are a, b and c, each declaring job t.

# Inputs Base keys Head keys
S1 one package a t t
S2 a, then b b:t, t b:t, t
S3 a, b, c b:t, c:t, t b:t, c:t, t
S4 a, b, reinstall b, reinstall a b:t, t b:t, t
S5 a, b, uninstall a, then c b:t, t b:t, t
S6 restart, a alone with its stored bare key t t
S7 a, b; restart in the same order b:t, t b:t, t
S8 a, b; restart scheduling b first a:t, t (t now runs b) b:t, t (t runs a)
S9 a, b, uninstall a; restart b t (b leaves b:t) b:t
S10 a, b, uninstall b; restart, b alone t t

The keys are byte-identical on base and head for 8 of the 10 inputs: every input whose keys base keeps stable. S8 and S9 differ. They are the restarts where base moves a key that the catalogue holds for one package to another package or another name. Keeping those keys is what the card asks: "the old bare key and APPID:jobName keys stay readable for stored jobs". Nothing is refused in either case. Seat 3 ruled option A: keeping the catalogue key across a restart falls inside the card's "stored keys stay readable". S8 and S9 are the base defects this closes. In S8, base hands the first package's bare key, with its catalogue row and run history, to the second package's job, and moves the first job to a new name. In S9, base moves the surviving scoped holder off its catalogued b:t onto the bare t, which leaves its run history behind.

Tests

New top-level describe in the existing harness file packages/runtime/src/app-artifact-handlers.jobs.test.ts, called ADR-0150 E7: a job's package coordinate keeps the key the job catalogue holds for it, across a restart. It has six cases:

  • a job stored under its bare key before the upgrade is scheduled under it again, and fires (the card's pin);
  • a restart in the install order keeps every key (control);
  • a restart that schedules the second package first hands it its own scoped key, and each key runs its own package's body;
  • a scoped holder whose bare holder was uninstalled keeps its scoped key across a restart;
  • a scoped key the catalogue holds cancelled keeps nothing: the package, alone after a restart, takes its authored name;
  • a catalogue read that fails still schedules the job and warns once.

No existing case was changed: the nine #21602 cases pass unmodified. One forced edit to the file: check:test-source-alias judges an edited test file, and this one imports AppPlugin, whose source loads @objectstack/metadata dynamically. So the file now carries the gate's prescribed module-top import '@objectstack/metadata';, which pays that load at collection instead of inside a timed case.

Ablation: put back "the first scheduler takes the bare key"

Through node scripts/ablation-replace.mjs. The anchor const catalogued = await catalogueHoldsScopedKey(svc, coordinate, logger, tag); was replaced by const catalogued = false as boolean;. The anchor went from 1 hit to 0, and the blob from 1fb1fd13059d to e0c1c52e8cc3. The test imports the source, so there is no dist leg.

× a restart that schedules the second package FIRST still hands it its own scoped key — never the first package's bare one
× a scoped holder whose bare holder was uninstalled keeps its scoped key across a restart, as it does in-process
× a catalogue read that fails schedules the job anyway, by this run's order, and says so at warn
Tests  3 failed | 38 passed (41)

The restore matched the blob at HEAD (1fb1fd13059d), and git diff HEAD came back empty.

Acceptance notes

ADR-0150 readings:

  • (a) The phase-1 one-holder tests stay green. Job is not a phase-1 covered type, and bare-name-one-holder.ts is untouched. The runtime suite: 563 files, 9045 passed, 19 skipped.
  • (b) Bare-name behaviour is unchanged for every name with one holder: S1, S6 and S10 give the same keys on base and head. No door addresses a job by bare name. The only readers are the catalogue (sys_job.name) and run history (sys_job_run), and they read the key this card keeps stable.
  • (c) There is no migration. The catalogue read is read-only and re-runnable. It reads only the asking package's own scoped key, and never guesses who holds a bare one.
  • (d) Clause-②: no. No protocol, spec or metadata change, so no ADR-0087 entry.

Red lines:

  1. Resolution never loosens. No first-registered fallback is added. The first-holder-keeps-the-bare-key rule is the ADR row's own reading, and E7 keeps it as it is. Step 2 removes the case where the order of first scheduling overrode a stored identity. This card resolves no bare name to a package: a job is scheduled by its own package, so its coordinate is known. R's resolvePackageScopedName therefore has no call site here, and no second resolver is written.
  2. Ambiguity is reported, never guessed. The bare catalogue row names no package and is never attributed to one.
  3. Independently releasable. The change is additive and needs no migration. A job service without readArming (the interval and cron adapters) behaves exactly as before.

Also noted, not filed:

Verification

Suites on fc99b124 (the implementation commit). The final head ffa24bd0 adds only the module-top import in the test file, and the edited file was re-run there (41 passed):

  • pnpm --filter @objectstack/runtime test: 563 files, 9045 passed, 19 skipped. Exit 0.
  • pnpm --filter @objectstack/runtime typecheck: exit 0.
  • pnpm --filter @objectstack/service-job test: 11 files, 121 passed. Exit 0.
  • pnpm --filter @objectstack/service-job typecheck: exit 0.
  • npx eslint --no-inline-config packages/runtime: exit 0.
  • pnpm --filter @objectstack/dogfood test (full battery, closure built first): 440 files passed, 3 skipped; 3652 tests passed, 41 skipped. Lock VERDICT command-exit 0.
  • Derived gates (dispatch-gates --commands on the final head ffa24bd0): 64 of 64 exit 0. --ran reconciliation: 64 derived, 64 run, 0 NOT-MEASURED, 0 UNRUN. (A first round on fc99b124 caught check:test-source-alias on the edited test file, fixed in ffa24bd0. It also had two prerequisite exit-3s that cleared once the fixture commit was fetched and the packages were built.)

Human Minimum

  • 1. Schemas start as Zod; packages/spec holds no runtime logic.
  • 2. Every name I added uses the casing for its kind.
  • 3. No formatter ran; pnpm lint passes.
  • 4. One issue, one branch off an up-to-date main, one pull request.
  • 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset.
  • 6. packages/spec touched: its generated artifacts are regenerated and committed. (Not touched.)
  • 7. No hand edits to content/docs/references/, packages/*/CHANGELOG.md or content/docs/releases/.
  • 8. Title, body and comments are in English.
  • The Before Submitting list in CONTRIBUTING.md is done.

…he catalogue's scoped key across a restart

jobKeyFor now takes the job's package coordinate (package id, job name).
Before falling back to this run's scheduling order it reads the job
catalogue (DbJobAdapter.readArming, optional and adapter-local) for the
coordinate's own package-scoped key: held active, it stays the job's key,
so a restart that schedules the packages in another order no longer hands
one package's bare key and run history to another package's job. The
first holder still keeps the bare key and a second its <packageId>:<name>.

Claude-Session: https://claude.ai/code/session_01ADHkmNp8ARUNvemy119dY1
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/m documentation Improvements or additions to documentation tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface labels Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/runtime, touching 15 documentable anchor(s).

17 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: node scripts/docs-audit/affected-docs.mjs --json 82a3ed2e861d260fdd65517e6b17f03c42eb90c2.

⛔ 8 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails.

What this run could not see
  • the SDK route bridge reached 65 of 231 client-bound route-ledger rows — the other 166 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 166: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 59 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 107 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 28 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 82a3ed2e861d260fdd65517e6b17f03c42eb90c2 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from a8e78dcadfc85da09b203c50ab06202158814768 — the merge of head ffa24bd0eb22f067c8176b70b12f6b2889a929cb into base 82a3ed2e861d260fdd65517e6b17f03c42eb90c2, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin a8e78dcadfc85da09b203c50ab06202158814768 && git checkout a8e78dcadfc85da09b203c50ab06202158814768
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 82a3ed2e861d260fdd65517e6b17f03c42eb90c2 ffa24bd0eb22f067c8176b70b12f6b2889a929cb && git checkout -B drift-repro 82a3ed2e861d260fdd65517e6b17f03c42eb90c2 && git merge --no-ff ffa24bd0eb22f067c8176b70b12f6b2889a929cb

node scripts/docs-audit/affected-docs.mjs --json 82a3ed2e861d260fdd65517e6b17f03c42eb90c2

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 82a3ed2e861d260fdd65517e6b17f03c42eb90c2 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

epic:#31527#1 · os-steve · session_01ADHkmP7o3uDTti21wLTfPf

Epic red-line review: E7 #31679, head ffa24bd · 2026-11-09T00:34Z

Verdict: PASS. Option A (keep the catalogue key) is accepted as part of this card, not a separate fix.

Red line Reading
1. Never loosens jobKeyFor takes the package coordinate. The order is: own in-process key; else this package's own <packageId>:<name> when the catalogue holds it active (readArming); else the authored name when no other package holds it; else the scoped key. A job's package is its scheduler, so no bare name is resolved, and no resolver call is correct here. Nothing is refused, and no key is opened to another package.
2. Ambiguity reported, never guessed The catalogue is read only for the asking package's own scoped key, and the bare row is never attributed to anyone. A failed read takes today's in-process answer and warns. Base was the silent pick: on S8/S9 a restart's scheduling order handed one package's bare key and run history to another package. The head removes that pick, so the change is inside the card's "stored keys stay readable" and is ADR-0150 D4 (runtime keys by coordinate).
3. Independently releasable 8 of 10 key-table inputs are byte-identical to base. The 2 that differ are exactly the order-dependent restarts, now stable. A job stored under its bare name before this release keeps it. The changeset is a patch, Clause-②: no, with the affected parties stated. Ablations: 3 red.

Seat 3 accepts and lands this after green CI. It is not a governed surface.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

domain:server#3 · os-steve · session_01ADHkmQXLfvyB5R53VDm3Ka

席位核对 · ACCEPT · #31730 (Fixes #31679, ADR-0150 E7) · 2026-11-09T00:51Z

Head-sha: ffa24bd0eb22f067c8176b70b12f6b2889a929cb

  • Form: draft to main; first body line Fixes #31679, the only closing keyword in the body (read); the whole card lands here.
  • Premise measured: jobKeyFor takes { packageId, name }; before the scheduling-order fallback it keeps <packageId>:<name> when the job catalogue (readArming, optional on the job service) holds it active; a catalogue read failure warns once and falls back to today's order, never to a guess on another package's key. No resolver call: E7 resolves no bare name (seat read the diff).
  • Gates (check-runs on this head): 54 runs, 37 success, 17 skipped, 0 other; Lint & Repo Gates success, TypeScript Type Check success. check-governed-merges --pr 31730: NOT governed, 320 changed lines. K = 2 first-parent commits on main since the PR base.
  • Accepted deviations: two of ten key-table inputs differ from base (S8, S9: restarts where base handed one package's catalogue row and run history to another by scheduling order). Seat ruling A, confirmed by the epic seat as the silent pick D4 removes; stated in the PR body under the key table. One forced test edit (the module-top @objectstack/metadata import check:test-source-alias prescribes), named in the body.
  • Epic red-line review: PASS 6729440308 at this head.
  • Out of scope: none filed. Skills-docs: not owed (Metadata-protocol: none, no narrowing).
  • Landing: ready + auto-merge in this stroke. On merge: landing record on runtime/service-job: jobKeyFor on the package coordinate, keeping the bare key readable (ADR-0150 child E7) #31679, pm:dispatched removed, card closed by Fixes.

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review November 9, 2026 00:52
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge November 9, 2026 00:52
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 42542605342 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

分类:failure —— 按下面的日志分诊。

失败的 job(日志抽取,best effort):

  • Test Core (1/8) — 失败步骤: Run this shard's tests

    @objectstack/cli:test:  FAIL   integration  test/validate-namespace-name.test.ts > ADR-0150 child N — os validate and os build refuse what the package door refuses, in its words > premise: the package
      ↳ 失败原因: @objectstack/cli:test: AssertionError: expected BareNameConflictError: Bare-name conflict… { …(8) } to be undefined
    

↳ 失败原因 是判读的关键:超时(Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言(AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

⚠️ 断言这一侧有一类例外,判据是断言在测什么,不是它是不是 AssertionError。 断言的对象是产品行为(一个值、一个形状、一次拒收)⇒ 照上面读:真实的行为改变,去查,⛔ 不要重排掉;
断言的对象是这次实验自身的有效性前提(跑完的耗时、负载下的先后、任何只在时间预算内才成立的条件)⇒ 它跟超时是同一类,同样对负载敏感,重排一次是合法的判别手段。
识别是机械的:断言的消息或它比较的值本身点名了一段时长、一个时间戳、一个耗时计数。实测过的一对 —— AssertionError: SecurityPlugin.init() ran: expected false to be true 测的是产品行为(真回归);
AssertionError: this run took over a second, so second-precision stamps could have differed too: expected 1006 to be less than 1000 测的是实验前提:它守护的那条不变式当时是绿的,同一个 head 原样重排一次即成功。
穿着 AssertionError 外衣的时间测量,仍然是时间测量。(⛔ 这只改「怎么读一次红」,不改「哪些测试可以重排」——后者由别处管。)

跨 PR 相同签名(24h,按失败测试文件聚合):

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 14 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

Merged via the queue into main with commit f2abd47 Nov 9, 2026
55 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/project-thread-znr079 branch November 9, 2026 01:18
rajeev-kumar-se pushed a commit to rajeev-kumar-se/objectstack that referenced this pull request Nov 10, 2026
…dule): automation stored references gain a package column, and a dispatch key names its flow's package (ADR-0150 D8 S3) (objectstack-ai#31903)

<!-- claude-projects-attribution: {"github_login":"os-steve"} -->
_Requested by **Steve** · [project
thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi9G8vs4TWZnL2DqxZzpGY1rv)_

Fixes objectstack-ai#31684

Clause-②: no
Metadata-protocol: none

## 维护者速读

- 需要你做的:**审批本 PR**。它是 3011 行(+2963 / −48),超过人工合并线 3000 行 11 行,所以要
GOVERNED_APPROVERS 批准后才能由席位走队列合入。没有触及任何受治理面(0 of 68 路径命中登记表),超线只因体量。
- 体量构成:业务源码约 1 370 行(含注释);测试约 1 420 行(5 个新测试文件 + 2 个现有测试文件追加,含真实启动的
dogfood);生成物约 150 行(两个包的 11 种语言 i18n 包,tenant-audit census
两个文件);changeset、vitest 配置、lockfile 若干行。没有为压线删改测试。
- 改了什么:
- 五处按名字存的引用各加一列可空、只读、由平台填写的软件包列:审批请求 `process_package`(R7,只回填
pending/returned)、暂停的流程运行 `flow_package`(R8)、流程凭据
`flow_package`(R9)、连接器连接 `connector_package`(R11)、旧邮件模板行
`package_id`(R20,内置模板名绑到 `com.objectstack.service.email`)。
- 启动(`kernel:bootstrapped`)时按 ADR-0150 D3
的唯一持有者规则(不带调用方包)回填;两个包都持有的名字留空并告警列出全部候选,绝不取第一个;第二次启动不写任何行。
  - 定时与相对时间触发的派发键(R10)带上流程所属的包:`schedule:包:流程:窗口`。
- 为什么改:ADR-0150 D8 要求先把包坐标存下来,后续读者子卡才能按 (包, 名)
读;派发键带上包,两个包同名的流程就不会共用一个时间窗。
- 风险与为什么安全:
  - 新列目前没有任何读者,写入方也不填(交接点在 PR 正文点名);五张表和派发账本的唯一键一个不动,均有逐字节钉住。
- 派发键改形:启动时先把 30 天保留期内的旧键复制成新键,复制完成前账本的认领、结算、读取都等待(最多 120
秒);新版本节点认领时**先插旧键、再插新键**,谁先插入旧键谁赢得该时间窗,所以滚动升级期间新旧节点共用一个仲裁,无论先后同一时间窗只触发一次;旧键与新键一起结算,旧节点的重放保护也能读到已送达。唯一不插旧键的是两个包同名的流程(防撞),本版本注册时就会拒绝,到不了。
  - 回填某张表失败只记警告、保留原样,下次启动再试,不阻断启动。
- 回滚:还原本 PR 即可。新版本一直同时写旧键,旧版本照常读旧键;新列可空、无读者,留着无害。
- 已核验:ADR-0150 史诗红线评审 PASS(R1 滚动升级双触发已按根治修复后复审);119
个派发门禁全过;四个包全量套件、dogfood 12/12、CI 全绿;关键机制(先插旧键、防撞、账本等待、过渡读、不取第一个、覆盖绑到
P)各自消融后变红再复原,表在 PR 正文。
- 先看:`packages/triggers/trigger-schedule/src/schedule-trigger.ts` 的
`claimDispatch` /
`dispatchKeys`,`packages/services/service-automation/src/engine.ts` 的
`flowDispatchHolder`,以及 `flow-dispatch-store.ts` 的 `holdUntil`。
- 两分钟验证:看 `schedule-dispatch-claim.test.ts` 的 describe「one arbiter
across a rolling upgrade」(旧节点与新节点两种先后各只触发一次),以及 dogfood 文件
`automation-stored-reference-package-column.dogfood.test.ts`
的「启动中的认领在重键之后作答」。

## What changes

### Five stored references gain a package column (R7, R8, R9, R11, R20)

Each column uses the name D8 gives it. Each is nullable, read-only to
users, and filled in by the platform.

| Row | Table | Name column | New column | Rows asked | Package |
|:--|:--|:--|:--|:--|:--|
| R7 | `sys_approval_request` | `process_name` (`flow:NAME`) |
`process_package` | open: `pending`, `returned` (seat Q5) |
plugin-approvals |
| R8 | `sys_automation_run` | `flow_name` | `flow_package` | `paused`
only (seat Q4) | service-automation |
| R9 | `sys_flow_credential` | `flow_name` | `flow_package` | all |
service-automation |
| R11 | `sys_connector_connection` | `connector_name` |
`connector_package` | all | service-automation |
| R20 | `sys_email_template` | `name` | `package_id` | all (legacy
overlay rows) | platform-objects (declared), plugin-email (backfill) |

The backfill runs at `kernel:bootstrapped`, once the registry has sealed
its holders (`bareNameHoldersAreSealed`). It walks the rows whose new
column is null (keyset walk) and asks each name the one D3 rule with no
caller package. That answer is the item's one holder in this
environment, which is the D4 identity of the row (seat answer 2,
6736840427; epic input 6736817259):

- Flows (R7, R8, R9) ask through the new
`AutomationEngine.flowHolder(name)`, which is the resolver the plugin
already attaches (`flowNameResolver`) asked with no caller. An
environment override of P's flow is P's flow overlaid, not a second
item, so its runs, requests and credentials bind to P. ⛔
`armedFlowPackage` is not used here; it stays the caller-side reader.
- Connectors (R11) ask `connectorNameResolver`, the same resolver the
connector reader uses.
- Email templates (R20) ask objectql `resolvePackageScopedName` with
type `email_template`. A name only the email service's built-in
templates hold resolves to that baseline (D3 (b), `via: baseline`), so
the row binds to `com.objectstack.service.email` (seat Q6).

For every reference:

- One package holds the name: the column is set to that package, by id.
Nothing else on the row is written.
- The rule refuses (two or more holders): the column stays null, and a
warning names the name, its row count and every candidate. There is no
first-registered fallback.
- Only the environment or the platform holds it, or nothing does: the
column stays null and the name is reported.
- A row that already names a package is never read again, so a second
boot writes 0 rows and reports the same names.
- If a table cannot be read, a warning says so, its rows stay as they
are, and the next boot tries again. The boot goes on.

### The dispatch key names its flow's package (R10)

The `sys_flow_dispatch` key (its primary `id`) is built in
`packages/triggers/trigger-schedule`. A flow that one package holds is
now keyed

    schedule:PACKAGE:FLOW:WINDOW_START
    time-relative:PACKAGE:FLOW:SCOPE:RECORD_ID

A flow the environment holds, or nothing holds, keeps the bare shape
`schedule:FLOW:WINDOW_START` / `time-relative:FLOW:SCOPE:RECORD_ID`. A
flow name is snake_case and a window start or scope begins with a date,
so the shapes never collide. The package comes from the same
`flowHolder`, through the new `AutomationEngine.flowDispatchHolder`, so
an overridden P flow keys as P. Because the key is the primary key, a
package-less key is still one row per window. S1's NULL-distinct problem
does not arise, and no index changes.

The seat's condition is that a window never fires twice across the
change. Three mechanisms meet it:

1. **The boot re-key, and the ledger held until it finishes.** At
`kernel:bootstrapped`, each bare-shape key inside the ledger's 30-day
retention (by `created_at`, the field the Reaper ages by) whose flow one
package holds gets a copy under the scoped key. The copy keeps the same
`dispatched_at`, `outcome` and `settled_at`. The bare row stays: the
ledger never deletes, and the Reaper owns deletion. A second run finds
every copy present and writes nothing. If a scoped key already exists (a
claim got there first, or another replica's re-key did), it is reported
`present` and never overwritten.
From `start()` until that run finishes, `ObjectStoreFlowDispatchStore`
holds every `claim`, `settle` and `read` (`holdUntil`). The hold is
released when the run settles either way, or after 120 s, which is
logged. So a trigger's first claim or replay-guard read after an upgrade
sees the copied rows. A kernel that dispatches no boot-phase hook holds
nothing and runs no re-key.
2. **The bare claim first (one arbiter across a rolling upgrade).** A
node of an older release claims only the window's bare-shape key. A node
of this release now inserts the bare key first and the scoped key
second, in one claim (`ScheduleTrigger.claimDispatch`,
`TimeRelativeTrigger.claimDispatch`; the replay guard's re-dispatch runs
through the same claim). Whoever inserts the bare key wins the window,
on either release and in either order, and two nodes of this release are
arbitrated by the same insert. A bare row an older node wrote, or the
re-key copied from, loses the window, with nothing claimed under the
scoped key. The schedule trigger settles the bare claim beside the
scoped one, so an older node's replay guard, which reads only the bare
key, reads the window as delivered.
3. **The transition reads.** The schedule replay guard reads both keys
(delivered under either one is refused), and
`AutomationEngine.recordMissedRun` takes `legacyDispatchKey` (a claim
under either key is `dispatched`, not missed).

**The collision guard.** `AutomationEngine.flowDispatchHolder(name)`
answers `{ packageId, claimBare }`. When two or more packages hold the
flow, `claimBare` is false and the key is scoped to the armed body's
package: the bare key would be one row for every holder, the
cross-package collision R10 removes, so it is neither claimed nor read.
That case is unreachable in this release, because `registerFlow` refuses
a second package's same-named flow (objectstack-ai#31754), so the pins drive it
through a doubled holder answer.

**Retiring the bare claim** is a later card, independent of D9: with the
guard in place the bare claim stays safe once D9 narrows the flow
backstop. The transition reads retire once the ledger's retention (30
days) has passed since every node runs this release.

### R14: no change. D8's note is met on `main`

E7 (objectstack-ai#31679, PR objectstack-ai#31730) already keys `sys_job` by runtime's `jobKeyFor`:
the first holder keeps the bare name, a second holder gets
`PACKAGE_ID:NAME`, and E7 never attributes a bare row to a package. The
run history stays as written. Nothing in this PR touches `sys_job`,
`jobKeyFor` or service-job. The pins are E7's and objectstack-ai#8578's, and both pass
on this head:

- `packages/runtime/src/app-artifact-handlers.jobs.test.ts`, the
describe "ADR-0150 E7: a job's package coordinate keeps the key the job
catalogue holds for it, across a restart": the key shapes. 41 passed.
- `packages/platform-objects/src/audit/sys-job.global-unique.test.ts`:
one unique index on `(name)`, `'global'`, byte-identical physical shape.
10 passed.

## What does not change

- **No reader reads the new columns.** The approval revise and resubmit
checks, resume, the credential channel, the connection door and the v18
ceremony all read the name as before.
- **No unique key changes** (seat Q2 A). Every index on the five tables,
and on `sys_flow_dispatch`, is pinned byte-identical to main.
- **Writers do not stamp the columns** (seat Q7 A). A row written after
boot stays null until the next boot binds it. R10's key writers are the
exception, because they are the key change. Hand-off, one per reader
child:
- R7 `process_package`: the flow child B, at
`ApprovalService.openNodeRequest` and its readers (`assertReviseEdge`,
the resubmit window, `attachFlowSteps`, the stranded sweep, the labels).
- R8 `flow_package`: child B, at `ObjectStoreSuspendedRunStore` (write
and `listByFlow` / resume).
  - R9 `flow_package`: child B, at `flow-credential-channel.ts`.
- R11 `connector_package`: the connector reader child, at
`connector-connections.ts` and `connector-credentials-ui.ts`.
- R20 `package_id`: the v18 promotion ceremony's reader (cli
`organization-ownership-plan.ts` / `-apply.ts`).
- The ambiguous set's operator report and a reviewed bind command: S4
(D10).
- No `packages/spec` change, and no metadata shape an author writes. Two
new package-root exports from service-automation, `bareDispatchKeyFlow`
and `scopedDispatchKey` (the key grammar the re-key reads, imported by
trigger-schedule's tests), and three new public engine methods,
`flowHolder`, `flowHolderPackage` and `flowDispatchHolder`. The seat
ruled the first four names stand; `flowDispatchHolder` is new in this
round.

## ADR-0150 readings

- **(a) The phase-1 one-holder tests stay green.** objectql
`registry-bare-name-one-holder*.test.ts` and
`package-scoped-resolver.test.ts`: 5 files, 231 passed. core
`package-scoped-resolution.test.ts` and
`security/security-catalog.test.ts`: 2 files, 50 passed. Run on
`50bfaf16`.
- **(b) Bare-name behaviour is unchanged for every name with one
holder.** No reader reads a new column. The one key that changes shape
is R10's, which D8 names. Every trigger, replay and missed-run answer
for a window is pinned the same across the change (a bare claim still
holds its window). Full suites on `50bfaf16`: service-automation 268
files, 3544 passed; trigger-schedule 12 files, 252 passed (both on
`e3251a85`); plugin-approvals 116 files, 1507 passed; plugin-email 51
files, 834 passed; platform-objects 120 files, 1934 passed.
- **(c) The migration is re-runnable and never guesses.** For every
reference and for the re-key, a second run writes 0 rows (`update` /
`insert` never called), and the dogfood's third boot writes no dispatch
row and moves no bound row. A name two packages hold stays null, or
keeps the bare key, and is reported naming both. Each is ablated below.
- **(d) Clause-②: no.** Nothing accepted on main is refused or newly
accepted. The dispatch key is internal to the ledger (no route, no API,
no export reads it), so no externally visible identifier changes. No
protocol change, so no ADR-0087 migration entry.

## Tests

New tests are added to existing files where noted; no existing assertion
was changed.

- `service-automation/src/stored-reference-package-column.test.ts` (21,
new). It runs on a real `ObjectQL` over `SqlDriver` (better-sqlite3) and
a real `SchemaRegistry` asked through `flowNameResolver` /
`connectorNameResolver`, as the plugin wires them. Covers:
- R8: the paused-only bind, decided rows untouched, the preset column
never moved, two holders null with both named, the environment's and the
unheld names reported.
  - **A paused run of an overridden P flow bound to P.**
- Second run 0 writes, dry run, R9, R11, the boot pass (only carried
tables; a failing table not fatal).
- R10: key grammar; the re-key copies in-retention keys only, keeps
every bare row, **keys an overridden P flow as P**, writes 0 on a second
run, and never overwrites a scoped key a claim took first.
- The store hold: claim and read wait for the re-key and then see the
copy; the 120 s bound releases; CONTROL with no hold.
- `flowDispatchHolder`: one package, the environment's and nobody's
flow, and the collision guard for a name two packages hold (armed
package doubled).
- The missed-run transition read; every index byte-identical; the
columns nullable and read-only.
- `trigger-schedule/src/schedule-dispatch-claim.test.ts` (+14 in three
new describes, existing file): the scoped key agrees with the re-key's
grammar; a tick claims the bare key first, then the scoped key, and
settles both; an environment flow keeps one bare claim; CONTROL with a
ledger without `flowDispatchHolder`; a settled and an unsettled bare
claim both hold the window; the replay guard and the missed-run check
read both keys; **an older node and a node of this release fire the
window once in either order; an older node's replay guard reads the new
node's delivery; two nodes of this release fire once; a name two
packages hold fires once per package and writes no bare row**.
- `trigger-schedule/src/time-relative-trigger.test.ts` (+8 in one new
describe, existing file): the key builder, the bare-then-scoped claim,
the bare key for no single holder, an older node's bare claim holds the
record, **an older node and a node of this release fire the record once
in either order, two nodes of this release fire once, a name two
packages hold fires once per package with no bare row**, and CONTROL. It
reuses the file's own doubles.
- `plugin-approvals/src/stored-process-package-backfill.test.ts` (5,
new): open statuses only, an overridden P flow's request bound to P,
refusals, second run 0, indexes byte-identical.
- `plugin-email/src/stored-template-package-backfill.test.ts` (3, new):
a baseline-only name bound to `com.objectstack.service.email`, an app's
override of a baseline bound to the app, refusals, second run 0, indexes
byte-identical. Forced: `@objectstack/driver-sql` added as a
devDependency of plugin-email (lockfile +3 lines) for the real engine.
- **Forced edit to an existing object, named:**
`sys_approval_request.searchableFields` gains `process_package` beside
`process_name`. `sys-approval-request.search-fields.test.ts` (objectstack-ai#24625)
holds the declared search set to the ADR-0061 auto-default minus the
snapshot, so a new text column must be searched or that pin goes red.
The test file itself is unchanged. The column is platform-filled and
holds a package id; a `$search` for a package id now also matches the
open requests bound to it. This is the one reader-visible change outside
R10's key, and it is in `open_questions`.
- Forced config edits: plugin-approvals' `vitest.config.ts` declares
`OS_REGISTRY_LOG: 'warn'` (`check:registry-log-declared`: the new suite
boots a real `SchemaRegistry`), and plugin-email's aliases
`@objectstack/driver-sql` to source (`check:test-source-alias`).
-
`packages/qa/dogfood/test/automation-stored-reference-package-column.dogfood.test.ts`
(5, new). The showcase boots across restarts of one SQL file:
- **A probe plugin that starts after the automation service claims the
scoped key without waiting. Its claim answers only after the boot's
re-key, and reads the window as already claimed** (seat condition 1, on
a real boot with an in-retention old-key row).
- The scoped copy keeps `dispatched_at` and `outcome`; the bare row
stays; a key of an undeclared flow and a key outside retention are not
copied.
  - R8, R7 and R20 rows bind to the showcase and to the email baseline.
  - A third boot writes nothing.

## Ablations

Each mutation was applied to the committed file with
`scripts/ablation-replace.mjs`. The named tests went red, and the file
was restored to its HEAD blob with `git diff HEAD` empty.

| Mutation | Red |
|:--|:--|
| schedule `claimDispatch`: drop the bare-first claim | 6, including "an
older node first, then a node of this release: the window fires once",
"a node of this release first, then an older node …" and "an older
node's replay guard reads the window a node of this release delivered as
delivered" |
| time-relative `claimDispatch`: drop the bare-first claim | 3,
including "rolling upgrade: an older node and a node of this release
fire the record once, in either order" |
| schedule `dispatchKeys`: drop the collision guard (`claimBare`
ignored) | "the collision guard: a name two packages hold fires once per
package and writes no bare row" |
| time-relative sweep: drop the collision guard | "the collision guard:
a name two packages hold fires the record once per package and writes no
bare row" |
| engine `flowDispatchHolder`: a refusal answers `claimBare: true` |
"the collision guard: a name two packages hold never claims the bare key
…" |
| schedule replay guard: read the scoped key only | "the replay guard
reads both keys" |
| engine `recordMissedRun`: ignore `legacyDispatchKey` | "a window
claimed under its bare key … is dispatched, not missed" |
| `storedReferenceHolderOf`: a refusal takes the first candidate | 8
tests (R8, R9, R11 two-holder pins, second run, dry run, the re-key's
ambiguous report) |
| `flowHolder` answers from `armedFlowPackage` (the caller reading) | 11
tests, including "a paused run of an overridden P flow is bound to P"
and the re-key's "keys an overridden P flow as P" |
| the store hold removed (`holdUntil` sets nothing), unit | "a claim, a
settle and a read made while held answer only once the re-key settles",
"a re-key that never finishes releases the ledger at the bound" |
| **the store hold removed, dist leg on the real boot** (rebuilt;
`ablation-dist-preflight` saw the marker in 2 built files, then absent
after restore) | the dogfood "a claim made during the boot answers after
the re-key" (`expected true to be false`: the probe minted the window
again) and "gains its scoped copy" (the scoped row carries the probe's
new `dispatched_at`) |

## Acceptance notes

- **Surface.** Every changed source file is in the amended surface.
These are outside it, each forced:
- The dogfood test: the seat asked for the ordering pin "on a real
boot".
- `pnpm-lock.yaml` and plugin-email's `package.json`: the devDependency.
- The i18n bundles of platform-objects and plugin-approvals
(`check-i18n-bundles --write`, with Chinese for the two new labels).
- The tenant-audit census and its page: the four boot-pass writes are
four more engine write sites, each with an elevated system context and
an object chosen at run time; after merging main (S2 objectstack-ai#31872 included),
279 to 283 sites, 156 to 160 elevated, 82 to 83 undecidable
(`tenant-audit-census.mjs --write` plus the hand-written figures).
- **No rolling-upgrade residual.** The seat's earlier acceptance of one
is withdrawn (epic review, R1): the bare-first claim gives old and new
nodes one arbiter, so the changeset's "no window fires twice across the
change" holds as written. The one case that claims no bare key, a flow
two packages hold, is unreachable under phase 1 and named in the
changeset.
- **The key grammar is spelled twice.** The writer (`trigger-schedule`)
and the reader (service-automation's `bareDispatchKeyFlow` /
`scopedDispatchKey`) each spell it, held equal by tests, because
trigger-schedule has no runtime dependency on service-automation.
- **A resolver that throws keys by the bare shape.** If
`flowDispatchHolder` (or the resolver behind it) throws, both triggers
fall back to the bare key and one claim, as before the package segment
existed. For flow's B card: once a reader binds by package, that
fallback is where a resolver failure would show.
- **Step 0 and counts** are in the card thread (`31684-step0.md`). On
first-party boots every reference has 0 rows except R14's one
`approvals-sla-escalation` job, and there are 0 ambiguous names.
- The keyset walk and report are written per package rather than shared,
as S2 did: sharing them would need a new cross-package export. The email
walk pages by id by hand, because plugin-email does not depend on
`@objectstack/types`.

## Verification

- Gates: `node scripts/pm/dispatch-gates.mjs --commands` listed 119
commands on `50bfaf16`, and the same 119 on `e3251a85` (the R1 round),
all exit 0 on both, `--ran` 119/119, 0 not measured; all 119 exit 0
after a full build, re-run in full after merging main (the first round,
before the build, had two `PREREQUISITE NOT MET` exits and three
findings, fixed in this branch: a test stand-in carrying `update`,
plugin-approvals declaring no `OS_REGISTRY_LOG`, and plugin-email
importing `@objectstack/driver-sql` unaliased). `--ran` reconciles 119
derived, 119 run, 0 not measured, 0 unrun.
- `eslint --no-inline-config` on the 23 changed source and test files:
exit 0 (the six files the R1 round touched re-linted on `e3251a85`: exit
0; `tsc --noEmit` of trigger-schedule, service-automation and its test
config: exit 0).
- Typecheck (`tsc --noEmit`) of service-automation, plugin-approvals,
plugin-email, trigger-schedule and platform-objects: exit 0 each. The
test configs of service-automation and plugin-email: exit 0.
plugin-approvals' test config reports 321 errors, all in test files this
PR does not touch (0 in the new one).
- Dogfood files re-run on `e3251a85` under `os-verify-lock`: 12 of 12
(this file 5, S2's 7), VERDICT command-exit 0.
- Dogfood file on `50bfaf16`, through `os-verify-lock` after building
its closure (turbo 70/70): 5 of 5 passed, run beside S2's dogfood file
(7 of 7) after the merge (lock VERDICT command-exit 0). CI on
`50bfaf16`: every check run completed green or skipped.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01ADHkmQCZK8QWzn7ZRaop1P

<!-- signed: domain:server#3 · os-steve ·
session_01ADHkmQCZK8QWzn7ZRaop1P -->

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: objectstack-fleet[bot] <332303061+objectstack-fleet[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runtime/service-job: jobKeyFor on the package coordinate, keeping the bare key readable (ADR-0150 child E7)

1 participant