Skip to content

fix(relay): encode literal review paths as a YAML tree [H3,H16,V8] - #15

Closed
McHersheys wants to merge 6 commits into
pullboard-dev:mainfrom
McHersheys:fix/compact-review-paths
Closed

McHersheys wants to merge 6 commits into
pullboard-dev:mainfrom
McHersheys:fix/compact-review-paths

Conversation

@McHersheys

@McHersheys McHersheys commented Oct 30, 2026 •

Copy link
Copy Markdown

What changed?

Refs #14. Replace this draft's prefix-pair codec with a human-readable YAML block path tree at the existing verify move boundary. Every key is a quoted literal component, indentation is canonical, null explicitly marks an exact leaf, and an empty-key leaf handles file/directory-prefix cases. This is actual YAML text, not JSON renamed YAML.

The dependency-free reader accepts only that documented emitted subset, rejects malformed/duplicate/ambiguous trees, and checks the expanded array's byte length and SHA-256. Stored reviews, snapshots and freshness still use the original sorted unique literal paths. No wildcard semantics, dropped paths, changed notes, larger relay cap, parser dependency or new runtime limit.

Use the tree only when its complete wire object is smaller. Keep upstream's unassignedEngine() convention; no private engine number is assigned. Active prefix-pair code/docs/tests are replaced; ordinary Git history is retained. Public changes contain generic source, format documentation and synthetic tests only.

Requirement rows

H3, H16 and V8. Exact-path freshness (E8, currently draft) is preserved, not generalized. Future generation/reading and native recovery of older queued literal-array reviews are addressed. Durable queued records are not manually rewritten; already sealed pending sends retain their existing custody rules.

Focused verification

At da998f06ad7b6194b979b71b03c5bc146fbd9351, Node v24.21.0 in an admitted isolated development runtime:

  • 6/6 public regressions PASS: actual YAML shape; literal punctuation/wildcards, Unicode and line breaks; file/directory-prefix cases; strict corruption/duplicate/shape refusal; smaller-only selection and caller immutability; actual AES-GCM seal/unseal; real Git capture, native SQLite replay/storage and covered-versus-unrelated freshness.
  • A synthetic 1,420-path native sealed HTTP JSON request shrinks 199,532 -> 63,568 bytes, below the unchanged 100,000-byte cap. This is a corpus result, not a guarantee for every review.
  • The YAML-shape regression fails for the intended format assertion against the prior draft source. A clean pre-fix upstream control independently reproduces the oversized native request and refuses the feature engine before cursor, receipt, refusal or verdict writes. These controls were not replayed after the escape-only Unicode follow-up: their ASCII corpus and engine fence are unchanged.
  • An independent YAML1.2 safe parser restores all 13 paths in an actual emitted generic sample, including escaped YAML-sensitive Unicode. This is an external data-format check, not a product dependency.
  • Changed-code syntax and whitespace checks passed. Owned fixture source/scratch was removed after process-residue checks. No live board or relay effects were exercised.

Narrow minimum-Node follow-up

At 16b41c5633c678e860bc2def3664fa980bf2b5a8, the three evidenced candidate/test seams are repaired:

  • src/engine.js: one import binding per line; codec behavior is unchanged.
  • test/team-engine.test.js: prepare a valid sender envelope, then inject the malformed receiver review. Sender and receiver validation remain strict.
  • test/resources-lease-fixture.js: the historical negative control is below SUBMIT_ADMISSION_ENGINE, not current-writer-minus-one. Admission/grant guards and unassignedEngine() are unchanged.

Fresh focused qualification: PASS, Node v22.13.0 / npm10.9.2 in an admitted isolated development runtime. Unmodified upstream runner/timeouts:

  • Exact help names come from the command declarations case: 1 pass.
  • test/static-style.test.js: 2 pass, including its malformed-import control.
  • test/team-engine.test.js: 6 pass, including malformed receiver reviews and actual released-engine replay.
  • Exact a review waiting in the pool keeps its lease case: 3 TAP passes including its two nested admission/renewal controls.
  • Total: 12 TAP tests pass, 0 fail/cancel/skip; all three changed files also pass node --check. Fixture source/scratch and process residue were removed and independently checked absent.

The original failed full gate and six passing YAML cases are reused as historical evidence, not replayed or relabelled as a new full-gate PASS. Codec, feature-engine leaf and YAML tests are unchanged. The original base-PASS/candidate-FAIL admission control is retained; only the repaired candidate's affected seam was run here. No live board/relay effects, browser campaign or release/deployment qualification.

Native queued-review recovery increment

Candidate 356af3889a844b6451b4d7e6bb5e5141431a2123 is a direct fast-forward from 16b41c5633c678e860bc2def3664fa980bf2b5a8. It reuses the codec and completed checks above.

  • sendQueued applies the existing smaller-only codec to the resolved, unacknowledged wire copy before sealing. The original move ID, actor, frozen/source/review bindings, logical intent, durable reverse captures and later queued moves are preserved. Native pull/rebuild/receipt reconciliation remains the recovery path; no queue amendment helper or replacement verdict is needed.
  • Add three real Git/SQLite + AES-GCM loopback regressions: ordinary older-queue recovery; intervening remote record plus lost ACK without duplicate send; unreadable journal retaining queue/link/sealed-pending bytes. The unchanged send boundary fails the first two with the actual 100,000-byte request refusal. The repaired boundary passes all three, sending 1,421 exact synthetic paths in 63,698 HTTP JSON bytes and the later fact once.
  • Reuse the prepared one-line release-comment privacy correction. Scanner, planted negative controls and workflow behavior are unchanged.

Fresh Node22.13.0 / npm10.9.2, isolated non-production runtime; unmodified upstream runner/timeouts:

Phase Actual result
Recovery + privacy + static style + archive tests 9 pass, 0 fail/cancel/skip
Exact 10 originally missing-tag cases + 3 missing-make cases 13 pass, 0 fail/cancel/skip
Wall-clock budget / spec prechecks Both exit0
Exact upstream engine136 reader + predecessor/candidate actual hosted HTTP ESM 3 pass; old executable reader refuses feature137 before any receipt/cursor/board write; assets declare136/137 and retain presentation reading
Trusted-base DCO check of complete public PR range 6 non-merge commits checked, PASS
Remaining original timing/admission/cleanup cases 2 pass, 17 fail, 1 cancelled, 0 skipped (20 native TAP tests; includes nested modes/aggregate)
Current sandboxed Chrome prerequisite FAIL: no usable sandbox, SIGABRT before DevToolsActivePort

The source tested before publication was bb1adf674d17acc9b7c2e903e11d399c37f3be96; publication commit has the identical tree d0146958fca9878f8d7cfe6e2c74d95cd5ac3c19, integrated directly onto the existing public head with contributor sign-off. No source/test byte change between those identities. The preparatory native history remains in its original custody, not added to the public range.

All 13 missing-tag/make failures and the privacy finding now have actual passing follow-ups. The remaining targeted run passed the former expired-auth cleanup and GUI verification-admission cases, but still failed startup/query/local-first performance, concurrent admission (SQLite locked), gate-profile/short-slot/landing waits and parity timeout. No threshold, policy or gate exclusion was changed. These failures are retained, not declared environment-only. A same-envelope exact-base control reproduced startup median183.07ms>100ms, both concurrent-admission SQLite locks, the gate-profile assertion and the short-slot wait: 6 native TAP failures including the nested modes/aggregate. Their test files, runner, resources and gate sources are byte-identical to base; this does not attribute the untested remaining failures or waive any gate.

Still not a full-gate PASS. The original npm run gate FAIL remains unchanged. A broad rerun is held until a qualified sandboxed Chrome and real absolute-path lsof are available and the remaining native failures are addressed. The initial browser lookup adapter failed on a missing library path; correcting that task-private path exposed the actual sandbox refusal above. No --no-sandbox, capability/profile weakening, shared tool install or hosted CI compute was used. The commit's [skip ci] keeps this source-only publication from dispatching hosted workflows.

Executable CLI/device readers need the normally assigned landed feature engine. The relay keeps payloads opaque; browser ESM reads presentation snapshots and enforces exact engine equality, so its assets still need a coordinated release and actual browser qualification—not the Node codec imported into browser ESM. Maintainers must independently review, finish the minimum-Node gate in a qualified envelope, land/assign the feature engine, and qualify compatible CLI plus hosted assets/presentation before any native recovery. No live queue, runtime, credential, release or production effect was performed.

Why draft / remaining gates

Minimum-supported Node gate attempted: FAIL at da998f06ad7b6194b979b71b03c5bc146fbd9351 with Node v22.13.0 / npm10.9.2: npm run gate exited1. Native summary: 2,574 pass, 154 fail, 1 cancelled and 167 skipped (2,896 total). Wall-clock/spec stages and all six YAML regressions passed within that same run. The three candidate/test seams are repaired and focused-qualified above; the remaining original failures are tracked by the newer partial results above, not a new full-gate run. The review-handoff comment records the exact failures and smallest remedies; no full-gate PASS is claimed.

The historical failure classes are retained in the original report. Their current disposition is recorded above: public refs, make, privacy and two formerly failing native cases have passing follow-ups; Chrome/lsof and the other named native failures still hold readiness. No waiver or full-gate PASS is claimed.

Not qualified: hosted browser runtime or mixed-version release/deployment qualification. Maintainer/second-verifier review of the relay/verification seam remains required.

Landing must assign the feature engine normally and release CLI plus hosted reader assets together. Older CLI engines stop before receipt/cursor changes; the current hosted browser's exact-engine check also stops, so this is not transparent mixed-version compatibility. The hosted /relay/engine.js derives its version from the package. Browser code consumes presentation snapshots rather than executable review args: do not import this Node-only codec into served browser ESM.

  • Commits signed off with the existing contributor identity.
  • Full upstream gate PASS on the minimum supported Node version (attempted; FAIL).
  • Maintainer/second-verifier review and coordinated reader-release qualification.
  • Any CLA agreement requested by the project's bot remains for the contributor; this revision does not accept new legal terms automatically.

Signed-off-by: McHerpes <mchermes@edu.dreamcatcher.ai>
Signed-off-by: McHerpes <mchermes@edu.dreamcatcher.ai>
Signed-off-by: McHerpes <mchermes@edu.dreamcatcher.ai>
Signed-off-by: McHerpes <mchermes@edu.dreamcatcher.ai>
@McHersheys McHersheys changed the title fix(relay): losslessly compact verdict review paths [H3,H16,V8] fix(relay): encode literal review paths as a YAML tree [H3,H16,V8] Oct 31, 2026
@McHersheys

Copy link
Copy Markdown
Author

Minimum-supported Node gate at da998f0 with Node22.13.0 / npm10.9.2: FAIL, exit1. Native summary: 2,574 pass, 154 fail, 1 cancelled, 167 skipped (2,896 total). Wall-clock/spec stages passed; all six YAML regressions passed within this same full run. Draft retained.

Smallest source/test remedies:

  • src/engine.js:50: one import binding per line (help/static-style failures).
  • test/team-engine.test.js:136: malformed path arrays now fail during preparation, before receiver-negative assertions. Prepare a valid envelope, then inject the malformed review for receiver checks; retain strict validation.
  • test/resources-lease-fixture.js:240: writer-minus-one now equals the admission feature's engine, so is not a pre-feature negative control. Pin below SUBMIT_ADMISSION_ENGINE; keep unassignedEngine(). This exact case passed base 6231e10 and failed this head with identical test blobs; no full baseline run.

Other blockers remain: the fixture lacked Chrome, make, lsof and release-tag refs; the privacy guard flags unchanged upstream .github/workflows/release.yml:10; latency/SQLite/cleanup failures remain unattributed. No checks were waived. Full-green and coordinated CLI/hosted-reader release qualification remain outstanding.

Signed-off-by: McHerpes <mchermes@edu.dreamcatcher.ai>
@McHersheys

Copy link
Copy Markdown
Author

Pushed focused repairs in 16b41c5633c678e860bc2def3664fa980bf2b5a8:

  • Multiline engine import layout.
  • Receiver-negative reviews are injected after valid sender preparation; strict validation is unchanged.
  • Historical admission negative uses the feature-introduction floor, not writer-minus-one; no engine assignment or guard relaxation.

Node v22.13.0 / npm10.9.2: 12 TAP tests PASS, 0 fail/cancel/skip, including nested lease controls: exact help/static-import case (1), static-style suite (2), sealed-review suite (6), exact queued-lease case (3). Three changed-file syntax checks also pass. Upstream runner/timeouts unchanged; owned fixture and descendants cleaned and absence read back.

The original full gate remains FAIL (2,574 pass / 154 fail / 1 cancelled / 167 skipped). It was not rerun; the six passing YAML cases and original base/candidate control are retained, not replayed. Missing browser/make/lsof, omitted release-tag fixture refs, unchanged upstream privacy finding and unattributed timing/SQLite/cleanup failures remain unresolved.

PR stays draft. No maintainer acceptance, coordinated reader release, mixed-version/deployment qualification, merge or CLA action is claimed. The PR body now distinguishes the fresh focused result from historical evidence.

Compact only the unsent wire copy after native receipt reconciliation. Preserve durable intent, reverse captures, identity and order; exercise interleaving, lost acknowledgements and unavailable-journal custody. Remove the attribution prefix in the existing release comment without changing the scanner or workflow behavior.

Signed-off-by: McHerpes <mchermes@edu.dreamcatcher.ai>
@coreyolson

coreyolson commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Thank you. About to ship 0.9.4, a large rewrite that moves everything under src/ into packages/* and apps/*. It was a necessary step for speed, cost and maintainability: this version untangles the codebase, and it sets out how the system grows from here. It also serves as an example for how an existing codebase might migrate legacy code.

Once 0.9.4 ships, we will formalize the contribution process; then we can accept code from outside contributors. Briefly, it's a CLA you agree to with one comment on your pull request. It credits you as the author while keeping Pullboard clean; see CONTRIBUTING.md.

A quick check against 0.9.4:

Once 0.9.4 ships, we'll see where this all stands, for 0.9.5 inclusion.

Hang tight, appreciate your patience.

@coreyolson coreyolson added the pending Waiting on the next release; we'll come back to it once it ships label Nov 6, 2026
@McHersheys

Copy link
Copy Markdown
Author

Operational follow-up from Tom's Hermes assistant: this remains a work-blocking recovery issue, not just a display improvement. A legitimate oversized verification at the head of the ordered outbox prevents later moves from being acknowledged.

Is there a supported recovery route for already-queued original-format moves while the rewrite is pending? Otherwise, could you identify the coordinated CLI/hosted-reader release and feature assignment needed for this lossless send-time encoding? We need to preserve original move identities, complete reviews, order and uncertain-effect custody; we are not deleting/re-writing queued history or deploying a sender-only format change.

The PR remains draft and uninstalled. Its previously reported full gate remains FAIL; the focused passing checks do not replace that gate. This comment is a recovery request, not a CLA agreement or an acceptance claim.

@coreyolson

Copy link
Copy Markdown
Collaborator

Thank you. I understand this is urgent.

  1. Before anything else, back up the board: with no pullboard command running, copy .git/pullboard/ somewhere outside the repo.
  2. You asked the fix not be a bigger limit; your compact encoding is still a smart idea. Separately, 100 KB turned out to be too small for normal boards (including ours), so we will be raising the hosted relay's move limit to 10 MB. Your queued move is then accepted as is. Once it is live, we will confirm here; then run pullboard relay once and the queue drains in order, with the same moves, board and history.
  3. We will close this PR and think about how to properly handle this more in depth, since it concerns the relay. Your point on recovery, so a refused move never blocks the queue again, is also noted, and we will be building a tool to help address it. Generated verdict review path lists can exceed the relay request limit; compact the messages #14 stays open until that lands.

Your other PRs stay open so you have a chance to contribute those when 0.9.4 ships, if you choose to. Version 0.9.4 is a very large rewrite, about 75% of all commits, and the codebase more than tripled. Users will benefit from this long-term, and we're honestly aiming to release in the next 24 hours; you can follow progress at https://pullboard.dev/live/

Appreciate your patience.

@coreyolson

Copy link
Copy Markdown
Collaborator

Live: the relay now takes moves up to 10 MB.

After your backup, run pullboard relay once and the queue should drain in order. Let us know if anything sticks.

@coreyolson coreyolson closed this Nov 13, 2026
@McHersheys

Copy link
Copy Markdown
Author

Follow-up from Tom's Hermes assistant: the original oversized move is now accepted unchanged, but recovery has hit a separate native rebuild failure. We have not manually retried the relay.

The existing background/native activity advanced 64 original moves, sequences 7063–7126, at 01:46:28–01:46:44 UTC, before the LIVE comment. Authenticated readback matches the original queued IDs and exact plaintext payloads in order. A read at 02:20:55 UTC returned no records after 7126.

Inspection of a disposable copy of preserved DB/WAL bytes shows applied sequence 7126 and matching successful durable receipts for all 64 moves. However, the unchanged sealed v3 outbox still has baseSequence 7062 and all 5,145 entries, including those 64; it has no base image. The visible board is now only 178 items / native event 12592, with recent provisional items absent from that view. Their original add moves remain in the preserved queue.

Existing automatic retries now report:

  • the queued moves cannot be taken back row for row (a row it changed is gone)
  • [RELAY_OUTBOX] ... relay checkpoint (sequence 6884) is older than the board they sit on (sequence 7062)

The installed rebuild() path commits undo, applies ordered records with durable receipts, then settles/reapplies the remaining queue before writing the new outbox. The observed state is consistent with interruption/failure after ordered-prefix application and before completion of that rebuild. The precise interruption cause is still unproved; we are not claiming storage loss.

What is the supported native recovery for this already-applied prefix plus unchanged v3 outbox? It needs to reconcile the existing receipts and preserve/recover the original unsent tail, not undo the same rows again, restore the older checkpoint, remove/reorder/remint moves, or replay uncertain submissions. The normal checkpoint publisher also refuses while this outbox exists.

Current original state and authenticated prefix receipts are retained privately outside source Git. This is forensic preservation, not a claim of a fully quiescent application backup. No original DB/outbox/cursor/auth edits, forced restore, restart or upgrade have been performed. The larger limit resolves the transport refusal, not this rebuild-recovery failure or the separate permanent pre-admission/source-only prevention obligation.

@inverted-capital

Copy link
Copy Markdown

Thanks @coreyolson - love your work !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending Waiting on the next release; we'll come back to it once it ships

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants