Skip to content

process(poc-build): one end-of-build journey walks every rule, 2 runs instead of 3 cold (#770) - #772

Open
MendixMau wants to merge 2 commits into
masterfrom
triage/770-poc-streamline
Open

MendixMau wants to merge 2 commits into
masterfrom
triage/770-poc-streamline

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

Fixes #770

Direct lane

What changed and why (one paragraph): this changes the test cadence in skills/poc-build.md, as approved in #770.

  • Per page: one LOOK at 1280 beside the wireframe and one polish round. The 390 LOOK runs on the first page only (was: 1280 + 390 on every page, two polish rounds).
  • Once at the end: one journey with the happy path, one negative step per business rule and per must-have, and one cross-user refusal per role. Its last step logs out.
  • Runs: the journey runs 2×: warm, then one restart, then cold on the used DB. This replaces the 3 cold runs. After that comes the broken control without a restart, then one LOOK pass at 1280 + 390 over every page.
  • "Done when" now says "2 of 2 runs pass" and lists the logout ending, the negative and cross-user steps, and the final 1280 + 390 pass.
  • Status stays EXPERIMENTAL, with one added line saying the cadence changed after both measured builds and is unmeasured.

The per-role cross-user refusal paragraph under the must-have list moved into step 8; it was not deleted. A few sentences elsewhere were condensed (the page pre-flight step names, the commit rule wording, the Nothing-path design questions) to keep the length flat. No rule was removed.

Session-leak assertion: a follow-up, not built here. The issue asks to assert an open-session count of 0 after each run.

  • project-tests/e2e/journey-runner.js already counts System.Session through the admin API, but only in its preflight (ADMIN_PROBE_SQL, classifyAdminProbe, JOURNEY_SESSION_CAP, around :1405–1550).
  • The POC journey is "one script (not the journey-proof.md runner)". A runner flag would only reach it if the POC moved onto the runner, which is a wider change.
  • "0 sessions right after logout" cannot be proven without a live runtime: session removal timing and anonymous sessions are unknown, and CLAUDE.md "Shipping an instrument" rule 4 needs a field run.

So the skill instead requires the journey to end with a logout step and run 2 (cold, after a restart) to pass. A leak then shows up as run 2 failing at login. Suggested follow-up: an opt-in post-run probe that reuses classifyAdminProbe, field-proven on a trial runtime.

Impact

Row Answer
Process POC builds only (Build type: poc). Removed: 2 of 3 cold restarts, the third journey run, the second per-page polish round, and the per-page 390 LOOK after page 1. Added: a negative step per business rule and per must-have, a logout as the journey's last step, and a final LOOK pass at 1280 + 390 over every page. Nothing new blocks. No new stage artifacts.
Dependencies skills/poc-build.md and CHANGELOG.md only. No shared scripts, no skill-routing.tsv change. No fixture targets poc-build.md (grep of tests/). No new artifact reads.
Context cost none for always-loaded text: poc-build is ondemand in skill-routing.tsv. bin/render-routing.sh --check: baseline 66196 → 66196 words (budget 80000) on both master and the branch. The skill file goes from 2144 to 2178 words (+34).
Behaviour NOT RUN: no A/B of the new cadence. It needs the 2-build POC comparison re-run, which is why the status line now says unmeasured. Maintainer to accept or request an A/B.
Runtime Estimated in the issue, not measured here: about 10–15 min saved on a 36–64 min build (2 fewer app restarts, 1 fewer full journey run, about 1 polish round and one 390 LOOK fewer per page after the first).

Time cost: per project · not measured (est. −10 to −15 min, from the issue) · can force rework: y (phone-layout bugs on pages 2+ surface only in the final pass) · offset: removes 2 cold restarts, 1 journey run, per-page second polish round and 390 LOOK

Field evidence:

Proof (grep check, text-only change): a scratch script runs 11 assertions against the skill file. They check for:

  • no "3 cold runs" or "3 of 3 cold";
  • no "two polish rounds";
  • the "2 runs: warm, then one restart, then cold on the used database" rule;
  • "Journey: 2 of 2 runs pass";
  • "one negative step per business rule and per must-have";
  • "one cross-user refusal per role";
  • the logout ending, in step 8 and in Done when;
  • "broken control, no restart";
  • per page: one LOOK, one polish round, and the 390 LOOK on the first page only;
  • the final 1280 + 390 pass;
  • EXPERIMENTAL kept, plus the unmeasured line.

Results:

  • master's skills/poc-build.md: 0 passed, 11 failed
  • branch: 11 passed, 0 failed

Also ran:

  • bin/render-routing.sh --check: in sync, baseline 66196/80000;
  • bin/check-scripts.sh: 137/137 shell, 30/30 Node parse;
  • the leak guard vs origin/master: clean.

No selftest was added, since the runner was not changed.

Review: pending independent review


Generated by Claude Code

… instead of 3 cold (#770)

Per page: one LOOK at 1280 and one polish round; 390 on the first page
only. Once at the end: one journey with the happy path, one negative step
per business rule and per must-have, one cross-user refusal per role,
ending in a logout; run it 2x (warm, then cold after one restart on the
used DB), then the broken control without a restart, then one LOOK pass
at 1280 + 390 over every page. "Done when" matches. Status stays
EXPERIMENTAL and notes the new cadence is unmeasured. Skill 2144 -> 2178
words, on demand (baseline unchanged at 66196).

The open-session-count assertion is a follow-up: the POC journey is its
own script, not journey-runner.js, and a zero-after-logout check cannot be
proven without a live runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE-AFTER-FIX. "The script's last step logs out" is weaker than #729's actual fix, which logs out every session before closing it. A journey that now logs in as several users for the cross-user steps can still leak sessions and hit the 6-session cap. The broken-control run, which is meant to FAIL anyway, can then fail at sign-in and still look like a pass.

Value: Implements #770's approved cadence (skills/poc-build.md:29-30, :68-69): one LOOK and one polish round per page, 390 on the first page only, one end-of-build journey with a negative step per rule and per must-have and a cross-user refusal per role, 2 runs, the broken control, and a final 1280 + 390 pass.

Regression risk:

  • The skill is on-demand and applies only to Build type: poc.
  • No script, gate or fixture reads it (grep tests/ finds nothing), so the change affects only the agent's behaviour on POC builds.
  • The PR states the accepted risk that phone-layout bugs on pages 2+ surface late.

Correctness:

  • In [idea] Light mode: requirements or an idea to a running app in a 2-hour workshop, tested (27 min, $2.41, journey 15/15) #729 finding 3, each run opened 2-3 sessions, and the fix was "mx.logout() per session before close". Step 8 (:30) and Done when (:68) require only a logout at the end, while the new cross-user steps make the script switch users mid-run.
  • "The run must FAIL" for the broken control (:30, no restart, after two runs) does not say where it must fail. A sign-in failure from a leak would count as the expected failure.
  • The claim "a session leak fails run 2 at login" is not measured. The Status line correctly marks the cadence unmeasured (:6).
  • No conflict with origin/master.

Context cost: 66196 → 66196 always-loaded words (measured). The on-demand file grows by +34 words.

Time cost: per POC build, an estimated −10 to −15 min (from the issue, not measured). 2 app restarts and 1 journey run are removed. It can force rework at the end for phone-layout bugs.

Exact fixes required:

  1. Step 8 and Done when: "every session the script opens logs out before its browser or context closes (each user switch included), not only the last step" ([idea] Light mode: requirements or an idea to a running app in a 2-hour workshop, tested (27 min, $2.41, journey 15/15) #729 finding 3).
  2. Broken control: "it must FAIL at the changed message's step; a sign-in failure does not count".

Generated by Claude Code

… at the changed step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review fixes pushed in e94e2a2:

  1. In step 8 and in Done when, every session the script opens now logs out before its browser context closes, each user switch included. Before, only the last step did.
  2. The broken control must now FAIL at the step whose message was changed. A sign-in failure does not count.

Needs a re-review before merge.


Generated by Claude Code

@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE. Re-review: both requested fixes landed in e94e2a2, and nothing else changed:

  • every session the script opens now logs out, each user switch included;
  • the broken control must FAIL at the changed message's step, and a sign-in failure does not count.

Value:

Regression risk: The fix commit touches only skills/poc-build.md (2 lines). The file is on demand and applies to Build type: poc only, and no script or fixture reads it. Merges cleanly onto origin/master.

Correctness: No new issues. The session-count assertion is still deferred, as stated, and the Status line still marks the cadence unmeasured (:6).

Context cost: 66196 → 66196 always-loaded words (measured). The on-demand file grows from 2144 to 2208 words.

Time cost: per POC build, an estimated −10 to −15 min (from the issue, not measured). Nothing per build row.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

POC build: cut repeated test runs, make the one end-of-build journey walk every rule

2 participants