Skip to content

Rust #325: repair a run without a recoverable task as published PHP does - #887

Merged
rmcdaniel merged 3 commits into
mainfrom
rust-325-repair-recovery
Oct 30, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
rust-325-repair-recovery

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Part of #325.

What changes

Native's repair command (POST /api/workflows/{id}/repair) refused with 422 development_workflow_task_repair_unqualified in two cases:

  • a run whose summary needed repair but had no task to recover;
  • a run with an expired activity-task lease.

The repair pass created only a missing workflow task. Following published PHP's TaskRepair::repairRun:

  • Missing tasks. A missing workflow, activity or timer task is created again by task_repair::create_missing_task, which the command and the pass now share:

    • an activity task for the pending execution, with its latest retry's fields and its attempt_count;
    • a timer task, available at the timer's fire time;
    • otherwise a workflow task.

    Each counts one repair, and the run's open activity or timer wait moves to the new task.

  • Existing tasks. The command recovers an existing task through the pass's recover_task_row. A running activity's expired lease is settled by its retry policy in the command's own transaction (settle_lost_lease), and the repaired task is the enforcer's next task.

  • Still refused. The missing task of an update, signal or child wait, or of a condition or signal wait's timeout timer.

Evidence

I ran six cases against published 2.11.7 and against native, both on SQLite, with the maintenance loop stopped. Each run was set up over HTTP, then its task row was deleted or its lease expired in the server's own database:

  • a missing workflow, activity or timer task;
  • an expired workflow lease;
  • an expired activity lease, with attempts left and on its last attempt.

PHP records every case as repair_dispatched. Native matches on:

  • the task rows (type, payload, repair_count, attempt_count, availability, dispatch);
  • the RepairRequested events;
  • the run's description afterwards.

One exception: native's background lease enforcement settles an expired activity lease before a later command can, so the command then answers repair_not_needed. PHP with its repair loop running does the same. The native tests switch off background maintenance to cover the command's own path.

No shared fixture: these states are reachable only by editing the server's database, which a probe cannot do on every backend. The repair-pass fixture avoids real candidates for the same reason. The details are in the checkpoint.

Checks

  • New tests:
    • repair_creates_a_missing_workflow_activity_or_timer_task_like_php;
    • repair_settles_an_expired_activity_lease_by_its_retry_policy_like_php.
  • Rust suites: library 363 and integration 644, all passing.
  • Node unit suite: 6431 pass. The PhpImageContextTest local false failure remains. Two heartbeat cleanup tests failed under build load and passed when rerun alone.

🤖 Generated with Claude Code

rmcdaniel and others added 2 commits October 30, 2026 21:29
Native's repair command refused a run whose summary needed repair but had
no task to recover, and an expired activity-task lease, with
development_workflow_task_repair_unqualified; the repair pass created
only a missing workflow task. Following TaskRepair::repairRun:

- a missing workflow, activity or timer task is created again
  (create_missing_task, shared by the command and the pass): the activity
  task for the pending execution with its latest retry's fields, the
  timer task at its fire time; the run's open wait moves to the new task;
- the command recovers an existing task through the pass's
  recover_task_row, and settles a running activity's expired lease by its
  retry policy in its own transaction, repairing the enforcer's next task.

Recorded against published 2.11.7 over HTTP with the task row deleted or
its lease expired in the server's SQLite database; native matches on all
six cases. Missing update, signal and child tasks stay refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PostgreSQL refuses to drop a database that the runtime's pool still has
open, so the per-case databases are now removed after the runtime closes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 30, 2026
…test

Native's timer ticker settles expired activity leases every 250 ms
whatever the maintenance setting, so on a slower MariaDB cell it settled
the test's lease before the repair command ran. The test now asserts the
lease is settled exactly once by the retry policy, and accepts either a
dispatched repair of the next task or no repair needed, as the repair
pass test already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit d41917d Oct 30, 2026
58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant