Skip to content

feat(stovepipe): Recover failed issue tracking - #961

Merged
mnoah1 merged 1 commit into
mainfrom
codex/stovepipe-issuetrack-dlq
Oct 28, 2026
Merged

feat(stovepipe): Recover failed issue tracking#961
mnoah1 merged 1 commit into
mainfrom
codex/stovepipe-issuetrack-dlq

Conversation

@prathshenoy

@prathshenoy prathshenoy commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

What:

  • Add dead-letter recovery that stops failed issue-tracking passes while keeping completed issue changes.
  • Record incomplete issue tracking without changing the build's validation outcome.

Why:

  • Unblock later builds when issue tracking exhausts normal retries. Today the unfinished pass remains saved, so later notifications try to resume the same failed work. Recovery needs to stop that pass without throwing away issue changes that already succeeded.
  • Keep delayed workers from changing issue state after recovery. Simply clearing progress would let a paused worker write an old opening or closing after a newer build has run. Recovery protects the current issue and retains its history before allowing newer work through.

Test Plan

  • Add unit tests.
  • Verify recovery progress persists against a real MySQL database.

Revert Plan

  • Revert this PR before enabling consumption. If already enabled, stop consumers and reconcile recovery progress before rolling back so an older consumer does not resume abandoned work.

Issue Links

Stack

  1. feat(stovepipe): Retain versioned issue state #942
  2. feat(stovepipe): Store applied issue state #948
  3. feat(stovepipe): Apply issue cursor updates #949
  4. feat(stovepipe): Track issues in bounded passes #950
  5. feat(stovepipe): Publish issue-tracking updates #951
  6. fix(stovepipe): Keep issues on task-wide failures #958
  7. feat(stovepipe): Recover failed issue tracking #961
  8. feat(stovepipe): Wire issue tracking consumers #962

This was referenced Oct 27, 2026
Merged
Merged
Merged
Merged
Merged
Merged
@prathshenoy
prathshenoy added this pull request to stack #953 October 27, 2026 21:46
Base automatically changed from codex/stovepipe-taskwide-closure to main October 27, 2026 21:59
@prathshenoy
prathshenoy force-pushed the codex/stovepipe-issuetrack-dlq branch from aaa9c09 to 292529b Compare October 27, 2026 21:59
@prathshenoy prathshenoy mentioned this pull request Oct 27, 2026
Merged
2 tasks done
This was referenced Oct 27, 2026
Merged
Open
@prathshenoy
prathshenoy force-pushed the codex/stovepipe-issuetrack-dlq branch from 292529b to bc1421e Compare October 27, 2026 23:00
@prathshenoy
prathshenoy marked this pull request as ready for review October 27, 2026 23:18
@mnoah1
mnoah1 added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 25bc437 Oct 28, 2026
38 checks passed
@mnoah1
mnoah1 deleted the codex/stovepipe-issuetrack-dlq branch October 28, 2026 02:43
DivyanshGarg380 pushed a commit to DivyanshGarg380/submitqueue that referenced this pull request Oct 28, 2026
@prathshenoy
feat(stovepipe): Track issues in bounded passes (uber#950)
078c40d 
## Summary

**What**:
- Process build issues in bounded batches and request redelivery until
the pass finishes.
- Open reported issues during refreshes and close missing issues after
final analysis, except when the build was canceled.

**Why**:
- Avoid doing an entire queue’s issue tracking in one delivery. Saving a
cursor lets later deliveries continue the pass while completing any
interrupted issue and history writes first.
- Keep delayed notifications from undoing newer issue state. Finish an
accepted pass before admitting another build, reject older sources, and
ignore refreshes after finalization; leave consumer activation and
terminal recovery for follow-up work.

## Test Plan

- [x] Add unit tests.

## Revert Plan

- Revert dependent PRs first, then this PR. Issue-tracking consumption
remains disabled, so reverting does not interrupt an active
issue-tracking worker.

## Issue Links

- [CODEM-607](https://linear.app/uber/issue/CODEM-607/build-analysis)

## Stack

1. uber#942
1. uber#948
1. uber#949
1. uber#950
1. uber#951
1. uber#958
1. uber#961
DivyanshGarg380 pushed a commit to DivyanshGarg380/submitqueue that referenced this pull request Oct 28, 2026
@prathshenoy
feat(stovepipe): Publish issue-tracking updates (uber#951)
438ab74 
## Summary

**What**:
- Publish issue-tracking refresh and finalization notifications after
build analysis persists its results.
- Add the publication route without enabling issue-tracking consumption.

**Why**:
- Give issue tracking a reliable handoff from build analysis. Publishing
after persistence ensures it can read the build’s recorded issues, and a
failed publish can retry without losing the handoff. Require a complete
cumulative terminal result so finalization does not depend on an earlier
worker finishing its writes.
- Keep terminal recovery separate from this rollout. Registering only
the publication route allows the processing layers to be reviewed
without starting a consumer whose recovery policy is not yet wired.

## Test Plan

- [x] Add unit tests.

## Revert Plan

- Revert dependent PRs first, then this PR. Issue-tracking consumption
remains disabled, so reverting does not interrupt an active
issue-tracking worker.

## Issue Links

- [CODEM-607](https://linear.app/uber/issue/CODEM-607/build-analysis)

## Stack

1. uber#942
1. uber#948
1. uber#949
1. uber#950
1. uber#951
1. uber#958
1. uber#961
DivyanshGarg380 pushed a commit to DivyanshGarg380/submitqueue that referenced this pull request Oct 28, 2026
@prathshenoy
fix(stovepipe): Keep issues on task-wide failures (uber#958)
125bea0 
## Summary

**What**:
- Keep existing issues open when any task-wide failure is recorded for a
build.
- Apply all reported issues before reconciling absent ones, using the
existing processing phases.

**Why**:
- Avoid treating missing target results as recovery. A task-wide failure
can mean validation never ran or could not identify failed units, so
closing earlier issues would claim recovery without evidence.
- Avoid a separate precheck and new storage operations for data we
already need to read. Recording task-wide failures while applying
reported issues lets the closing pass use a decision established over
the whole build.

## Test Plan

- [x] Add unit tests.
- [x] Verify terminal intent and task-wide failure information persist
against a real MySQL database.

## Revert Plan

- Revert this PR before enabling issue-tracking consumption. The added
columns can remain unused; once consumption is enabled, drain active
passes before rolling back.

## Issue Links

- [CODEM-607](https://linear.app/uber/issue/CODEM-607/build-analysis)

## Stack

1. uber#942
1. uber#948
1. uber#949
1. uber#950
1. uber#951
1. uber#958
1. uber#961
DivyanshGarg380 pushed a commit to DivyanshGarg380/submitqueue that referenced this pull request Oct 28, 2026
@prathshenoy
feat(stovepipe): Wire issue tracking consumers (uber#962)
a158dfb 
## Summary

**What**:
- Enable normal issue tracking and dead-letter recovery in the reference
service.

**Why**:
- Consume the issue notifications that build analysis already publishes.
Until now they accumulated on the queue without updating current issues
or their history. Registering both consumers makes that path
operational, while deployments can still supply their own build-issue
detector.
- Keep failed tracking from blocking later builds indefinitely. Normal
processing uses bounded retries, and recovery stops incomplete work
safely instead of repeating the same failed pass forever.

## Test Plan

- [x] Add unit tests.
- [x] Verify analysis handoff and dead-letter recovery through the
running service and MySQL.

## Revert Plan

- Revert this PR to disable both consumers without deleting persisted
issue state, history, or progress.

## Issue Links

- [CODEM-607](https://linear.app/uber/issue/CODEM-607/build-analysis)

## Stack

1. uber#942
1. uber#948
1. uber#949
1. uber#950
1. uber#951
1. uber#958
1. uber#961
1. uber#962
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mnoah1 mnoah1 mnoah1 approved these changes
@behinddwalls behinddwalls Awaiting requested review from behinddwalls behinddwalls is a code owner
@sbalabanov sbalabanov Awaiting requested review from sbalabanov sbalabanov is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@prathshenoy @mnoah1