Skip to content

FIX Count built nodes instead of timing the budget-exhaustion hierarchy test - #3609

Closed
Shivam Sharma (ShivamSharma43) wants to merge 1 commit into
microsoft:mainfrom
ShivamSharma43:fix/hierarchy-budget-test-deterministic
Closed

Shivam Sharma (ShivamSharma43) wants to merge 1 commit into
microsoft:mainfrom
ShivamSharma43:fix/hierarchy-budget-test-deterministic

Conversation

@ShivamSharma43

Copy link
Copy Markdown
Contributor

Description

Fixes #3608.

test_hierarchy_traversal_halts_immediately_on_budget_exhaustion timed build_scenario_hierarchy (1,000 roots, max_nodes=1) and asserted elapsed < 0.05. The call takes about 8 ms, but on CI it intermittently measures 1.6–2.25 s and fails build_and_test on unrelated PRs. #3608 lists today's runs from several contributors.

Changes:

  • The test no longer times the call. It wraps ScenarioProgressReadModel._map_hierarchy_node with patch.object(..., wraps=...) and asserts it's called exactly once: only the node the budget allows is built, and the other 999 roots are never mapped.
  • The existing assertions are unchanged: truncated is True, one root is in the projection, and overall_counts.completed == 1000.

No production code changes.

Tests and Documentation

  • The test passes on main in about 2 s, with no timing dependency.
  • To check that it still catches the regression it guards against, I removed both budget checks in build_scenario_hierarchy (the early return None in _build_node and the break in the root loop). The test then fails with assert 1000 == 1.
  • ruff check, ruff format and ty check pass on the changed file.
  • No documentation changes.

🤖 Generated with Claude Code

…hy test

test_hierarchy_traversal_halts_immediately_on_budget_exhaustion asserted that
build_scenario_hierarchy finished in under 50 ms, which fails on loaded CI
runners (1.6-2.25 s) although the call takes about 8 ms. Spy on
_map_hierarchy_node and assert only the one node the budget allows is built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 08:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@ShivamSharma43

Copy link
Copy Markdown
Contributor Author

Roman Lutz (@romanlutz) Thanks for assigning #3608. This PR's workflows are waiting for approval to run (all five show action_required). Could you approve them? Once this lands, the timing test stops failing #3606 (fixes #3605, approved), which failed it again on its rerun, and other PRs that hit it. #3592 (fixes #3591) is approved and green.

@ShivamSharma43

Copy link
Copy Markdown
Contributor Author

Closing: #3770 already replaced the wall-clock assertion in test_hierarchy_traversal_halts_immediately_on_budget_exhaustion with a _map_hierarchy_node call-count spy, which is the same fix this PR made. #3608 can be closed as fixed by #3770. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG test_hierarchy_traversal_halts_immediately_on_budget_exhaustion fails intermittently on CI (wall-clock bound)

2 participants