Skip to content

fix(bigtable): throttle session creation on consecutive failures - #15036

Closed
mutianf wants to merge 7 commits into
mainfrom
fix-pool-sizer-max-capacity
Closed

mutianf wants to merge 7 commits into
mainfrom
fix-pool-sizer-max-capacity

Conversation

@mutianf

@mutianf mutianf commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Part 1 of 4 for b/584671437.

What changed

  • In PoolSizer.getScaleDelta(), ignore queued pendingRpcs for scale-up while consecutiveFailures > 0 so accumulating pending requests do not trigger unbounded session creation when session establishment is failing.
  • Once consecutiveFailures >= consecutive_session_failure_threshold, allow at most 1 STARTING probe session at a time (startingCount == 0 ? 1 : 0).
  • In SessionPoolImpl.onSessionReady(), scale the pool back up when recovering from consecutiveFailures > 0.
  • Check isDisabled() at the start of getScaleDelta() and remove unnecessary volatile modifiers from PoolSizer fields guarded by poolLock.

@mutianf
mutianf requested review from a team as code owners November 16, 2026 16:59
@mutianf
mutianf added this pull request to stack #15040 November 16, 2026 16:59
@mutianf mutianf changed the title fix pool sizer max capacity fix(bigtable): cap PoolSizer desiredCapacity at maxSessions Nov 16, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request renames maxIdleSessions to maxSessions in PoolSizer and caps the total desiredCapacity at maxSessions. It also adds a unit test to verify this capping behavior. The reviewer pointed out that the desiredCapacity calculation in the existing testEquilibriumWhenStartingWithLoad test should also be updated to cap at MAX_SESSIONS to match the new production logic and prevent potential test failures.

Comment on lines +30 to +32
private volatile float idlesSessionHeadRoom;
private volatile int minIdleSessions;
private volatile int maxIdleSessions;
private volatile int minSessions;
private volatile int maxSessions;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these need to be volatile? I thought PoolSizer was only used under lock?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we probably didn't clean this up after the re-write

@mutianf mutianf changed the title fix(bigtable): cap PoolSizer desiredCapacity at maxSessions fix(bigtable): throttle session creation on consecutive failures Nov 16, 2026
@mutianf mutianf closed this Nov 16, 2026
@mutianf
mutianf removed this pull request from stack #15040 November 16, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants