Skip to content

fix(bigtable): fail pendingRpcs on consecutive failures when pool has no established sessions - #15038

Open
mutianf wants to merge 4 commits into
mainfrom
fix-session-pool-pending-rpcs-bound
Open

mutianf wants to merge 4 commits into
mainfrom
fix-session-pool-pending-rpcs-bound

Conversation

@mutianf

@mutianf mutianf commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Part 1 of 2 for b/584671437.

What changed

  • In SessionPoolImpl, fail pendingRpcs on consecutiveFailures >= consecutive_session_failure_threshold if and only if the pool has no established (READY or IN_USE) sessions (readyCount + inUseCount == 0).
  • Enforce this invariant symmetrically in both onSessionClose() (draining queued pendingRpcs when transitioning to 0 established sessions while consecutiveFailures >= threshold, while preserving pendingRpcs when inUseCount > 0) and PendingVRpc.start() (rejecting new calls during the 60s SessionCreationBudget lockout when no sessions are closing).

@mutianf
mutianf requested review from a team as code owners November 16, 2026 16:59
@mutianf
mutianf added this pull request to stack #15040 November 16, 2026 16:59

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces queue bounding and fast-fail mechanisms to the SessionPoolImpl when consecutive session establishment failures occur or when the pending RPC queue exceeds its maximum capacity. Feedback on these changes highlights two potential issues: first, if lastCloseStatus is OK, rejecting the RPC with an OK status could lead to silent failures, so it should be defensively mapped to UNAVAILABLE; second, trailers can occasionally be null in gRPC, so a defensive null check should be added when assigning lastCloseTrailers to prevent null pointer issues.

…secutive failures

# Conflicts:
#	java-bigtable/google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/internal/session/PoolSizerTest.java

# Conflicts:
#	java-bigtable/google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/internal/session/SessionPoolImplTest.java
@mutianf
mutianf removed this pull request from stack #15042 November 16, 2026 21:05
@mutianf
mutianf force-pushed the fix-session-pool-pending-rpcs-bound branch from c6ec691 to 54bb35c Compare November 16, 2026 21:05
@mutianf
mutianf changed the base branch from fix-session-list-pending-vrpc-cleanup to main November 16, 2026 21:05
@mutianf
mutianf added this pull request to stack #15043 November 16, 2026 21:05
@mutianf mutianf changed the title fix(bigtable): bound SessionPool pendingRpcs and fast-fail during consecutive failures fix(bigtable): fail pendingRpcs on consecutive failures when pool has no established sessions Nov 16, 2026
private int consecutiveFailures = 0;

@GuardedBy("poolLock")
private Status lastCloseStatus = Status.UNAVAILABLE;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be null?

handle.getSession().close(closeReq);
}
} else if (poolState == PoolState.STARTED) {
if (isFailingWithNoEstablishedSessions()) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this the right place for this check? this is the listener for client config udpates, is this the right place to pop closable RPCs? Also, client configuration responses arrives every 5 minutes. So it's longer interval then normal vrpcs deadline.

afterRelease(handle);
toBeClosed = afterRelease(handle);
if (!toBeClosed.isEmpty()) {
closeResult = createConsecutiveFailureResult(lastCloseStatus, lastCloseTrailers);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this mean consecutiveFailure?

afterRelease(handle);
toBeClosed = afterRelease(handle);
if (!toBeClosed.isEmpty()) {
closeResult = createConsecutiveFailureResult(lastCloseStatus, lastCloseTrailers);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant