Skip to content

feat(bigtable): plumb per-operation timeouts through CbtTestProxy - #14997

Open
alexoneill wants to merge 1 commit into
googleapis:mainfrom
alexoneill:feat-bigtable-test-proxy-per-op-timeout
Open

alexoneill wants to merge 1 commit into
googleapis:mainfrom
alexoneill:feat-bigtable-test-proxy-per-op-timeout

Conversation

@alexoneill

@alexoneill alexoneill commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Add an optional google.protobuf.Duration per_operation_timeout field to each operation request message in test_proxy.proto (ReadRowRequest, ReadRowsRequest, MutateRowRequest, MutateRowsRequest, CheckAndMutateRowRequest, SampleRowKeysRequest, ReadModifyWriteRowRequest, and ExecuteQueryRequest) so callers can override CreateClientRequest.per_operation_timeout (in both directions) and the client library default timeout on a per-call basis.

  • Pass callContext into retryAlgorithm.createFirstAttempt(callContext) in MutateRowsAttemptCallable so per-call RetrySettings overrides on bulkMutationCallable().call(batch, callContext) apply to entry-level retry decisions as well as the parent RetryingFuture.
  • Store Optional<Duration> perOperationTimeout() on CbtClient while keeping client.settings() at the unmodified library defaults, and resolve the effective timeout per operation via resolveTimeout (request-level timeout supersedes client.perOperationTimeout()).
  • Plumb per-operation timeout overrides through mutateRow, bulkMutateRows, readRow, readRows, sampleRowKeys, checkAndMutateRow, readModifyWriteRow, and executeQuery.
  • Add unit tests in CbtTestProxyTest and MutateRowsAttemptCallableTest covering shorter and longer per-operation timeout overrides across all operations.

@alexoneill
alexoneill requested review from a team as code owners November 13, 2026 02:34

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces per-operation timeout overrides for various Bigtable operations in the test proxy, updating CbtTestProxy to apply these timeouts and extending test_proxy.proto with per_operation_timeout fields. It also includes comprehensive tests verifying that these overrides correctly supersede client-level timeouts, and updates MutateRowsAttemptCallable to pass the call context on the first attempt. Feedback on the changes highlights a potential NullPointerException in CbtTestProxy.java if previousContext is null when calling detach, which should be guarded to prevent masking other exceptions.

Comment on lines +846 to +849
if (cancellableContext != null) {
cancellableContext.detach(previousContext);
cancellableContext.close();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If cancellableContext is initialized but previousContext remains null (for example, if an exception occurs or the thread is interrupted before attach() completes), calling cancellableContext.detach(previousContext) will throw a NullPointerException because Context.detach does not accept null. This NPE would mask the original exception.

To prevent this, guard the detach call by checking if previousContext is not null.

      if (previousContext != null) {
        cancellableContext.detach(previousContext);
      }
      if (cancellableContext != null) {
        cancellableContext.close();
      }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — guarded cancellableContext.detach(previousContext) with if (previousContext != null).

request.getPerOperationTimeout().getNanos());
cancellableContext =
Context.current()
.withDeadlineAfter(timeout.toNanos(), TimeUnit.NANOSECONDS, null);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, use MoreExecutores.directExecutor() instead of null?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added a ScheduledExecutorService deadlineScheduler on CbtTestProxy (shut down in close()) and passed it to withDeadlineAfter (which requires a ScheduledExecutorService), along with a CancellationListener to interrupt the caller on deadline expiration.

Comment on lines +144 to +175
private static void updateTimeout(RetrySettings.Builder settings, Duration newTimeout) {
Duration rpcTimeout = settings.getInitialRpcTimeoutDuration();
Duration maxRpcTimeout = settings.getMaxRpcTimeoutDuration();
Duration oldTotalTimeout = settings.getTotalTimeoutDuration();

// TODO: this should happen in gax
// Clamp the rpcTimeout to the overall timeout
if (rpcTimeout != null && rpcTimeout.compareTo(newTimeout) > 0) {
settings.setInitialRpcTimeoutDuration(newTimeout).setMaxRpcTimeoutDuration(newTimeout);
// Clamp the rpcTimeout to the overall timeout, or raise it if it was previously clamped to the
// old total timeout.
if (rpcTimeout != null
&& (rpcTimeout.compareTo(newTimeout) > 0 || rpcTimeout.equals(oldTotalTimeout))) {
settings.setInitialRpcTimeoutDuration(newTimeout);
}
if (maxRpcTimeout != null
&& (maxRpcTimeout.compareTo(newTimeout) > 0 || maxRpcTimeout.equals(oldTotalTimeout))) {
settings.setMaxRpcTimeoutDuration(newTimeout);
}

settings.setTotalTimeoutDuration(newTimeout);
}

private static boolean hasPositiveTimeout(
boolean hasTimeout, com.google.protobuf.Duration timeout) {
return hasTimeout && (timeout.getSeconds() > 0 || timeout.getNanos() > 0);
}

private static ApiCallContext createCallContextWithTimeout(
RetrySettings baseRetrySettings, com.google.protobuf.Duration protoTimeout) {
Duration timeout = Duration.ofSeconds(protoTimeout.getSeconds(), protoTimeout.getNanos());
RetrySettings.Builder builder = baseRetrySettings.toBuilder();
updateTimeout(builder, timeout);
return GrpcCallContext.createDefault().withRetrySettings(builder.build());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of mutating BigtableDataSettings in createClient (which overwrites the library defaults and requires the equals(oldTotalTimeout) heuristic to unclamp rpcTimeout / maxRpcTimeout), could we store Optional<Duration> perOperationTimeout on CbtClient and leave client.settings() at the library defaults (similar to the C++ test proxy)?

Then every operation can resolve the effective timeout (request override -> client override -> empty) and only clamp down from the untouched library defaults:

  @AutoValue
  abstract static class CbtClient {
    static CbtClient create(BigtableDataSettings settings, BigtableDataClient dataClient) {
      return create(settings, dataClient, Optional.empty());
    }

    static CbtClient create(
        BigtableDataSettings settings,
        BigtableDataClient dataClient,
        Optional<Duration> perOperationTimeout) {
      return new AutoValue_CbtTestProxy_CbtClient(settings, dataClient, perOperationTimeout);
    }

    abstract BigtableDataSettings settings();

    abstract BigtableDataClient dataClient();

    abstract Optional<Duration> perOperationTimeout();
  }

  private static Optional<Duration> resolveTimeout(
      @Nullable CbtClient client,
      boolean hasRequestTimeout,
      com.google.protobuf.Duration requestTimeout) {
    if (hasRequestTimeout && Durations.compare(requestTimeout, Durations.ZERO) > 0) {
      return Optional.of(
          Duration.ofSeconds(requestTimeout.getSeconds(), requestTimeout.getNanos()));
    }
    return client != null ? client.perOperationTimeout() : Optional.empty();
  }

  private static ApiCallContext createCallContextWithTimeout(
      RetrySettings defaultRetrySettings, Duration timeout) {
    RetrySettings.Builder builder = defaultRetrySettings.toBuilder();
    Duration initialRpcTimeout = builder.getInitialRpcTimeoutDuration();
    if (initialRpcTimeout != null && initialRpcTimeout.compareTo(timeout) > 0) {
      builder.setInitialRpcTimeoutDuration(timeout);
    }
    Duration maxRpcTimeout = builder.getMaxRpcTimeoutDuration();
    if (maxRpcTimeout != null && maxRpcTimeout.compareTo(timeout) > 0) {
      builder.setMaxRpcTimeoutDuration(timeout);
    }
    builder.setTotalTimeoutDuration(timeout);
    return GrpcCallContext.createDefault().withRetrySettings(builder.build());
  }

And in each operation handler:

    Optional<Duration> timeout =
        resolveTimeout(client, request.hasPerOperationTimeout(), request.getPerOperationTimeout());
    if (timeout.isPresent()) {
      ApiCallContext callContext =
          createCallContextWithTimeout(
              client.settings().getStubSettings().mutateRowSettings().getRetrySettings(),
              timeout.get());
      client.dataClient().mutateRowCallable().call(mutation, callContext);
    } else {
      client.dataClient().mutateRow(mutation);
    }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — stored Optional<Duration> perOperationTimeout() on CbtClient while keeping client.settings() at the unmodified library defaults, and added resolveTimeout + simplified createCallContextWithTimeout.

Add an optional `google.protobuf.Duration per_operation_timeout` field to each operation request message in `test_proxy.proto` (`ReadRowRequest`, `ReadRowsRequest`, `MutateRowRequest`, `MutateRowsRequest`, `CheckAndMutateRowRequest`, `SampleRowKeysRequest`, `ReadModifyWriteRowRequest`, and `ExecuteQueryRequest`) so callers can override `CreateClientRequest.per_operation_timeout` (in both directions) and the client library default timeout on a per-call basis.

- Pass `callContext` into `retryAlgorithm.createFirstAttempt(callContext)` in `MutateRowsAttemptCallable` so per-call `RetrySettings` overrides on `bulkMutationCallable().call(batch, callContext)` apply to entry-level retry decisions as well as the parent `RetryingFuture`.
- Store `Optional<Duration> perOperationTimeout()` on `CbtClient` while keeping `client.settings()` at the unmodified library defaults, and resolve the effective timeout per operation via `resolveTimeout` (request-level timeout supersedes `client.perOperationTimeout()`).
- Plumb per-operation timeout overrides through `mutateRow`, `bulkMutateRows`, `readRow`, `readRows`, `sampleRowKeys`, `checkAndMutateRow`, `readModifyWriteRow`, and `executeQuery`.
- Add unit tests in `CbtTestProxyTest` and `MutateRowsAttemptCallableTest` covering shorter and longer per-operation timeout overrides across all operations.
@alexoneill
alexoneill force-pushed the feat-bigtable-test-proxy-per-op-timeout branch from e06e97a to 7641d2f Compare November 13, 2026 22:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants