[logachev]

Summary

Queries routed through the BigQuery Storage Read API (HTAPI / Arrow) or Storage Write API fail with UNAUTHENTICATED when connecting to non-default universe domains (TPC / custom universeDomain) using service account or Application Default Credentials.

Root Cause

1. BigQueryJdbcOAuthUtility.getCredentials(...) returns unscoped GoogleCredentials (createScopedRequired() == true) unless RequestGoogleDriveScope=1 or service account impersonation is configured.
2. For the REST BigQuery client, BigQueryOptions (ServiceOptions.getScopedCredentials()) automatically scopes credentials, causing ServiceAccountCredentials in non-default universes to mint a scope-based self-signed JWT (scope claim, no aud claim).
3. For BigQueryReadClient and BigQueryWriteClient, BigQueryConnection bypassed BigQueryReadSettings.defaultCredentialsProviderBuilder() / BigQueryWriteSettings.defaultCredentialsProviderBuilder() and wrapped the unscoped credential directly in FixedCredentialsProvider.create(...). Because createScopedRequired() remained true, ServiceAccountCredentials fell back to minting an audience-based self-signed JWT (aud set to the custom universe Storage endpoint, no scope claim), which is rejected with UNAUTHENTICATED.

Fix

• Added BigQueryConnection.scopeStorageCredentials(Credentials, List<String>) mirroring GoogleCredentialsProvider in BigQueryReadSettings and BigQueryWriteSettings:
  • Applies default service scopes (BigQueryReadSettings.getDefaultServiceScopes() / BigQueryWriteSettings.getDefaultServiceScopes()) when createScopedRequired() is true, while preserving explicit scopes when already configured (such as RequestGoogleDriveScope=1).
  • Enables createWithUseJwtAccessWithScope(true) on ServiceAccountCredentials so self-signed JWTs include the scope claim without requiring extra OAuth token endpoint round-trips in GDU.
• Re-enabled Read API integration tests in TPC (@Tag("disable_tpc")) that were previously disabled due to this authentication failure.

Tests

• BigQueryConnectionTest: verifies BigQueryReadClient and BigQueryWriteClient preserve custom universeDomain settings and mint scope-based self-signed JWTs (scope claim present, aud absent), including when explicit scopes are pre-applied.
• ITBigQueryJDBCTest: re-enabled testReadAPIPathLarge, testReadAPIPathLargeWithThresholdParameters, and testReadAPIPathLargeWithThresholdNotMet for TPC using self-contained UNNEST(GENERATE_ARRAY(...)) queries and explicit result set type assertions.
• ITNightlyBigQueryTest: re-enabled testHTAPIWithValidDestinationTableSavesQueriesWithStandardSQL, testReadAPIPathLarge, and testReadAPIPathLargeWithThresholdParameters for TPC.

[gemini-code-assist]

Code Review

This pull request introduces credentials scoping for BigQuery Read and Write clients in BigQueryConnection to support Trusted Partner Cloud (TPC) environments, and updates several integration tests by removing the @Tag("disable_tpc") annotation and asserting specific result set types. The review feedback suggests improving test robustness by adding null and presence checks for the Authorization header in the test helper extractBearerJwtPayload to avoid potential NullPointerException or IndexOutOfBoundsException, and using try-with-resources for jsonResultSet in ITBigQueryJDBCTest to prevent resource leaks if assertions fail.

[keshavdandeva]

A few nits on the tests, mainly to make them more readable and to reuse the setup and helpers we already have. Could you do a quick cleanup pass along these lines? Something like this prompt should cover it:

Refactor the tests added in this PR for readability without changing what they cover: reuse the existing fixtures in BigQueryJdbcBaseTest (fake_pkcs8_key, getUriOAuthServiceAccount()) instead of generating a key and building the PEM and URL by hand, extract the duplicated Read/Write credential assertions into one helper, and add cases for a non-service-account credential and for the default universe. In the integration tests, move the shared GENERATE_ARRAY query and the Arrow result set class check into ITBase as a constant and a helper, and run the formatter.