Skip to content

feat(bigquery-jdbc): read large results from the job stream - #15078

Open
keshavdandeva wants to merge 2 commits into
jdbc/arrow-support-featurefrom
feat/jdbc-arrow-job-stream-wiring
Open

keshavdandeva wants to merge 2 commits into
jdbc/arrow-support-featurefrom
feat/jdbc-arrow-job-stream-wiring

Conversation

@keshavdandeva

@keshavdandeva keshavdandeva commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

b/582676785

With EnableQueryResultsArrowFormat=true, a large SELECT result is read as Arrow from the query job's _default stream, using the reader from #15048. With the property off, nothing changes.

Changes

  • BigQueryStatement.processQueryResponse gets one new branch. It covers jobs.query, long-running jobs and each child job of a script.
  • useJobStream(TableResult) holds the rule. A result is streamed from row 0 when no LargeResultTable or LargeResultDataset is set, it has a next page, and it returns at least HighThroughputMinTableSize rows (default 10,000), counted after setMaxRows. Any other result is read as JSON. With a destination table set, today's High-Throughput API rules apply.
  • Both Arrow paths share the result-set construction and the error mapping, and hand the queue to the reader and the result set directly. The statement field that held the queue is removed: a concurrent cancel() could set it to null.
  • The descriptions of EnableQueryResultsArrowFormat and EnableHighThroughputAPI say how the two combine.

Not in this PR

  • A fallback to JSON when the stream can't be read (b/582677588). Until then a stream failure reaches the caller as SQLException.
  • Cancelling the stream once maxRows is reached (b/586660667).

Decisions to check

  1. A result needs both a next page and the row threshold. A multi-page result with fewer rows stays on JSON.
  2. The threshold is the existing HighThroughputMinTableSize, not a new property.
  3. The first JSON page is downloaded and then discarded for a streamed result. Its size stays MaxResults.
  4. An incomplete job reference keeps the JSON path, with a warning.
  5. The new integration tests are tagged disable_tpc.

@keshavdandeva
keshavdandeva requested review from a team as code owners November 19, 2026 14:59

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enables reading query results directly from a query job's stream when EnableQueryResultsArrowFormat is enabled and no destination table is set, bypassing the need for a read session. It refactors BigQueryStatement to support this flow, including extracting helper methods for result set creation and exception handling, and adds comprehensive unit and integration tests. The reviewer suggests improving the extracted arrowResultSetException method by returning the exception directly if it is already an instance of BigQueryJdbcException, thereby avoiding redundant wrapping and preserving original exception properties.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant