Repository navigation
[DO NOT MERGE] feature branch for arrow support in bq jdbc - #15067
Draft
keshavdandeva wants to merge 5 commits into
Draft
keshavdandeva wants to merge 5 commits into
keshavdandeva wants to merge 5 commits into
Conversation
#14964) b/582677347 This PR adds the `EnableQueryResultsArrowFormat` connection property (default `false`, accepts 0/1/true/false). It goes through the usual path: URL -> DataSource -> BigQueryConnection -> BigQuerySettings for each statement, same as `EnableTimestampPicos`. The property doesn't change any behaviour yet. Reading it and asking BigQuery for Arrow results comes in a follow-up PR (b/582676785). This PR only sets up the configuration so that work stays small. Added unit tests
) b/582676539 Prepares the Arrow reader for the job stream (b/581544197). - Producer skips Read API responses without rows, fails fast when a response reports rows but has no record batch, and forwards a schema sent in the stream. - Result set skips zero-row batches and builds its decoder from the first schema it sees. - `isBeforeFirst()`/`isLast()` return false for an empty result.
b/582677690 Adds a reader for a query job's `_default` stream, so results can be read through the Storage Read API without creating a read session. Nothing calls it yet; Task 4 (b/582676785) wires it into statement execution. ## Changes - Build the stream name from the job reference (project, location, job ID). - Send the Arrow options on every `ReadRows` request. A resumed request is the first request with only the offset changed. - The reader forwards each message's `row_count` and decides nothing. `BigQueryArrowResultSet.loadNextBatch()` fails when a batch doesn't hold the rows its message reports. - Any stream error ends iteration: the next call to `next()` returns false, and getters throw `SQLException` until then. ## Behaviour changes on the existing HTAPI path - A `row_count` that differs from the rows in the batch is now an error. - After a stream error, `next()` returns false. Before, it read on from the following message. ## Testing - `make lint` and `make unittest`: 1280 tests, 0 failures, 1 skipped in a class this PR doesn't touch. - 16 new unit tests. The result-set tests use real Arrow batches.
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces the EnableQueryResultsArrowFormat configuration to support reading query results in Apache Arrow format. It updates BigQueryArrowBatchWrapper to include schema and row count details, refactors BigQueryArrowResultSet to dynamically load batches and validate row counts, and adds corresponding unit tests. The review feedback highlights a compilation error in BigQueryArrowResultSet due to an unhandled InterruptedException in loadNextBatch(), an off-heap memory leak in the test helper idRoot from unclosed Arrow resources, and improper interruption handling during stream retries in BigQueryStatement.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.