Repository navigation
refactor(bigquery-jdbc): move Arrow batch validation out of the result-set loop - #15069
Open
keshavdandeva wants to merge 1 commit into
Open
keshavdandeva wants to merge 1 commit into
keshavdandeva wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors Arrow batch handling and row-count validation in the BigQuery JDBC driver. It introduces explicit validation of reported row counts within BigQueryArrowBatchWrapper and improves memory management in BigQueryArrowResultSet by ensuring Arrow vectors are cleared and released upon stream completion, iteration limits, or decoding failures. Additionally, comprehensive unit tests have been added to verify these behaviors. Feedback on the changes suggests wrapping the arrowDeserializer.close() call in a try-finally block to guarantee that allocator.close() is always executed, ensuring exception-safe resource cleanup.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
b/582676785
Follow-up to #15048, for two review comments there: the old wrapper factory and the checks in
loadNextBatch(). This is a refactor: the same checks, with the same error messages.Changes
BigQueryArrowBatchWrapperowns the row-count rule.checkRowCount(actualRows)holds the comparison and the error text.validate()applies it to a message without batch data, which holds no rows. Both were inBigQueryArrowResultSet.ArrowDeserializerowns the vectors it decodes into and their loader, as final fields.deserializeArrowBatchchecks the row count before it loads the batch, then returns the decoded rows. If it fails, the vectors hold nothing. Before, it filled a field of the result set, and the loop checked the count afterwards.releaseBatch()). ItsvectorSchemaRootfield is now only what the getters read.loadNextBatch()keeps the control flow and the two checks on the result set's own state: a batch before its schema, and the total at the end of the stream.BigQueryArrowBatchWrapper.of(batch, isLast...)is replaced byofEndOfStream(), the nameBigQueryFieldValueListWrapperalready uses. Main code used the old factory only for that marker.getNestedFieldValueListWrapperloses itsisLastvararg, which no caller passed.private final. Before,ArrowDeserializer.close()closed it.Three differences in behaviour, all on a stream that is already broken:
ReadRowsResponse.row_count, which is not negative.next()orclose().Decisions to check
validate()is called fromloadNextBatch(), not from the wrapper's factory. The reader thread calls the factory, and an earlier review asked that the reader only move data.checkRowCount, notvalidate(). The row count is in the Arrow message, which only the deserializer parses, so the wrapper can't check it by itself. The rule and its message are still in the wrapper.close(), as before. A batch that fails to decode never reaches the result set: the deserializer drops it.getAllocatedMemory()is a new test-only accessor on the result set. Without it, no unit test can tell whether a batch was released beforeclose().ArrowDeserializerstays an inner class. It reads the result set's allocator and logger, and writes none of its fields.