Repository navigation
deps: [branch-1.1] bump arrow and parquet to 59.3.2 (#7419) - #7448
Merged
andygrove merged 1 commit intoNov 3, 2026
Merged
Conversation
* deps: bump arrow and parquet to 59.3.2 arrow-rs 59.3.2 includes apache/arrow-rs#11754, the 59.x backport of apache/arrow-rs#11021: DeltaBitPackDecoder::skip reads miniblocks wider than 64 values instead of failing with "cannot skip miniblock of size N". Snowflake writes 512-value miniblocks and Databricks Photon 256, and a native scan reaches skip when it skips a long run of rows, such as a run of Iceberg position deletes. The native and Delta contrib lockfiles move arrow-*, parquet and parquet-variant* from 59.3.0 to 59.3.2. See apache#7417. * deps: keep prost-build and prost-derive on itertools 0.14.0 cargo update re-picked itertools 0.13.0 for them when moving arrow and parquet. Restore the previous choice, so the lockfiles only change the arrow and parquet crates. * test: update the RSS frame size for arrow-rs 59.3.2 (cherry picked from commit 644a870)
github-actions
Bot
added
dependencies
andygrove
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backport of #7419 to
branch-1.1, for #7417.Rationale for this change
The native Iceberg scan fails with
cannot skip miniblock of size Nwhen position deletes make it skip a long run of rows in aDELTA_BINARY_PACKEDcolumn with miniblocks wider than 64 values, which Snowflake and Databricks Photon write (#7417).branch-1.1locks parquet 59.3.0 and has the bug. arrow-rs 59.3.2 fixes it.Moving from 59.3.0 also picks up 59.3.1, a patch release of bug fixes that
mainhas had since #7169.The fix also goes to
branch-1.2in #7447, opened together with this one. Per the backporting guide, this one should not merge before that one is ready.branch-1.0locks parquet 58.4.0, whose decoder fails the same way, but no 58.x release has the fix (58.4.1 doesn't), so it gets no backport here.What changes are included in this PR?
git cherry-pick -x 644a870of #7419, with one adaptation. The lockfiles conflicted, because this branch locks 59.3.0 rather thanmain's 59.3.1. I kept the branch's lockfiles, rerancargo update -p arrow --precise 59.3.2andcargo update -p parquet --precise 59.3.2innative/andcontrib/delta/native/, and restored itertools 0.14.0 forprost-buildandprost-derive, as #7419 does. The lockfiles change only the arrow-, parquet and parquet-variant crates. The RSS frame-size test change applied cleanly.How are these changes tested?
On this branch:
./mvnw test-compilenative/:cargo fmt --all -- --check,cargo check --workspace --all-targets --locked,cargo clippy --workspace --all-targets --locked -- -D warningsandcargo test --workspace --locked(1861 passed)cargo check --locked --all-targetsincontrib/delta/native#7419 adds no Comet test, as arrow-rs covers the decoder. The reproduction in #7417 fails on 1.1.0, and returns Spark's results on 1.2.0-rc1 built with parquet 59.3.2.