Skip to content

GH-1417: Validate decimal value length before raw writes in DecimalVector and Decimal256Vector - #1418

Merged
jbonofre merged 2 commits into
apache:mainfrom
andygrove:fix/decimal-unchecked-raw-writes
Nov 1, 2026
Merged

jbonofre merged 2 commits into
apache:mainfrom
andygrove:fix/decimal-unchecked-raw-writes

Conversation

@andygrove

@andygrove andygrove commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

What's Changed

DecimalVector and Decimal256Vector have three public write paths that copy straight into native memory with MemoryUtil, bypassing ArrowBuf bounds checking: setBigEndian(int, byte[]) (also reached via setBigEndianSafe and the generated DecimalWriter.writeBigEndianBytesToDecimal(byte[])), setSafe(int, long, ArrowBuf, int), and setBigEndianSafe(int, long, ArrowBuf, int). They bounds-check the destination slot but not the value length, so a value longer than TYPE_WIDTH overruns the slot. The byte[] variant throws only after copying; the ArrowBuf variants never throw. The ArrowBuf variants also sign-extend a zero-length value from a byte outside the value range.

This adds a checkValueLength guard that runs before any write in all three paths, removes the branches in setBigEndian(int, byte[]) that the up-front check makes unreachable, and makes the ArrowBuf variants treat length == 0 as a zero value, matching the byte[] variant. The exception message now reports the accepted range as [0 - N] instead of [1 - N], since zero-length values were already accepted; the two existing tests that pin the message are updated accordingly.

Safe vs. unsafe methods and overhead

This change follows the existing convention that *Safe methods validate their inputs while the plain set/get variants trust the caller:

  • setSafe(int, long, ArrowBuf, int) and setBigEndianSafe(int, long, ArrowBuf, int) are Safe methods that did not validate length. The new check is a single fused range compare against a constant, placed before handleSafe so a bad length cannot trigger a wasted reallocation. The zero-length early return replaces a getByte on that path and costs one predictable compare otherwise. Both methods already pay for handleSafe, setBit, and two checkBytes calls per value, so this is marginal.
  • setBigEndian(int, byte[]) is a plain variant, but it already threw IllegalArgumentException for oversized input (pinned by the existing decimalBE tests). The existing check is moved before the write rather than a new one added, and the now-unreachable branches are removed, so the little-endian path has fewer conditionals than before.
  • The unsafe paths that rely on the caller (set(int, long, ArrowBuf), set(int, ArrowBuf), the holder variants, and set(int, long)) are intentionally unchanged.

Tests

The new tests rely on the value buffer and validity buffer being adjacent slices of one allocation: each oversized-write test writes a TYPE_WIDTH + 1 byte value into the last slot and asserts that an exception is thrown and that no validity bit was cleared. The zero-length tests place a negative trap byte where the old code read for sign extension. All new tests fail on main and pass with this change. The big-endian branches are covered by inspection only, as CI and my machine are little-endian.

The fix and tests were written with the help of an AI coding assistant (Claude Code). I have reviewed every line of the generated code and verified the failing-then-passing behaviour of the tests locally.

Closes #1417.

The byte[] and ArrowBuf variants of setBigEndian/setBigEndianSafe and
setSafe(int, long, ArrowBuf, int) in DecimalVector and Decimal256Vector
copy directly into native memory with MemoryUtil. Check that the value
length fits in the TYPE_WIDTH slot before any write, and treat a
zero-length ArrowBuf value as zero instead of sign-extending from the
byte before start.

Closes apache#1417.
@andygrove andygrove added the bug-fix PRs that fix a big. label Oct 31, 2026
Use Preconditions.checkArgument with TYPE_WIDTH in the message, replace a
stray 32 literal, collapse the oversized-write tests into one helper, add a
zero-length test for setBigEndianSafe(ArrowBuf), and shorten comments so
they fit the 100 column limit enforced by spotless.
@andygrove

Copy link
Copy Markdown
Member Author

Note for reviewers: this overlaps with #1247, which fixes the setBigEndian(int, byte[]) and setBigEndianSafe(int, long, ArrowBuf, int) paths. This PR additionally covers setSafe(int, long, ArrowBuf, int) (no length check at all today) and the zero-length sign-extension read in the ArrowBuf variants.

I've left a comment on #1247 suggesting those be folded in there. If that happens I'll close this one; otherwise I'll rebase this on top of #1247 once it merges so the remaining fixes land as a follow-up.

@jbonofre jbonofre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove, this looks good to me!

The length check runs before any write in all three paths, the zero-length handling now matches the byte[] variant, and the new tests fail on main and pass with the fix (I tested 😄 ).

@jbonofre
jbonofre merged commit 12918c6 into apache:main Nov 1, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PRs that fix a big.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DecimalVector and Decimal256Vector raw write paths do not validate the value length before writing

2 participants