Repository navigation
GH-1417: Validate decimal value length before raw writes in DecimalVector and Decimal256Vector - #1418
Conversation
The byte[] and ArrowBuf variants of setBigEndian/setBigEndianSafe and setSafe(int, long, ArrowBuf, int) in DecimalVector and Decimal256Vector copy directly into native memory with MemoryUtil. Check that the value length fits in the TYPE_WIDTH slot before any write, and treat a zero-length ArrowBuf value as zero instead of sign-extending from the byte before start. Closes apache#1417.
Use Preconditions.checkArgument with TYPE_WIDTH in the message, replace a stray 32 literal, collapse the oversized-write tests into one helper, add a zero-length test for setBigEndianSafe(ArrowBuf), and shorten comments so they fit the 100 column limit enforced by spotless.
|
Note for reviewers: this overlaps with #1247, which fixes the I've left a comment on #1247 suggesting those be folded in there. If that happens I'll close this one; otherwise I'll rebase this on top of #1247 once it merges so the remaining fixes land as a follow-up. |
jbonofre
left a comment
There was a problem hiding this comment.
Thanks @andygrove, this looks good to me!
The length check runs before any write in all three paths, the zero-length handling now matches the byte[] variant, and the new tests fail on main and pass with the fix (I tested 😄 ).
What's Changed
DecimalVectorandDecimal256Vectorhave three public write paths that copy straight into native memory withMemoryUtil, bypassingArrowBufbounds checking:setBigEndian(int, byte[])(also reached viasetBigEndianSafeand the generatedDecimalWriter.writeBigEndianBytesToDecimal(byte[])),setSafe(int, long, ArrowBuf, int), andsetBigEndianSafe(int, long, ArrowBuf, int). They bounds-check the destination slot but not the value length, so a value longer thanTYPE_WIDTHoverruns the slot. Thebyte[]variant throws only after copying; theArrowBufvariants never throw. TheArrowBufvariants also sign-extend a zero-length value from a byte outside the value range.This adds a
checkValueLengthguard that runs before any write in all three paths, removes the branches insetBigEndian(int, byte[])that the up-front check makes unreachable, and makes theArrowBufvariants treatlength == 0as a zero value, matching thebyte[]variant. The exception message now reports the accepted range as[0 - N]instead of[1 - N], since zero-length values were already accepted; the two existing tests that pin the message are updated accordingly.Safe vs. unsafe methods and overhead
This change follows the existing convention that
*Safemethods validate their inputs while the plainset/getvariants trust the caller:setSafe(int, long, ArrowBuf, int)andsetBigEndianSafe(int, long, ArrowBuf, int)areSafemethods that did not validatelength. The new check is a single fused range compare against a constant, placed beforehandleSafeso a bad length cannot trigger a wasted reallocation. The zero-length early return replaces agetByteon that path and costs one predictable compare otherwise. Both methods already pay forhandleSafe,setBit, and twocheckBytescalls per value, so this is marginal.setBigEndian(int, byte[])is a plain variant, but it already threwIllegalArgumentExceptionfor oversized input (pinned by the existingdecimalBEtests). The existing check is moved before the write rather than a new one added, and the now-unreachable branches are removed, so the little-endian path has fewer conditionals than before.set(int, long, ArrowBuf),set(int, ArrowBuf), the holder variants, andset(int, long)) are intentionally unchanged.Tests
The new tests rely on the value buffer and validity buffer being adjacent slices of one allocation: each oversized-write test writes a
TYPE_WIDTH + 1byte value into the last slot and asserts that an exception is thrown and that no validity bit was cleared. The zero-length tests place a negative trap byte where the old code read for sign extension. All new tests fail onmainand pass with this change. The big-endian branches are covered by inspection only, as CI and my machine are little-endian.The fix and tests were written with the help of an AI coding assistant (Claude Code). I have reviewed every line of the generated code and verified the failing-then-passing behaviour of the tests locally.
Closes #1417.