[professor-moody]

Rationale for this change

When IpcReadOptions::included_fields is set, RecordBatchFileReader reads only the parts of a record batch body that the selected fields need. ReadFieldsSubset in cpp/src/arrow/ipc/message.cc obtains those ranges by running the fields loader against an IoRecordedRandomAccessFile, and that pass (LoadFieldsSubset in reader.cc) uses the default V5 buffer layout.

The two layouts differ for unions: with V4 metadata a union array has a validity buffer slot, with V5 it does not. For a V4 message that contains a union column, the recorded ranges therefore belong to the wrong buffers. The batch is then loaded with the message's own V4 layout, from parts of the body buffer that ReadFieldsSubset did not fill, so the selected union column does not hold the values stored in the file. This also affects Feather V2 files, since the Feather reader reads a column subset through included_fields.

What changes are included in this PR?

In ReadFieldsSubset, when the message metadata version is older than V5, the whole body is read into the body buffer after the fields loader has run, instead of only the recorded ranges. V5 messages keep the selective reads.

Are these changes tested?

Yes. cpp/src/arrow/ipc/read_write_test.cc adds ReadFieldSubsetWithUnionsV4RoundTrip and ReadFieldSubsetWithUnionsV5RoundTrip for the stream reader, the file reader and the two file generator fixtures. Each writes a batch with a sparse union, a dense union and an int32 column using the given metadata version, reads several field subsets, checks ValidateFull() and compares the result with the expected columns. Without this change TestFileFormat.ReadFieldSubsetWithUnionsV4RoundTrip fails and the V5 tests pass.

Are there any user-facing changes?

Reading a subset of fields from a V4 IPC file (or a Feather V2 file written with V4 metadata) that contains union columns now returns the stored values. For messages with pre-V5 metadata a subset read now reads the full message body.

This PR contains a "Critical Fix". Reading a subset of fields from such a file could return values that are not in the file.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

• GitHub Issue: [C++] IPC metadata V4 union field-subset reads can return incorrect values #55369

[github-actions]

⚠️ GitHub issue #55369 has been automatically assigned in GitHub to PR creator.

[Reranko05]

@professor-moody Could you follow the Arrow PR template and add AI disclosure checkbox in the PR desc. Thanks!

[professor-moody]

Thanks @Reranko05! Updated the PR description to follow the Arrow PR template and include the AI disclosure checkboxes.