Repository navigation
[Python] pa.record_batch raises TypeError for tuple data #55418
Description
Describe the bug, including details regarding any error messages, version, and platform.
pa.record_batch() checks isinstance(data, (list, tuple)) before passing data to RecordBatch.from_arrays
arrow/python/pyarrow/table.pxi
Lines 6046 to 6048 in 802102a
but a tuple of arrays raises a TypeError as arrays is typed as list in Cython:
arrow/python/pyarrow/table.pxi
Line 3431 in 802102a
Example:
>>> import pyarrow as pa
>>> pa.record_batch((pa.array([1]),), names=["a"])
Traceback (most recent call last):
File "<python-input-2>", line 1, in <module>
pa.record_batch((pa.array([1]),), names=["a"])
File "pyarrow/table.pxi", line 6080, in pyarrow.lib.record_batch
return RecordBatch.from_arrays(data, names=names, schema=schema,
TypeError: Argument 'arrays' has incorrect type (expected list, got tuple)The same data passed as a list works.
Note: pa.table() works correctly:
>>> pa.table(([1],), names=["a"])
pyarrow.Table
a: int64
----
a: [[1]]I have found the issue when reviewing __lib/table.pyi in #55264
Component(s)
Python
Activity
Moemenmohamed24 commentedon Nov 5, 2026
Hello @AlenkaF
I would like to work on this issue and submit a solution along with the necessary tests.
Would it be possible for me to take responsibility for this issue ?
Thank you
kartik00052 commentedon Nov 6, 2026
can I work on this now...??
AlenkaF commentedon Nov 6, 2026
This issue needs some discussion on the approach to take first, before any PR is made.
raulcd commentedon Nov 6, 2026
@kartik00052 please stop indiscriminately posting on all newly created issues or we will have to ban you. There is a reason the previous PR was closed. Please investigate a little bit before indiscriminately pinging if you can work on an issue.
Check whether there's someone else working on it (does it have PRs? is it assigned?) also check if there are related closed PRs what is the reason, if there is a conversation take that into account.
Awesome06 commentedon Nov 8, 2026
There appear to be two ways to resolve this:
listtype restriction onRecordBatch.from_arrays(arrays, ...)to accept any sequence, matchingTable.from_arraysand theSequence[...]typing stubs in GH-55263: [Python][Annotations] Add internal types and core data structures #55264.RecordBatch.from_arraysrestricted to lists and convert tuples vialist(data)insidepa.record_batch().Option 1 provides consistency across both tabular constructors. Does removing the
listrestriction onRecordBatch.from_arraysmatch the intended design?Happy to put together a PR with tests for (1) if that is the preferred direction.
AlenkaF commentedon Nov 9, 2026
My first question, which I haven't investigated yet, is: why is arrays parameter typed to list in Cython, is there a special reason behind that decision? If yes, we can not blindly remove the type restriction there.
No PRs yet, please.
Awesome06 commentedon Nov 9, 2026
I checked the history and implementation of
RecordBatch.from_arraysto see whyarrayswas typed tolistin Cython:There does not appear to be an architectural or safety reason for it, looks to be a historical artifact from 2017.
A few details:
RecordBatch.from_arraysonly checkslen(arrays)andarrays[0]before passing it to_sanitize_arrays(arrays, ...)._sanitize_arraysproduces an internal Python list (converted_arrays), which is then used to populate the C++vector[shared_ptr[CArray]]. The input sequence is never passed directly to C++.Table:Table.from_arrayshas always keptarraysuntyped and accepts tuples without issue.pa.record_batch()was added in ARROW-6263, it was explicitly written asif isinstance(data, (list, tuple)): return RecordBatch.from_arrays(data, ...). Tuples were clearly intended to be supported, but were blocked by the legacy Cython typing.Given that, dropping the
listrestriction onRecordBatch.from_arrays(def from_arrays(arrays, ...)) should be completely safe and aligns it withTable.from_arraysand the typing stubs in #55264.Commit history details
a9747ceac, ARROW-312): BothRecordBatch.from_arraysandTable.from_arrayswere originally untyped (def from_arrays(names, arrays)).7d433dc27, ARROW-483 / PR ARROW-483: [C++/Python] Provide access to "custom_metadata" Field attribute in IPC setting #588):RecordBatch.from_arrayshad Cython types added (def from_arrays(list arrays, list names, dict metadata=None)), whileTable.from_arrayskeptarraysuntyped.ec9b72a23, ARROW-1715):list nameswas relaxed tonamesinRecordBatch.from_arraysto support schemas during unpickling, leavinglist arrays.9da458437, ARROW-2624):dict metadatawas relaxed tometadatato allow arbitrary mappings, leavinglist arrays.05bc63c3b, ARROW-6263):pa.record_batchandpa.tablefactory functions were added withisinstance(data, (list, tuple)).pa.tableworked becauseTable.from_arrayswas untyped, whilepa.record_batchfailed on tuples due to the remaining Cython check.