[rahulsmahadev]

Rationale for this change

_determine_partitions adds the transformed partition values to the Arrow table as temporary columns named _partition_<partition field name>. If the data already has a column with that name, the lookup by name is ambiguous and append() fails:

schema = Schema(
    NestedField(1, "id", LongType()),
    NestedField(2, "_partition_region", StringType()),
    NestedField(3, "region", StringType()),
)
spec = PartitionSpec(PartitionField(source_id=3, field_id=1000, transform=IdentityTransform(), name="region"))
table = catalog.create_table("default.t", schema=schema, partition_spec=spec)
table.append(pa.table({"id": [1], "_partition_region": ["a"], "region": ["us"]}))
# KeyError: 'Field "_partition_region" exists 2 times in schema'

This change picks helper column names that are not already in the table, by prefixing underscores until the name is free. The helper columns are still dropped before the data files are written.

Are these changes tested?

Yes. test_determine_partitions_with_column_named_like_partition_helper in tests/io/test_pyarrow.py partitions a table that has a _partition_region data column and checks that every row lands in the right partition with that column unchanged. It fails without the fix.

Are there any user-facing changes?

Bug fix only: appends to such tables now succeed. No API changes.

This pull request and its description were written by Isaac.

[rahulsmahadev]

@geruh @DG47 could you take a look when you have a chance? It changes how _determine_partitions names its helper partition columns, which you both worked on in #2204 and #4121.

[Fokko]

Sounds like an edge case, but reasonable to add. Thanks @rahulsmahadev