Skip to content

Fix append to a table partitioned by bucket on a time column - #4343

Open
rahulsmahadev wants to merge 4 commits into
apache:mainfrom
rahulsmahadev:rahul/bucket-time-partition
Open

rahulsmahadev wants to merge 4 commits into
apache:mainfrom
rahulsmahadev:rahul/bucket-time-partition

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

A table partitioned by bucket[N] on a time column can be created, but every append() to it fails:

schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "event_time", TimeType()))
spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, transform=BucketTransform(4), name="event_time_bucket"))
table = catalog.create_table("default.t", schema=schema, partition_spec=spec)
table.append(pa.table({"id": [1], "event_time": pa.array([time(22, 31, 8)], pa.time64("us"))}))
# AttributeError: 'int' object has no attribute 'hour'

The spec lists time as a source type for bucket[N], and Java supports it. The bucket transform returns an int, which _to_partition_representation already passes through for date, timestamp and uuid sources, but the TimeType handler always called time_to_micros on it. This change makes the TimeType handler accept an int the same way.

Are these changes tested?

Yes. Both new tests fail without the fix:

  • test_to_partition_representation_time in tests/test_transforms.py covers None, a time value, and the int a bucket transform returns. test_to_partition_representation_unrecognized_type_raises now also covers TimeType.
  • test_append_to_table_bucketed_by_time in tests/io/test_pyarrow.py appends to a table bucketed on a time column and reads the rows back with and without a filter.

Are there any user-facing changes?

Bug fix only: appends to tables partitioned by bucket on a time column now succeed. No API changes.

This pull request and its description were written by Isaac.

rahulsmahadev and others added 3 commits November 10, 2026 23:52
The bucket transform returns an int, but the time partition representation
only accepted datetime.time, so every append raised AttributeError. Accept
the int, as the date, timestamp and uuid handlers already do.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@Fokko @mittalpk could you take a look when you have a chance? It is a small fix in _to_partition_representation (last changed in #3695) so appends work for bucket partitions on a time column.

Resolve the import conflict in tests/io/test_pyarrow.py with apache#4348.

Co-authored-by: Isaac <no-reply@databricks.com>
@mittalpk

Copy link
Copy Markdown
Contributor

Looks good to me. This matches how #3695 handles bucketed date/timestamp sources, and the tests cover the int passthrough and the error path.

@rambleraptor rambleraptor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable. Thanks for doing this!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants