Repository navigation
Fix append to a table partitioned by bucket on a time column - #4343
Open
rahulsmahadev wants to merge 4 commits into
Open
rahulsmahadev wants to merge 4 commits into
rahulsmahadev wants to merge 4 commits into
Conversation
The bucket transform returns an int, but the time partition representation only accepted datetime.time, so every append raised AttributeError. Accept the int, as the date, timestamp and uuid handlers already do. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
Resolve the import conflict in tests/io/test_pyarrow.py with apache#4348. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Looks good to me. This matches how #3695 handles bucketed date/timestamp sources, and the tests cover the int passthrough and the error path. |
rambleraptor
approved these changes
Nov 11, 2026
rambleraptor
left a comment
Collaborator
There was a problem hiding this comment.
This seems reasonable. Thanks for doing this!
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
A table partitioned by
bucket[N]on atimecolumn can be created, but everyappend()to it fails:The spec lists
timeas a source type forbucket[N], and Java supports it. The bucket transform returns anint, which_to_partition_representationalready passes through for date, timestamp and uuid sources, but theTimeTypehandler always calledtime_to_microson it. This change makes theTimeTypehandler accept anintthe same way.Are these changes tested?
Yes. Both new tests fail without the fix:
test_to_partition_representation_timeintests/test_transforms.pycoversNone, atimevalue, and theinta bucket transform returns.test_to_partition_representation_unrecognized_type_raisesnow also coversTimeType.test_append_to_table_bucketed_by_timeintests/io/test_pyarrow.pyappends to a table bucketed on a time column and reads the rows back with and without a filter.Are there any user-facing changes?
Bug fix only: appends to tables partitioned by bucket on a time column now succeed. No API changes.
This pull request and its description were written by Isaac.