Skip to content

Skip copying the table metadata when a transaction has nothing staged - #4302

Open
dedyoc wants to merge 1 commit into
apache:mainfrom
dedyoc:perf/transaction-metadata-short-circuit
Open

dedyoc wants to merge 1 commit into
apache:mainfrom
dedyoc:perf/transaction-metadata-short-circuit

Conversation

@dedyoc

@dedyoc dedyoc commented Nov 7, 2026

Copy link
Copy Markdown

Closes #4294

Rationale for this change

Transaction.table_metadata replays the staged updates on every read:

@property
def table_metadata(self) -> TableMetadata:
return update_table_metadata(self._table.metadata, self._updates)

and update_table_metadata always ends in a deep copy:

return new_metadata.model_copy(deep=True)

An append reads the property about 20 times before staging anything, so every read deep copies the full snapshot history. This returns the table's metadata directly when nothing is staged, as @geruh suggested on #3302. Table.metadata already shares that object uncopied.

The reproduction script from #4294, on main with and without this change (one timed append per depth):

Snapshots Before After update_table_metadata calls
100 101.3 ms 18.6 ms 22 → 1
200 105.6 ms 27.9 ms 22 → 1
400 245.6 ms 44.6 ms 22 → 1
800 541.5 ms 81.9 ms 22 → 1

Are these changes tested?

Yes. Two tests in tests/table/test_init.py check that with nothing staged the property returns the table's own metadata object and that staged updates are still applied without changing the table's metadata.

test_snapshot_producer_bounded_metadata_access counts update_table_metadata calls as a stand-in for reads of the property, so it now stages an update first, otherwise every read would skip the call. It still fails if a read is added back to the snapshot producer.

Are there any user-facing changes?

No.

Transaction.table_metadata deep-copied the whole metadata on every read, even
with nothing staged, so appends slowed down as the snapshot history grew.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transaction.table_metadata deep-copies the whole table metadata on every read, making commits quadratic in snapshot count

1 participant