Skip to content

[flink] Support managed memory configuration for writer slot sharing groups - #10767

Open
taoran92 wants to merge 1 commit into
apache:masterfrom
taoran92:feature/flink-writer-managed-memory
Open

taoran92 wants to merge 1 commit into
apache:masterfrom
taoran92:feature/flink-writer-managed-memory

Conversation

@taoran92

@taoran92 taoran92 commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Purpose

Closes #10765.

When sink.writer-memory configures an explicit writer slot sharing group, the group currently requests zero managed memory. Changing sink.managed.writer-buffer-memory only changes the operator's allocation weight, so users cannot request heap and managed resources independently.

Add the optional sink.writer-managed-memory property for ordinary writers and the CDC combined sink. It requires sink.writer-memory and a positive value, and sets the managed resource request for each shared slot of the writer group. For example, a writer group can request 512 MiB of task heap and 128 MiB of managed memory while retaining an operator weight of 256.

The option does not enable the managed allocator or change allocation weights. Existing behavior when the option is absent, downstream slot sharing, and explicit committer resources are preserved. Update the configuration guide and generated option reference to explain these boundaries.

Tests

  • WriterManagedMemoryTest
  • FlinkCdcMultiTableSinkTest
  • FlinkSinkTest
  • WriterChainingStrategyTest

The tests cover StreamGraph and JobGraph resources, allocator enabled and disabled, configuration validation, legacy behavior, and explicit committer resources. All tests passed.

@taoran92
taoran92 marked this pull request as ready for review October 23, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support managed memory configuration for Flink writer slot sharing groups

1 participant