Skip to content

fix(bthread): Destroy pooled KeyTables outside cache locks - #3682

Open
darion-yaphet wants to merge 1 commit into
apache:masterfrom
darion-yaphet:fix/keytable-pool-destructors-outside-lock
Open

darion-yaphet wants to merge 1 commit into
apache:masterfrom
darion-yaphet:fix/keytable-pool-destructors-outside-lock

Conversation

@darion-yaphet

Copy link
Copy Markdown
Contributor

Destroying a KeyTable may run a user-provided TLS destructor. The destructor can wait for another bthread or access the same pool.

Previously, pool destruction deleted cached KeyTables while holding both the pool write lock and ThreadLocal's internal mutex. A user destructor could therefore wait for a bthread that needs one of these locks, causing a deadlock.

The cleanup now works in two phases:

  1. Under the cache locks, mark the pool destroyed and detach cached KeyTables into a local list.
  2. Release the locks, then destroy the detached KeyTables and run their TLS destructors.

Summary

Pool destruction currently deletes cached KeyTables while holding both the pool write lock and ThreadLocal's internal mutex. TLS destructors can execute user code, which may wait for or interact with tasks that need these locks.

Detach KeyTable chains while the cache locks protect the pool structure, then destroy the detached KeyTables after releasing both locks. Delete the now-empty thread-local cache container afterward.

Add a regression test that attempts to acquire both locks from a TLS destructor. The test fails before the fix and verifies that both locks are available after it.

Testing

  • cmake --build build --target bthread_key_unittest -j6
  • bthread_key_unittest --gtest_filter=KeyTest.* — 12 tests passed.
  • KeyTest.pooled_destructor_runs_outside_cache_locks repeated 30 times — passed.

Risk and Rollback

The change preserves the pool shutdown sequence while moving user destructor execution outside cache-management locks. It does not change public APIs.

Detach thread-local cached keytables while holding the pool and cache locks, then destroy them after releasing both locks. Add a regression test that checks the locks from the TLS destructor.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused cleanup change preserves shutdown synchronization and has targeted regression coverage, with no blocking issues identified.

No open issues 🎉

What changed in this PR

This PR addresses a bthread keytable-pool shutdown deadlock by moving TLS destructor execution outside cache-management locks, without changing public APIs.

Changes:

  • Detach cached KeyTable chains under the locks, then destroy them after releasing both locks.
  • Add a regression test that verifies TLS destructors can acquire both cache-management locks.
File Description
test/​bthread_key_unittest.cpp Adds regression coverage for lock availability during pooled TLS destruction.
src/​bthread/​key.cpp Separates cache detachment from KeyTable destruction during pool shutdown.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants