Skip to content

Reserve extra bthread workers for blocking event loops - #3683

Merged
chenBright merged 2 commits into
apache:masterfrom
chenBright:event_dispatcher_concurrency
Oct 29, 2026
Merged

chenBright merged 2 commits into
apache:masterfrom
chenBright:event_dispatcher_concurrency

Conversation

@chenBright

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: resolve #1120

Problem Summary:

When event dispatchers occupy all bthread workers in epoll_wait or kevent,
other bthreads cannot run. This can stall request processing and server shutdown,
particularly when event_dispatcher_num >= num_threads.

What is changed and the side effects?

Changed:

  • Add bthread_add_reserved_concurrency_by_tag(num, tag) to increment reserved concurrency.
  • Reserve an extra worker for each event dispatcher and internal epoll bthread. Reserved workers are
    created eagerly, including when lazy growth is enabled.
  • Track normal and reserved concurrency separately. Keep TaskControl::concurrency() as their total.
  • Use independent per-tag limits of 1024 normal workers and 512 reserved workers, allowing reservations
    even when normal concurrency reaches its limit.

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

@wasphin
wasphin requested a balanced review from Copilot October 27, 2026 14:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Three unresolved moderate findings remain: the shared epoll thread can pin workers to the first caller's bthread tag, dispatcher reservation failures now abort at startup via CHECK_EQ, and a test timeout can leave a thread holding dangling stack references.

3 open issues

Review effort: Lite

Comment thread src/brpc/event_dispatcher.cpp Resolved
Comment thread src/bthread/fd.cpp Resolved
Comment thread test/brpc_event_dispatcher_unittest.cpp Resolved
@chenBright
chenBright requested a review from wasphin October 29, 2026 04:29
@wasphin

wasphin commented Oct 29, 2026

Copy link
Copy Markdown
Member

LGTM

@chenBright
chenBright merged commit bcaf803 into apache:master Oct 29, 2026
29 checks passed
@chenBright
chenBright deleted the event_dispatcher_concurrency branch October 29, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

当event_dispatcher_num 大于 num_threads 时,ctrl c失效

3 participants