Repository navigation
Capture every prompt end in the shared LLM serving prefix cache - #24624
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24624
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
d45a786 to
5e75b20
Compare
|
@claude /pr-review |
|
Claude finished @mergennachin's task in 20m 7s —— View job Summary: The new Detailed reviewPR Review: #24624 SummaryThis PR adds an opt-in
All 10 changed files were reviewed. Code Quality
Testing
API Design
Performance
RecommendationRequest Changes
|
5e75b20 to
2ea9667
Compare
3c7a6d3 to
25c8d8a
Compare
|
@claude /pr-review |
|
Claude finished @mergennachin's task in 20m 35s —— View job Summary: This needs discussion before merge: in prompt-end-only mode each continuation and replacement holds a transient row until it finishes, the flag behaves differently alongside an interval or policy, the new capacity check is untested, and a helper change is unreachable. Detailed reviewPR Review: #24624SummaryThis PR adds opt-in
All 15 changed files were reviewed. Code Quality
Testing
API Design
Performance
RecommendationNeeds Discussion
|
|
@claude /pr-review |
|
Claude finished @mergennachin's task in 33m 18s —— View job Request Changes: turning per-session prompt-end checkpoints on by default for Muse-Glimmer quadruples the default executor rows (4 → 16), which breaks existing 65536-cell CUDA exports and command lines. Separately, Detailed reviewPR Review: #24624SummaryThe PR gives each named session in
Code Quality
Testing
API Design
Backward Compatibility
RecommendationRequest Changes What blocks approval:
Review only: no code was changed, and nothing was built or run. |
|
Prefix caching configuration feels like getting complicated to me. Enabling --prefix-checkpoint-prompt-end saves every session request’s prompt end instead of only the initial prompt, and changes --prefix-cache-entries 8 from 8 checkpoints total to 8 per session. What do you think about separating checkpoint ownership from capture policy? Ownership: shared across sessions, or private per session. These can be expressed separately, for example: The entry limit would apply to the selected scope. We can still reject the unsupported combinations. However, separating ownership from capture policy might not worth redesigning if we don’t plan to support more combinations. For example, do we expect to support other capture policies with per-session ownership in the future, or is per-session caching intended to be limited to prompt-end checkpoints? Could be a follow up work as well. |
|
Agreed, it got too complicated. Rather than splitting, I'm removing per-session ownership from this PR. If we need per-session ownership later, --prefix-cache-scope shared|session can be added without breaking anything, with the entry limit applying to the selected scope as you suggested. We'll have three options: --prefix-cache-entries N: total shared checkpoints; 0 turns caching off |
Per-session ownership gave anonymous requests no reuse, multiplied reserved rows by max_sessions, cleared checkpoints on idle eviction, and gave --prefix-cache-entries two meanings. Prompt-end capture now uses the shared cache, selected with --prefix-checkpoint-prompt-end initial|every, and stays opt-in for Muse-Glimmer. Both settings skip prompt ends already retained and release capture rows once the clone settles; the control thread is notified only then, so collecting a capture never waits on the engine.
This PR needs a
|
|
@claude /pr-review |
|
Claude finished @mergennachin's task in 29m 18s —— View job Needs discussion: with Detailed reviewPR Review: #24624SummaryThe PR adds Code Quality
Testing
Performance
RecommendationNeeds Discussion
Review steps
This was a review only: no code was changed, and nothing was built or run. |
Skipping a prompt end that is already retained now refreshes its LRU position, as re-inserting it did before, so default-mode eviction order matches main without the extra clone; the Muse-Glimmer batching test now expects that one fewer clone. The docs, header comments and help text now say what each mode captures, when a snapshot is published, that `every` mainly helps the capturing session after an interrupted turn, and that MLX snapshots each keep their own KV buffers. New tests cover the retry of an interrupted continuation, wrap's immediate notification, the refreshed recency, and retention of a cancelled request's clone.
|
@claude Review |
|
Claude finished @mergennachin's task in 6m 18s —— View job Approve: I found no blockers. Two small things are worth fixing before merge: Detailed reviewPR Review: #24624SummaryThe PR adds API Design
Code Quality
RecommendationApprove
Review steps
Review only: no code was changed, and nothing was built or run. |
kiymetakdemir
left a comment
There was a problem hiding this comment.
Overall looks good to me. My initial concern was that "every" lets one long conversation evict other sessions' prompt ends from the shared cache. But the design leaves room for more modes. For example, a mode could keep each session's initial and newest prompt end in the shared cache (at most 2 entries per session), or just the newest. IIUC that would still cover the cancel, stop-string and failure retries this PR targets, while capping how much one session can evict.
It's similar to what you had initially, but without private ownership: entries stay in the shared cache, --prefix-cache-entries stays a single total. That would need captures to record which session saved them and could be a follow-up.
| capture_lanes_used_.fetch_sub(1); | ||
| checkpoint_lane_busy_.store(false); | ||
| if (capture) { | ||
| if (uses_checkpoint_observer(config_)) { |
There was a problem hiding this comment.
Claude raised this concern might worth to double check
"The loop picks the cleanup for a finished capture from the config (uses_checkpoint_observer(config_)), but the right cleanup depends on what kind of capture it is. This only works because a runtime can't produce both kinds today. If every is ever allowed together with an interval, a prompt-end capture would get the checkpoint cleanup: it would free the checkpoint lane it doesn't own and later free its slot a second time. Could this check the request instead?"
Oh yeah, that's a good idea so that one session doesn't destroy everything. Let's do in a follow-up PR. Created an issue for remembering later on #24914 |
Use the request capture_lane flag to select prompt-end cleanup instead of inferring ownership from runtime configuration. Observer captures continue to release their checkpoint lane. Validated with 19 focused capture lifecycle tests and all 299 serving tests. Formatting and whitespace checks pass, with no new cppcheck findings. Authored with OpenAI Codex.
Reading a prompt is the slow first step of answering it: the model processes every token to build its internal state (prefill). With prefix caching on, the serving runtime saves that state as a snapshot, so a later request that starts with the same tokens can skip re-reading it.
Today the server saves a snapshot only for the prompt that creates a session: every anonymous request, and the first request of a named session. Later turns of a named session aren't saved. Usually that doesn't matter, because the next turn extends the session's live state. It matters when the live state can't be reused: after a turn is cancelled, cut short by a stop string, or fails, the retry or next turn re-reads everything after the first prompt. On MLX the same happens when a long conversation goes back to an earlier turn, because the live session can't rewind far once its sliding-window buffer wraps.
This PR adds an option to save the prompt end of every request:
Use
--prefix-checkpoint-prompt-end everyto turn it on. It's off by default in both the generic and Muse-Glimmer servers, requires--prefix-cache-entries > 0, and can't be combined with--prefix-checkpoint-interval.The gain is mostly within one session. The Python servers splice a session's own generated tokens, including reasoning, into its earlier assistant turns. Another session, an anonymous request, or the same session after it is evicted sends those turns as re-rendered text instead, so with reasoning models such as Muse-Glimmer it usually matches the session's saved turns only up to the first assistant turn, which is already saved today.
Snapshots go into the existing shared cache and count against
--prefix-cache-entries, so the row count and the worst case don't change. Withevery, each turn keeps its own entry until the cache evicts it. Keeping only each session's newest snapshot would use fewer entries but lose the earlier turns, so there's no per-session limit; size the total for the sessions and turns you want to keep. On MLX, snapshots don't share KV memory with each other or with the live session, so actual memory grows with the number and length of saved snapshots; use fewer entries at long contexts. CUDA snapshots share cells by reference.The default mode changes in two small ways when no interval is set. A prompt that's already saved isn't copied again; its entry is refreshed instead, as re-inserting it did before. And the temporary slot used to make a copy is freed once the copy is published, instead of when the response finishes. With an interval, an already saved prompt end is still copied and then dropped, as before.
An earlier version of this PR kept a separate cache for each session. Based on review feedback, it now uses the shared cache, so
--prefix-cache-entrieskeeps a single meaning. A--prefix-cache-scope shared|sessionoption can be added later without breaking anything.The new flag makes argparse abbreviations such as
--prefix-checkpointambiguous; nothing in the repo uses them.Review order:
serving_runtime.handprefix_cache.h, thenserving_runtime.cpp. The rest is tests, launchers, and docs.Tested with the serving and batching C++ suites (299 and 227 tests), the launcher Python tests, and the Muse-Glimmer batching test built against the MLX runtime. A local run of Muse-Glimmer 30B on MLX retried a 561-token turn that was cancelled after its first token: it reused 560 tokens with
every, 221 withinitial, and none without the cache, and prefill took 0.6 s, 6.9 s, and 11.4 s. Greedy outputs after a cache hit can drift late from a full prefill, withinitialas well asevery.Authored with OpenAI Codex and Claude Code.