Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 22 additions & 4 deletions examples/llm_server/cpp/module_worker_main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,15 @@ DEFINE_int32(
DEFINE_int32(
prefix_checkpoint_interval,
0,
"Checkpoint interval in tokens; 0: initial prompt only. Uses global LRU.");
"Periodic checkpoint interval in tokens; 0 disables periodic capture.");
DEFINE_int32(
prefix_capture_slots,
0,
"Transient capture rows; 0: size automatically from session/request limits");
DEFINE_string(
prefix_checkpoint_prompt_end,
"initial",
"Prompt ends to capture, best effort: initial (requests that create a session) or every (every request)");
DEFINE_int32(max_session_tokens, 2048, "Maximum retained tokens per session");
DEFINE_int32(max_decode_sequences, 2, "Decode slots in one packed forward");
DEFINE_int32(
Expand All @@ -48,23 +52,37 @@ DEFINE_int32(

int main(int argc, char** argv) {
gflags::ParseCommandLineFlags(&argc, &argv, true);
const bool every_prompt_end = FLAGS_prefix_checkpoint_prompt_end == "every";
if (!every_prompt_end && FLAGS_prefix_checkpoint_prompt_end != "initial") {
std::cerr << "--prefix_checkpoint_prompt_end must be initial or every\n";
return 1;
}
if (every_prompt_end && FLAGS_prefix_checkpoint_interval) {
std::cerr << "--prefix_checkpoint_prompt_end=every cannot be combined with "
"--prefix_checkpoint_interval\n";
return 1;
}
if (FLAGS_pte.empty() || FLAGS_tokenizer.empty() || FLAGS_max_sessions <= 0 ||
FLAGS_max_session_tokens <= 0 || FLAGS_max_decode_sequences <= 0 ||
FLAGS_max_inflight_requests <= 0 || FLAGS_prefix_cache_entries < 0 ||
FLAGS_prefix_capture_slots < 0 || FLAGS_prefix_checkpoint_interval < 0 ||
(FLAGS_prefix_checkpoint_interval > 0 &&
((FLAGS_prefix_checkpoint_interval > 0 || every_prompt_end) &&
FLAGS_prefix_cache_entries == 0)) {
std::cerr
<< "--pte and --tokenizer are required; capacity limits must be positive"
"; prefix-cache limits must be nonnegative; "
"checkpoints require prefix_cache_entries > 0\n";
"--prefix_checkpoint_interval and --prefix_checkpoint_prompt_end=every require --prefix_cache_entries > 0\n";
return 1;
}
executorch::extension::llm::serving::ServingRuntimeConfig config;
namespace serving = executorch::extension::llm::serving;
serving::ServingRuntimeConfig config;
config.max_sessions = FLAGS_max_sessions;
config.prefix_cache_capacity = FLAGS_prefix_cache_entries;
config.max_prefix_captures = FLAGS_prefix_capture_slots;
config.prefix_checkpoint_interval = FLAGS_prefix_checkpoint_interval;
config.prefix_checkpoint_prompt_end = every_prompt_end
? serving::PromptEndCapture::Every
: serving::PromptEndCapture::Initial;
config.max_context_length = FLAGS_max_session_tokens;
config.max_requests = FLAGS_max_inflight_requests;
config.max_pending_operations = FLAGS_max_inflight_requests;
Expand Down
18 changes: 16 additions & 2 deletions examples/llm_server/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,23 @@ limit; choose a value supported by the export. Capacity defaults are
`--prefix-cache-entries 0`. With prefix caching enabled, `--prefix-capture-slots 0`
(the default) automatically allows one pending prompt capture per admitted request,
bounded by `min(max_sessions, max_inflight_requests)`. A positive value overrides
this allowance. With checkpoints enabled, automatic sizing uses at most two rows:
this allowance; with `every`, lower values can leave replacements and retries
without a row, so they prefill cold. `--prefix-checkpoint-prompt-end initial` (the
default) captures the prompt end of each request that creates a session, including
every anonymous request. `every` captures the prompt end of every request and
requires `--prefix-cache-entries > 0`; each captured turn takes a cache entry.
Without periodic checkpoints, a prompt end already in the cache is refreshed instead
of cloned again, and a snapshot becomes reusable and frees its capture slot at the
source request's first output after the clone completes, or when the request
finishes. Capture is best effort. See the [batching guide](../spec/batching.md) for
what `every` reuses in practice and its MLX memory cost. Retained snapshots are
shared by all clients, and `cached_tokens` reports matches, so enable prefix caching
only when clients trust each other.
With periodic checkpoints enabled, automatic sizing uses at most two rows:
checkpoint collection is serialized, but one replacement can overlap. Explicit
values above two add unused row capacity in that mode.
`--prefix-checkpoint-prompt-end every` cannot be combined with a nonzero
`--prefix-checkpoint-interval`; the launcher and native worker reject that combination.
Logical sessions, packed decode width, and in-flight request capacity are independent
limits. Python concurrency follows advertised request capacity, not the single
native delivery thread. Python reads native async streams directly through bounded
Expand Down Expand Up @@ -299,7 +313,7 @@ the generic control plane never imports an example. Backend specifics

The dedicated batching launcher supports concurrent text requests on one shared
runner, ordered persistent-session continuation, targeted in-band cancellation,
and optional prefix reuse for new sessions and clean prompt replacements. It
and optional prefix reuse for new sessions, retries, and prompt replacements. It
does not yet migrate Muse image or DFlash execution. The serialized-worker
limitations below describe the legacy path, not the new batching worker.

Expand Down
4 changes: 2 additions & 2 deletions examples/llm_server/python/protocol.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,8 +162,8 @@ def resolved_max_tokens(self) -> int:


class PromptTokensDetails(BaseModel):
# Warm-resume accounting, SGLang-compatible shape: tokens served from the
# session's resident state instead of prefetched this request.
# Warm-resume accounting, SGLang-compatible shape: tokens reused from the
# session's resident state or a shared prefix snapshot instead of prefilled.
cached_tokens: int = 0


Expand Down
64 changes: 46 additions & 18 deletions examples/llm_server/python/serve.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,23 +40,35 @@ def _parse_preflight_timeout(value):
return timeout


def _validate_limits(parser, args, *, prefix_capture_slots=0) -> int:
for flag, value in (
("--max-context", args.max_context),
("--max-sessions", args.max_sessions),
("--max-decode-sequences", args.max_decode_sequences),
("--max-inflight-requests", args.max_inflight_requests),
def _validate_limits(
parser, args, *, prefix_capture_slots=0, prefix_checkpoint_prompt_end="initial"
) -> int:
for flag, value, minimum in (
("--max-context", args.max_context, 1),
("--max-sessions", args.max_sessions, 1),
("--max-decode-sequences", args.max_decode_sequences, 1),
("--max-inflight-requests", args.max_inflight_requests, 1),
("--prefix-cache-entries", args.prefix_cache_entries, 0),
("--prefix-checkpoint-interval", args.prefix_checkpoint_interval, 0),
("--prefix-capture-slots", prefix_capture_slots, 0),
):
if not 1 <= value <= _MAX_INT32:
parser.error(f"{flag} must be between 1 and {_MAX_INT32}")
if not 0 <= args.prefix_cache_entries <= _MAX_INT32:
parser.error(f"--prefix-cache-entries must be between 0 and {_MAX_INT32}")
if not 0 <= args.prefix_checkpoint_interval <= _MAX_INT32:
parser.error(f"--prefix-checkpoint-interval must be between 0 and {_MAX_INT32}")
if args.prefix_checkpoint_interval and not args.prefix_cache_entries:
parser.error("--prefix-checkpoint-interval requires --prefix-cache-entries > 0")
if not 0 <= prefix_capture_slots <= _MAX_INT32:
parser.error(f"--prefix-capture-slots must be between 0 and {_MAX_INT32}")
if not minimum <= value <= _MAX_INT32:
parser.error(f"{flag} must be between {minimum} and {_MAX_INT32}")
every_prompt_end = prefix_checkpoint_prompt_end == "every"
if every_prompt_end and args.prefix_checkpoint_interval:
parser.error(
"--prefix-checkpoint-prompt-end every cannot be combined with "
"--prefix-checkpoint-interval"
)
if (
args.prefix_checkpoint_interval or every_prompt_end
) and not args.prefix_cache_entries:
flag = (
"--prefix-checkpoint-prompt-end every"
if every_prompt_end
else "--prefix-checkpoint-interval"
)
parser.error(f"{flag} requires --prefix-cache-entries > 0")
# Mirrors ServingRuntimeConfig::resolved_prefix_captures for row validation.
if prefix_capture_slots == 0:
prefix_capture_slots = min(args.max_sessions, args.max_inflight_requests)
Expand Down Expand Up @@ -105,6 +117,12 @@ def _parse_args(argv=None):
parser.add_argument("--max-decode-sequences", type=int, default=8)
parser.add_argument("--max-inflight-requests", type=int, default=64)
parser.add_argument("--prefix-cache-entries", type=int, default=0)
parser.add_argument(
"--prefix-checkpoint-prompt-end",
choices=("initial", "every"),
default="initial",
help="Prompt ends to capture, best effort: initial (requests that create a session) or every (every request)",
)
parser.add_argument(
"--prefix-capture-slots",
type=int,
Expand All @@ -115,7 +133,7 @@ def _parse_args(argv=None):
"--prefix-checkpoint-interval",
type=int,
default=0,
help="Checkpoint interval in tokens; 0: initial prompt only. Uses global LRU.",
help="Periodic checkpoint interval in tokens; 0 disables periodic capture.",
)
parser.add_argument(
"--api-dialect", choices=("openai", "openrouter"), default="openai"
Expand All @@ -130,7 +148,12 @@ def _parse_args(argv=None):
parser.add_argument("--port", type=int, default=8000)
args = parser.parse_args(argv)

_validate_limits(parser, args, prefix_capture_slots=args.prefix_capture_slots)
_validate_limits(
parser,
args,
prefix_capture_slots=args.prefix_capture_slots,
prefix_checkpoint_prompt_end=args.prefix_checkpoint_prompt_end,
)

args.worker_bin = os.path.expanduser(args.worker_bin)
if shutil.which(args.worker_bin) is None:
Expand Down Expand Up @@ -205,6 +228,11 @@ async def serving_factory():
]
if args.prefix_checkpoint_interval
else []
)
+ (
["--prefix_checkpoint_prompt_end", "every"]
if args.prefix_checkpoint_prompt_end == "every"
else []
),
stderr=stderr,
)
Expand Down
73 changes: 64 additions & 9 deletions examples/llm_server/python/tests/test_serve_launcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,19 +157,66 @@ def test_checkpoint_options_reach_worker(argv, launch):
assert command[command.index("--prefix_checkpoint_interval") + 1] == "512"


def test_prompt_end_capture_reaches_worker_and_requires_cache(argv, launch, capsys):
assert serve._parse_args(argv).prefix_checkpoint_prompt_end == "initial"
for value, error in (
("every", "requires --prefix-cache-entries > 0"),
("always", "invalid choice"),
):
with pytest.raises(SystemExit):
serve._parse_args(argv + ["--prefix-checkpoint-prompt-end", value])
assert error in capsys.readouterr().err
serve.main(
argv
+ ["--prefix-cache-entries", "3", "--prefix-checkpoint-prompt-end", "every"]
)
command = launch.stages["spawn"].call_args.args[0]
assert command[command.index("--prefix_checkpoint_prompt_end") + 1] == "every"


def test_every_prompt_end_rejects_periodic_capture(argv, capsys):
with pytest.raises(SystemExit):
serve._parse_args(
argv
+ [
"--prefix-cache-entries",
"3",
"--prefix-checkpoint-prompt-end",
"every",
"--prefix-checkpoint-interval",
"64",
]
)
assert "cannot be combined" in capsys.readouterr().err
serve._parse_args(
argv
+ [
"--prefix-cache-entries",
"3",
"--prefix-checkpoint-prompt-end",
"initial",
"--prefix-checkpoint-interval",
"64",
]
)


@pytest.mark.parametrize(
"sessions,requests,interval,override,expected",
"sessions,requests,interval,override,expected,prompt_end",
[
(16, 8, 0, 0, 8),
(4, 8, 0, 0, 4),
(16, 8, 64, 0, 2),
(1, 8, 64, 0, 1),
(16, 8, 0, 1, 1),
(16, 8, 64, 3, 3),
(16, 8, 0, 0, 8, False),
(16, 8, 0, 0, 8, True),
(4, 8, 0, 0, 4, False),
(4, 8, 0, 0, 4, True),
(16, 8, 64, 0, 2, False),
(1, 8, 64, 0, 1, False),
(16, 8, 0, 1, 1, False),
(16, 8, 0, 1, 1, True),
(16, 8, 64, 3, 3, False),
],
)
def test_capture_capacity_validation_and_worker_override(
argv, launch, sessions, requests, interval, override, expected
argv, launch, sessions, requests, interval, override, expected, prompt_end
):
serve.main(
argv
Expand All @@ -185,9 +232,11 @@ def test_capture_capacity_validation_and_worker_override(
"--prefix-capture-slots",
str(override),
]
+ (["--prefix-checkpoint-prompt-end", "every"] if prompt_end else [])
)
command = launch.stages["spawn"].call_args.args[0]
assert command[command.index("--prefix_capture_slots") + 1] == str(override)
assert ("--prefix_checkpoint_prompt_end" in command) == prompt_end
args = SimpleNamespace(
max_context=1024,
max_sessions=sessions,
Expand All @@ -198,7 +247,13 @@ def test_capture_capacity_validation_and_worker_override(
port=8000,
)
assert (
serve._validate_limits(Mock(), args, prefix_capture_slots=override) == expected
serve._validate_limits(
Mock(),
args,
prefix_capture_slots=override,
prefix_checkpoint_prompt_end="every" if prompt_end else "initial",
)
== expected
)


Expand Down
31 changes: 24 additions & 7 deletions examples/llm_server/spec/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,9 +112,10 @@ Non-streaming response: `chat.completion` with one `choice`
∈ `stop` | `length` | `tool_calls`) and a `usage` block.
When reasoning is returned, `message.reasoning_content` contains the extracted
reasoning text separately from visible `content` and `tool_calls`.
`usage.prompt_tokens_details.cached_tokens` reports prompt tokens served from
the session's resident state instead of prefetched this request (0 when the
turn fully prefilled); the streaming usage chunk carries the same field.
`usage.prompt_tokens_details.cached_tokens` reports prompt tokens reused from
the session's resident state or a shared prefix snapshot instead of prefilled
this request (0 when the turn fully prefilled); the streaming usage chunk
carries the same field.

Adapters with a generated-token counter also return
`usage.completion_tokens_details.reasoning_tokens`. It counts actual generated
Expand Down Expand Up @@ -251,7 +252,7 @@ identities; tokenization, decoding, stop handling, and session history belong to
the runtime. Chat templates and OpenAI response presentation remain in Python.
The legacy cancellation and cross-session cache limitations above do not describe
this opt-in path. See the [batching-backed serving guide](batching.md) for launch
selection, creation-only caching, and native-backed Python integration checks.
selection, prompt-boundary caching, and native-backed Python integration checks.

Before accepting requests the worker emits one readiness record, with no request
ID:
Expand Down Expand Up @@ -390,15 +391,31 @@ batched cache via `cache::kind::kBatched`, derives scheduler width from
`preferred_batch_tokens()`, and constructs
one `ServingRuntime`. It does not preload `forward`, run token-step loops, or use
Glimmer's legacy session implementation. Prefix caching is disabled by default.
`--prefix_cache_entries=N` enables creation-only reuse for both greedy and
`--prefix_cache_entries=N` enables retained-prefix reuse for both greedy and
sampled requests. With `--prefix_capture_slots=C`, it adds `N + C` physical rows
for retained snapshots and transient captures; logical `--max_sessions` remains
unchanged. The default `--prefix_capture_slots=0` resolves `C` automatically to
`min(max_sessions, max_inflight_requests)`, so overlapping requests can each capture
their initial prompt. With checkpoints enabled, automatic sizing uses at most two
their prompt. `--prefix_checkpoint_prompt_end=initial` (the default) captures the
prompt end of each request that creates a session, including every anonymous
request but not the first generation after open or reset.
`--prefix_checkpoint_prompt_end=every` captures the prompt end of every request
with the same concurrent allowance; it requires nonzero cache capacity. Without
periodic checkpoints, a prompt end already in the cache is
refreshed instead of cloned again, and a snapshot is published and its transient
slot released at the source request's first output after the clone completes, or
when the request finishes.
With periodic checkpoints enabled, automatic sizing uses at most two
rows because checkpoint collection remains serialized and one replacement can
overlap. A positive value overrides automatic sizing; values above two add unused
row capacity in checkpoint mode. Disabled prefix caching reserves no capture rows.
row capacity in periodic mode. Disabled prefix caching reserves no capture rows.
Capture is best effort. `--prefix_checkpoint_prompt_end=every` cannot be combined
with a nonzero checkpoint interval or a custom checkpoint policy; invalid
combinations are rejected. A C++ `prefix_checkpoint_policy` overrides the
interval and prompt-end capture.
Periodic/custom checkpoints share a serialized lane; a pending clone can skip
another boundary. See the [batching-backed serving guide](batching.md) for what
`every` reuses in practice and its MLX memory cost.

With an existing MLX-enabled ExecuTorch installation and matching gflags package,
configure the MLX LLM examples and build `llm_worker`. Launch with `--pte`,
Expand Down
Loading