[jkyberneees]

Summary

• Token selection runs on the GPU, inside the same command buffer as the forward pass: repetition penalty, exact top-k (adaptive-threshold pool, rank/bitonic sort), min-p, top-p and a seeded xorshift draw. The token lands in a per-position history buffer that the next step's embedding lookup reads, so the engine commits step N+1 before reading back step N and the GPU runs steps back to back.
• RMSNorm kernels use float4 loads with up to 1024 threads per row (Mistral 7B: the three norms per layer cost 0.21 ms instead of 0.62 ms per step).
• Speculative decoding and constrained tool-call names still sample on the CPU. Seeded sampling now uses the GPU's xorshift stream (a seed gives a different, still deterministic, sequence than 1.1.0); top-k above 1024 is capped.
• mlxr bench measures the pipelined decode, the app's default sampler and the token-selection cost; profile prints a per-kernel-class breakdown of one decode step.
• Version 1.1.1, CHANGELOG and README performance table updated.

Numbers (M2 Pro, 4 alternating pairs against 1.1.0)

Model	decode @16	decode @1k	real generation (mlxr run)
Qwen3 0.6B	+10.7%	+9.0%	238 → 282 tok/s
Llama 3.2 1B	+8.5%	+7.8%	180 → 201 tok/s
Mistral 7B	+4.1%	+4.0%	39.3 → 40.9 tok/s

Prefill is unchanged. Greedy output is token-identical to main on Qwen3 0.6B, Qwen2.5 0.5B, Llama 3.2 1B, Phi-3.5 and Mistral 7B.

Measured and rejected: a 2-rows-per-simdgroup matvec for small matrices (faster in isolated sweeps, 5% slower in the real decode step) and a ballot-based counting pass in the sampler.

Test plan

• swift test (116 tests) passes; new SampleKernelTests compare the GPU sampler with the CPU Sampler (greedy with ties and penalty, drawn distributions for four top-k / top-p / min-p settings, tied maxima, tiny vocabularies, huge top-k, non-finite logits).
• Greedy output identical on five architectures (mlxr run --temp 0, 300 tokens, long prompt).
• prefillMatchesIncrementalDecode and the server suites pass on the real models.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

🚀 Deploying Preview to Cloudflare 🚀

Preview Deployments by commit

Status	Deployment URL	Commit	Updated (UTC)	See this deployment's details
Build: Failed ❌ View logs ↗		af461c6	2026-11-01T08:57:03.476Z	View logs ↗