[kovashikawa]

Summary

AI summaries fail on self-hosted instances that use AI_PROVIDER=openai-compatible with Ollama. Every recording longer than a few minutes ends in AiUnavailableError: All AI providers failed for the "generation" role, with each attempt logging AI response did not contain a JSON object.

Root cause: transcripts are chunked at a fixed 24,000 characters, and the single-chunk prompt prefixes every segment with [m:ss] . Ollama serves a 4k-token context by default on machines under 24 GB of VRAM (docs), and its /v1 endpoint ignores num_ctx (ollama#5356, ollama#16825). A longer prompt is cut from the front with no error, which removes the JSON instructions, so the model answers in prose. Ollama reports exactly prompt_tokens: 4096 when this happens.

Found while testing #2397 (closed in favour of this PR). The STT side worked, but the AI summary for the same recording never completed.

Changes

• Chains without openai-compatible keep the existing 24,000-character chunker, unchanged from main.
• When openai-compatible is anywhere in the generation chain, including as a fallback, transcripts are chunked by estimated tokens with a 1,800-token budget, which leaves room for the instructions and the JSON reply inside a 4k window. Every provider in the chain receives the same chunks, so a local fallback means sizing for its context.
• The estimate counts about 4 ASCII characters per token, one token per non-ASCII character so CJK text doesn't overflow, and 6 tokens per segment for the timestamp prefix (word-level transcripts have thousands of segments).
• Self-hosting docs: explain the Ollama context limit and how to raise it with OLLAMA_CONTEXT_LENGTH.

Testing

• Unit tests in generate-ai-fallback.test.ts pin the hosted split points: 585 forty-character cues stay in one chunk and 586 split, exactly 24,000 joined characters stay together and 24,001 split, an oversized cue stays whole, and non-ASCII text is still measured by string length. Further tests cover openai-compatible as primary and as fallback, including the timestamp allowance. The AI provider, start, title and fallback suites pass (105 tests), Biome is clean, and bun run typecheck passes.
• Replayed 14 transcripts (10 real recordings of 6 to 33 minutes, plus 4 stitched ones of 66 to 133 minutes) through main and this branch with a mocked provider. For hosted chains (Groq, Groq then OpenAI, Anthropic) every prompt, including the final summary prompt, is byte-identical to main.
• On Ollama 0.24 with gemma3:12b at the default 4,096 context, main failed a 33-minute recording on all four attempts: the prompt was cut from 6,503 to 4,096 tokens and the reply ignored the instructions. This branch completed it in 4 calls with no truncation, and completed the 66, 100 and 133-minute transcripts that main failed or summarized from only part of the recording.
• Section requests stayed under the 4k window on every transcript. The final step that combines sections reached 3,679 tokens at 11 sections and passed 4k at 13, which is why the docs suggest raising the context for very long recordings.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
cap-status	Skipped		Oct 24, 2026 11:18pm UTC

[richiemcilroy]

@kovashikawa thanks for tracking this down, the Ollama truncation write-up is really helpful. One concern before we can merge: the new estimate also changes chunking for hosted providers. In apps/web/workflows/generate-ai.ts each segment now costs estimateTokens(text) + SEGMENT_TIMESTAMP_TOKENS, so a typical 8-word VTT cue (~40 chars) costs about 16 tokens against the 6,000 budget (~375 cues per chunk), where the old 24,000-character limit fit about 585 cues. Shorter punctuation-split cues shift even more. That means hosted recordings in roughly the 15 to 27 minute range would switch from the single-pass summary to the multi-chunk path, with more AI calls and different output. Could you keep the hosted path exactly as it is today (for example keep the MAX_CHARS_PER_CHUNK character logic when openai-compatible is not in the chain and only use the token budget for it), and add a test that a hosted transcript chunks the same as before? Everything else, including the docs note, looks good.

[kovashikawa]

Thanks @richiemcilroy, good catch. Hosted chains now go through the original 24,000-character chunker, unchanged from main, and only chains that include openai-compatible use the 1,800-token budget. That includes a hosted primary with openai-compatible as a fallback, since every provider in the chain receives the same chunks.

The new tests use your numbers: 585 forty-character cues stay in one chunk, 586 split, and exactly 24,000 joined characters stay together. I also replayed 14 transcripts through main and this branch, and hosted chains send byte-identical prompts. One small docs tweak: the note now says the final step that combines sections can still pass 4k tokens on very long recordings.