Repository navigation
feat(web): OpenAI-compatible STT endpoint for self-hosted transcription (whisper.cpp, faster-whisper) - #2397
kovashikawa wants to merge 5 commits into
Conversation
…ption Adds STT_BASE_URL / STT_MODEL / STT_API_KEY so self-hosters can transcribe with whisper.cpp, faster-whisper-server, speaches, or any endpoint that serves /audio/transcriptions with word timestamps. When set it takes precedence over AssemblyAI for recorded videos and editable-transcript backfills; live transcription stays AssemblyAI-only. Word timings feed the same createEditTranscript pipeline, so captions and the editable transcript are produced from one pass like the AssemblyAI path. whisper.cpp BPE tokens are folded back into words.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
- Fold whisper.cpp BPE tokens per token instead of per segment, so one-word
segments (" timest" + "amps") stay whole and providers that don't mark word
starts keep their words separate.
- Stretch zero-length words (whisper.cpp fillers like "Um") toward the next
word so the edit transcript keeps them for filler removal.
- Fall back to the last word end when the response has no duration, so words
are not clamped away on videos without a stored duration.
- Throw FatalError for 4xx responses, non-JSON bodies, and missing word
timestamps so deterministic misconfiguration is not retried; name the
endpoint on network errors and refuse redirects.
- Accept a blank STT_BASE_URL instead of failing env validation.
- Map full language names ("english") to codes.
- Docs: --vad for whisper-server, host.docker.internal on Apple Silicon,
STT_MODEL default, no AssemblyAI fallback.
|
Thanks for rebuilding this on the provider layer. We need Greptile at 5/5 on the latest commit and green CI before review. Please make sure the three open threads (word boundaries without leading spaces, the missing-duration fallback, zero-duration words) are handled, resolve them, and trigger a Greptile re-review. |
- Take word starts from the segment text when tokens align with it, so
responses that mark only some word starts (" Hello" + "world") keep
their words separate. The leading-space heuristic stays as the fallback.
- Treat a non-positive response duration as missing, so the last-word-end
fallback applies instead of clamping every word away.
|
@greptileai review |
When a zero-length word shares its start with the next word, take the span from the silence before it, or split the next word's time, instead of overlapping it. Trim comments that restated the code.
|
@greptileai review |
Process zero-length words back to front so the last word of a run takes its span first and the earlier ones split it, instead of being dropped.
|
@greptileai review |
|
@richiemcilroy Greptile is at 5/5 on the latest commit and all threads are resolved. CI is waiting on workflow approval since this is from a fork; could you approve the runs when you get a chance? |
|
Closing in favour of #2771, which fixes the AI summary failure I hit while testing this branch with a local OpenAI-compatible model. |
|
Thanks @kovashikawa, this looks good to go from our side. The branch is quite far behind main and we can't update it from here (maintainer edits are off), so could you merge the latest main into it? Once CI is green on the updated branch we'll merge. |
|
Thanks @richiemcilroy! I closed this in favour of #2771, which fixes the AI summary failure with local OpenAI-compatible models. Could you review that one instead? |
|
I picked this up in #3523: rebased onto current main with @kovashikawa credited as co-author, plus a few fixes noted there. Thanks for the original work! |
Summary
Adds an OpenAI-compatible transcription option so self-hosters can transcribe recorded videos without AssemblyAI, using whisper.cpp
whisper-server, faster-whisper-server, speaches, or any endpoint that serves/audio/transcriptionswith word timestamps.STT_BASE_URL=http://whisper:9000/v1 STT_MODEL=large-v3-turbo STT_API_KEY= # optionalThis is the STT half of #1877, rebuilt on top of the provider layer from #2113 (which already covers the LLM side via
AI_PROVIDER=openai-compatible).Behavior
STT_BASE_URLset: recorded-video transcription and editable-transcript backfills use it. Otherwise AssemblyAI is used exactly as today.response_format=verbose_jsonwithtimestamp_granularities[]=word. The org's AI language is passed aslanguageunless it is auto-detect.createEditTranscriptpipeline, so a single pass produces both the caption VTT and the editable word transcript, the same as the AssemblyAI path." timest"+"amps"," Hello"+",") and nests them under segments. These are folded back into words. For scripts written without spaces (CJK), only punctuation is folded. Its full language name ("english") is mapped to the code vialanguage_probabilities.NO_AUDIO), matching the AssemblyAI no-speech handling.get-status, the share page,lib/transcribe, andfinalize-desktop-recordingnow check for either provider instead ofASSEMBLY_API_KEYalone.Testing
__tests__/unit/stt.test.ts: provider selection, the parser against a response captured from a realwhisper-server(large-v3-turbo), OpenAI-shaped responses, CJK, missing word timestamps, and request shape and errors.__tests__/integration/transcribe-workflow.test.ts: full workflow through the STT endpoint (VTT plus encrypted edit transcript, AssemblyAI not called), and silent audio leading toNO_AUDIO.bun run typecheckis clean.whisper-serveron Apple Silicon with Metal). A 6m44s recording produced the VTT,transcription.edit.v3.json, and an AI summary. STT took 13.7s, versus about 140s with faster-whisper on CPU in Docker, with 99.3% word agreement between the two transcripts.The PR appears safe to merge based on this review.
Summary
This PR adds an OpenAI-compatible provider for recorded-video transcription and editable-transcript backfills.
Reviews (4) · Last reviewed commit: "fix(web): span runs of zero-length STT w..." · Reviewed by Greptile