[yuyixg]

Problem

Outbound Twilio calls synthesize speech with OpenAI /v1/audio/speech before dialing and on every turn. In production some of those requests never return. GenerateSpeechAsync used the SDK defaults (100s per attempt, 3 retries), so one stuck request blocked the call for about 7 minutes (Retry failed after 4 tries), and when it was the opening line the call was never placed.

Changes

1. OpenAI text-to-speech timeout (bf039def7)

• AudioSynthesisProvider waits at most 30s per attempt and keeps 3 retries.
• Both are configurable per model: Audio.Synthesis.TimeoutSeconds / MaxRetries in LlmModelSetting.
• ProviderHelper.GetClient takes the timeout and retry count as optional arguments; other callers keep the SDK defaults.

2. Twilio <Say> mode (Twilio:SayGeneratedSpeech, off by default)

• Generated speech is saved as a text file and spoken with <Say voice="Twilio:SayVoice"> instead of a pre-generated mp3: opening line, replies, hang-up, transfer, voicemail, tool-call indications.
• Prerecorded prompts (twilio/{name}.mp3) can be spoken from Twilio:PromptTexts so the whole call uses one voice; prompts without text keep playing.
• TwilioService.SaveSpeechAsync / AppendSpeech are now the single places that write and play speech. With the switch off, behaviour is unchanged.

3. Twilio ConversationRelay (Twilio:ConversationRelay, off by default; AgentIds limits it to listed agents)

• Outbound calls for enabled agents are answered with <Connect><ConversationRelay>: Twilio transcribes and speaks, and exchanges text with the new TwilioConversationRelayMiddleware at /twilio/relay/{agentId}/{conversationId}.
• ConversationRelaySession handles one call:
  • runs one agent turn at a time and merges words spoken meanwhile into the next turn;
  • handles DTMF (#-terminated) and speaks tool-call indications;
  • asks the caller to repeat after silence and hangs up after MaxIdlePrompts, or when the agent doesn't answer within AiResponseTimeoutSeconds;
  • ends the session with handoffData hangup or transfer.
• twilio/relay/action dials the CSR number on transfer, hangs up otherwise, and reconnects a failed session up to MaxReconnects times.
• The WebSocket handshake is checked against X-Twilio-Signature (ValidateSignature, on by default).
• The caller-message handling used by TwilioMessageQueueService moved to TwilioConversationProcessor so both flows share it.
• Hosts must register the middleware next to TwilioStreamMiddleware: app.UseMiddleware<TwilioConversationRelayMiddleware>();. Before enabling it, the account must accept Twilio's AI/ML features addendum.

Testing

• OpenAI timeout: against a local server that never answers, 4 attempts about 30s apart fail after 125.7s (was ~400s); a hang followed by a good response recovers on the retry.
• <Say> mode: 24 TwiML checks through TwilioService pass: each speech type, escaping, prompt replacement, config binding, unchanged behaviour with the switch off.
• ConversationRelay: 28 checks pass. They drive a real WebSocket with ASP.NET TestServer, playing Twilio's side against a fake agent: turns, partial prompts, merging, DTMF, indications, silence re-prompt and hang-up, agent timeout and exception, signature accept/reject, TwiML attributes, action callback (transfer, hang-up, reconnect, reconnect limit), per-agent switch.
• Downstream: a downstream app builds against this branch, and its unit and integration suites show no new failures.
• Not tested: no real phone call yet. Staging verification is pending.

🤖 Generated with Claude Code

[qodo-code-review]

Code Review by Qodo

Sorry, something went wrong

We weren't able to complete the code review on our side. Please try again manually by commenting /agentic_review on this PR.

[qodo-code-review]

PR Summary by Qodo

Add Twilio Say and ConversationRelay with bounded OpenAI TTS requests

✨ Enhancement 🐞 Bug fix 🕐 40+ Minutes

AI Description

• Bound OpenAI speech-synthesis attempts to prevent stalled requests from delaying outbound calls.
• Add opt-in Twilio Say speech and ConversationRelay for eligible outbound agents.
• Share caller-message processing across Twilio flows while retaining existing call behavior by
 default.

Diagram

sequenceDiagram
    participant Twilio
    participant Outbound as Outbound controller
    participant Service as Twilio service
    participant Relay as Relay middleware
    participant Session as Relay session
    participant Processor as Conversation processor
    participant Action as Action controller
    Twilio->>Outbound: Answer outbound call
    Outbound->>Service: Build opt-in relay TwiML
    Service-->>Twilio: Connect ConversationRelay
    Twilio->>Relay: Open signed WebSocket
    Relay->>Session: Start call session
    Twilio->>Session: Send caller text or DTMF
    Session->>Processor: Process serialized agent turn
    Processor-->>Session: Reply and indications
    Session-->>Twilio: Speak text or end session
    Twilio->>Action: Post session outcome
    Action-->>Twilio: Transfer, reconnect, or hang up

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Extend the existing Gather and Play flow only

• ➕ Reuses established webhook behavior and avoids a new WebSocket lifecycle.
• ➖ Does not provide ConversationRelay's text exchange, Twilio-managed transcription, and speech within one session.

Recommendation: Keep the opt-in dual-flow approach: it adds ConversationRelay without forcing existing agents to migrate, and centralizing speech handling preserves the MP3 fallback. Reviewers should verify host middleware registration and the signature-validation configuration before enabling relay; neither is changed by this diff.

Files changed (20) +1144 / -147

Enhancement (14) +914 / -62

ProviderHelper.csAllow optional OpenAI client timeout and retry overrides +17/-3

Allow optional OpenAI client timeout and retry overrides

• Accepts optional network timeout and retry count while leaving other callers on SDK defaults.

src/Plugins/BotSharp.Plugin.OpenAI/Providers/ProviderHelper.cs

TwilioConversationRelayController.csHandle ConversationRelay end callbacks +78/-0

Handle ConversationRelay end callbacks

• Adds a validated action endpoint that transfers handoffs, reconnects eligible failed sessions, or hangs up.

src/Plugins/BotSharp.Plugin.Twilio/Controllers/TwilioConversationRelayController.cs

TwilioInboundController.csPlay or say inbound voicemail through shared speech handling +2/-2

Play or say inbound voicemail through shared speech handling

• Resolves the saved voicemail format and delegates its TwiML playback to TwilioService.

src/Plugins/BotSharp.Plugin.Twilio/Controllers/TwilioInboundController.cs

TwilioOutboundController.csRoute eligible outbound calls to ConversationRelay +13/-2

Route eligible outbound calls to ConversationRelay

• Uses the stored opening reply as a relay greeting for enabled agents. Voicemail playback also uses the shared speech handler.

src/Plugins/BotSharp.Plugin.Twilio/Controllers/TwilioOutboundController.cs

TwilioReconnectController.csUse shared speech handling on call reconnect +1/-2

Use shared speech handling on call reconnect

• Replaces direct audio playback so a saved opening message can be spoken from text or played as audio.

src/Plugins/BotSharp.Plugin.Twilio/Controllers/TwilioReconnectController.cs

ConversationRelayActionRequest.csModel ConversationRelay action callback fields +33/-0

Model ConversationRelay action callback fields

• Captures session status, handoff data, errors, and the reconnect count from Twilio's callback.

src/Plugins/BotSharp.Plugin.Twilio/Models/Relay/ConversationRelayActionRequest.cs

ConversationRelayParameters.csDefine relay greeting and handoff constants +18/-0

Define relay greeting and handoff constants

• Names the greeting-length setup parameter and the hang-up and transfer handoff actions.

src/Plugins/BotSharp.Plugin.Twilio/Models/Relay/ConversationRelayParameters.cs

HangupPhoneCallFn.csSave closing speech in the selected Twilio format +3/-6

Save closing speech in the selected Twilio format

• Delegates optional ending-message generation to TwilioService instead of always synthesizing an MP3.

src/Plugins/BotSharp.Plugin.Twilio/OutboundPhoneCallHandler/Functions/HangupPhoneCallFn.cs

LeaveVoicemailFn.csSave voicemail in the selected Twilio format +3/-6

Save voicemail in the selected Twilio format

• Delegates voicemail speech storage to TwilioService, allowing text-backed Say playback.

src/Plugins/BotSharp.Plugin.Twilio/OutboundPhoneCallHandler/Functions/LeaveVoicemailFn.cs

OutboundPhoneCallFn.csAvoid pre-dial synthesis for relay calls +12/-12

Avoid pre-dial synthesis for relay calls

• Loads the agent profile before choosing the call path. Relay calls use the stored opening reply as a greeting; other calls save speech through TwilioService.

src/Plugins/BotSharp.Plugin.Twilio/OutboundPhoneCallHandler/Functions/OutboundPhoneCallFn.cs

TransferPhoneCallFn.csSave transfer speech in the selected Twilio format +3/-5

Save transfer speech in the selected Twilio format

• Delegates transition-message storage to TwilioService instead of always creating an MP3.

src/Plugins/BotSharp.Plugin.Twilio/OutboundPhoneCallHandler/Functions/TransferPhoneCallFn.cs

ConversationRelaySession.csManage one ConversationRelay WebSocket call +471/-0

Manage one ConversationRelay WebSocket call

• Processes speech and DTMF as serialized agent turns, speaks replies and indications, and manages idle prompts, response timeouts, and end handoffs.

src/Plugins/BotSharp.Plugin.Twilio/Services/ConversationRelaySession.cs

TwilioService.csCentralize Twilio speech and construct relay TwiML +175/-24

Centralize Twilio speech and construct relay TwiML

• Saves generated speech as text or MP3, emits Say or Play for saved speech and configured prompts, and builds ConversationRelay connections and URLs.

src/Plugins/BotSharp.Plugin.Twilio/Services/TwilioService.cs

TwilioConversationRelayMiddleware.csAccept signed ConversationRelay WebSockets +85/-0

Accept signed ConversationRelay WebSockets

• Adds the relay WebSocket route, optional Twilio signature validation, and per-message scopes for shared conversation processing.

src/Plugins/BotSharp.Plugin.Twilio/TwilioConversationRelayMiddleware.cs

Bug fix (1) +12 / -1

AudioSynthesisProvider.csBound OpenAI speech-synthesis attempts +12/-1

Bound OpenAI speech-synthesis attempts

• Reads per-model synthesis settings and defaults to a 30-second timeout per attempt with three retries.

src/Plugins/BotSharp.Plugin.OpenAI/Providers/Audio/AudioSynthesisProvider.cs

Refactor (2) +113 / -84

TwilioConversationProcessor.csShare caller-message processing between Twilio flows +103/-0

Share caller-message processing between Twilio flows

• Extracts conversation setup, postback handling, agent messaging, and tool-call indications for use by both webhook and relay sessions.

src/Plugins/BotSharp.Plugin.Twilio/Services/TwilioConversationProcessor.cs

TwilioMessageQueueService.csUse the shared Twilio conversation processor +10/-84

Use the shared Twilio conversation processor

• Replaces duplicated agent-turn handling with the shared processor and saves reply speech through TwilioService.

src/Plugins/BotSharp.Plugin.Twilio/Services/TwilioMessageQueueService.cs

Other (3) +105 / -0

LlmModelSetting.csAdd per-model audio synthesis settings +17/-0

Add per-model audio synthesis settings

• Adds optional timeout and retry settings under Audio.Synthesis for text-to-speech models.

src/Infrastructure/BotSharp.Abstraction/MLTasks/Settings/LlmModelSetting.cs

ConversationRelaySetting.csConfigure opt-in ConversationRelay behavior +70/-0

Configure opt-in ConversationRelay behavior

• Adds agent allowlisting, voice and transcription options, idle and response limits, reconnection limits, and signature-validation control.

src/Plugins/BotSharp.Plugin.Twilio/Settings/ConversationRelaySetting.cs

TwilioSetting.csExpose Twilio Say and relay settings +18/-0

Expose Twilio Say and relay settings

• Adds the Say switch, voice and prerecorded-prompt text mapping, and nested ConversationRelay settings.

src/Plugins/BotSharp.Plugin.Twilio/Settings/TwilioSetting.cs