Voice mode

1
2
3
4
5
6

{
  "title": "Voice mode",
  "audience": "developer",
  "summary": "How voice calls with AI work — architecture, ScaiVoice integration, LiveKit, on-the-record semantics.",
  "sort_order": 11
}

Voice mode#

Voice mode lets users have live spoken conversations with AI participants. It integrates LiveKit (audio transport), ScaiVoice (STT→cognition→TTS orchestration), and ScaiWave's existing AI bridge (tools, context, persona).

Architecture#

1
2
3
4
5

Browser ←→ LiveKit SFU ←→ Voice Bot Worker ←→ ScaiVoice ←→ ScaiWave API
                                 │
                                 ├── STT: ScaiEcho (streaming WS)
                                 ├── LLM: via /v1/voice-agent/turn callback
                                 └── TTS: ScaiSpeak (via ScaiVoice, 48kHz PCM)

• LiveKit owns the audio transport (WebRTC, Opus encoding, echo cancellation, multi-party mixing).
• Voice Bot Worker (make voice-bot) is a separate long-running process that joins LiveKit rooms as the AI participant.
• ScaiVoice is ScaiGrid's STT→TTS orchestration framework. We use it in cognition_mode='delegated' so our AI bridge keeps running the LLM + tools.
• ScaiWave API hosts the delegated-cognition callback at /v1/voice-agent/turn. It runs the same ContextAssembler + plugin registry as text chat.

Call lifecycle#

1. User clicks the call button in an AI-session room.
2. API creates the call as ACTIVE (no ringing for AI rooms), publishes swp.call.voice.started on NATS.
3. Voice bot worker picks up the signal, joins LiveKit with a unique identity, opens a ScaiVoice session.
4. Audio flows: user speaks → LiveKit → bot captures → VAD detects end-of-speech → streaming STT produces transcript → ScaiVoice WS text frame → callback → AI bridge → streamed reply → TTS → 48kHz PCM frames → bot publishes to LiveKit → user hears the AI.
5. On call end: API publishes swp.call.voice.ended, bot tears down all connections.

Turn detection#

Two layers:

1. Silero VAD (via livekit-agents plugin) detects speech boundaries from audio energy.
2. Linguistic check: after STT, a regex checks for terminal punctuation. Complete sentences submit immediately; incomplete sentences wait 1.5s for continuation.

On the record / off the record#

Per-room Redis flag at sw:{tenant_id}:voice_room:{room_id}:on_record. Flipped by:

• User toggle in the call controls (eye icon)
• LLM marker <<recording:on|off>> in its reply

On-record turns persist as swp.voice_transcript / swp.voice_reply events in the room timeline. Off-record turns are ephemeral.

Barge-in#

When VAD detects the user speaking while the AI is in speaking or thinking state, the bot sends {"type":"vad","speaking":true} to ScaiVoice, which cancels TTS immediately.

Several layers ensure clean interruptions:

• Generation counter: each LLM turn increments a generation counter. TTS audio frames carry the generation they belong to; the bot's playback filter drops frames from any stale generation, so leftover audio from a cancelled turn never reaches the user.
• Silence flush: on barge-in the bot pushes a short silence frame to LiveKit to clear the jitter buffer, preventing residual audio artifacts.
• Client-side ducking: the frontend ducks (lowers volume of) the AI audio track as soon as the user's microphone crosses the speech threshold, providing an instant perceptual interruption even before the server-side cancel round-trips.
• LLM interruption context: when a barge-in truncates the AI's reply, the interrupted text and a [user interrupted] marker are written to a Redis key scoped to the session. The next cognition turn prepends this context so the LLM knows what the user cut short and can adapt its response accordingly.

TTS queue decoupling#

TTS audio frames from ScaiVoice arrive over a WebSocket. To keep the WS connection healthy under load, incoming frames are pushed into an in-memory async queue rather than being written to LiveKit inline on the WS read loop. A separate playback task drains the queue and pushes frames to the LiveKit AudioSource at the correct cadence. This decoupling prevents slow LiveKit publishes from back-pressuring the ScaiVoice WS and triggering timeouts or missed heartbeats.

Voice control#

Users can set per-user TTS style and speed preferences via PUT /v1/users/me/voice-preference (see Voice API).

• Style (voice_style): free-text instructions forwarded to ScaiVoice at session create (e.g. "warm and professional, calm pace"). This influences prosody, tone, and emphasis.
• Speed (voice_speed): a float from 0.5 to 2.0 controlling TTS playback speed. Defaults to 1.0 when unset.

Both values are stored in the user's notification_prefs JSON and resolved at session creation time alongside the voice-id resolution chain (user pref → tenant default → first available).

Voice-mode system prompt#

When a turn is processed in voice mode, a voice-specific system message is prepended. It instructs the LLM to:

• Keep responses to 2–3 sentences to match the cadence of spoken conversation.
• Handle interruptions gracefully — if a [user interrupted] marker is present, acknowledge it briefly rather than repeating the truncated content.
• Avoid markdown, code blocks, and other visual formatting that does not translate to speech.
• Announce tool calls before executing them so the user hears activity rather than silence.

This prompt is layered on top of the room's existing persona and context; it does not replace them.

Tool support#

Voice mode has full tool calling — same tools as text chat (web search, notes, todos, files, drive, plugins). Up to 5 tool-call rounds per turn. The system prompt instructs the AI to speak briefly before calling a tool so the user hears activity.

Audio format#

• User → bot: 16 kHz mono PCM (LiveKit resamples from 48kHz Opus internally; bot streams to ScaiEcho STT WS)
• Bot → user: 48 kHz mono PCM (ScaiVoice TTS output, pushed to LiveKit AudioSource)

Configuration#

Deployment#

The voice bot is a separate process:

1
2

make voice-bot                               # production
SCAIWAVE_VOICE_BOT_LOG_LEVEL=DEBUG make voice-bot  # debug

Requires: NATS, LiveKit, ScaiGrid reachable. Subscribes to swp.call.voice.> on the SW_CALL_EVENTS JetStream stream.

Related#

• Tutorial: Talk to your AI
• API: Voice endpoints
• Concepts: Calls — LiveKit infrastructure