---
title: Voice mode
path: concepts/voice-mode
status: published
---

```json frontmatter
{
  "title": "Voice mode",
  "audience": "developer",
  "summary": "How voice calls with AI work — architecture, ScaiVoice integration, LiveKit, on-the-record semantics.",
  "sort_order": 11
}
```

# Voice mode

Voice mode lets users have live spoken conversations with AI
participants. It integrates LiveKit (audio transport), ScaiVoice
(STT→cognition→TTS orchestration), and ScaiWave's existing AI
bridge (tools, context, persona).

## Architecture

```text
Browser ←→ LiveKit SFU ←→ Voice Bot Worker ←→ ScaiVoice ←→ ScaiWave API
                                 │
                                 ├── STT: ScaiEcho (streaming WS)
                                 ├── LLM: via /v1/voice-agent/turn callback
                                 └── TTS: ScaiSpeak (via ScaiVoice, 48kHz PCM)
```

- **LiveKit** owns the audio transport (WebRTC, Opus encoding,
  echo cancellation, multi-party mixing).
- **Voice Bot Worker** (`make voice-bot`) is a separate long-running
  process that joins LiveKit rooms as the AI participant.
- **ScaiVoice** is ScaiGrid's STT→TTS orchestration framework. We
  use it in `cognition_mode='delegated'` so our AI bridge keeps
  running the LLM + tools.
- **ScaiWave API** hosts the delegated-cognition callback at
  `/v1/voice-agent/turn`. It runs the same ContextAssembler +
  plugin registry as text chat.

## Call lifecycle

1. User clicks the call button in an AI-session room.
2. API creates the call as ACTIVE (no ringing for AI rooms),
   publishes `swp.call.voice.started` on NATS.
3. Voice bot worker picks up the signal, joins LiveKit with a
   unique identity, opens a ScaiVoice session.
4. Audio flows: user speaks → LiveKit → bot captures → VAD detects
   end-of-speech → streaming STT produces transcript → ScaiVoice
   WS text frame → callback → AI bridge → streamed reply → TTS →
   48kHz PCM frames → bot publishes to LiveKit → user hears the AI.
5. On call end: API publishes `swp.call.voice.ended`, bot tears
   down all connections.

## Turn detection

Two layers:
1. **Silero VAD** (via livekit-agents plugin) detects speech
   boundaries from audio energy.
2. **Linguistic check**: after STT, a regex checks for terminal
   punctuation. Complete sentences submit immediately; incomplete
   sentences wait 1.5s for continuation.

## On the record / off the record

Per-room Redis flag at `sw:{tenant_id}:voice_room:{room_id}:on_record`.
Flipped by:
- User toggle in the call controls (eye icon)
- LLM marker `<<recording:on|off>>` in its reply

On-record turns persist as `swp.voice_transcript` / `swp.voice_reply`
events in the room timeline. Off-record turns are ephemeral.

## Barge-in

When VAD detects the user speaking while the AI is in `speaking`
or `thinking` state, the bot sends `{"type":"vad","speaking":true}`
to ScaiVoice, which cancels TTS immediately.

Several layers ensure clean interruptions:

- **Generation counter**: each LLM turn increments a generation
  counter. TTS audio frames carry the generation they belong to;
  the bot's playback filter drops frames from any stale generation,
  so leftover audio from a cancelled turn never reaches the user.
- **Silence flush**: on barge-in the bot pushes a short silence
  frame to LiveKit to clear the jitter buffer, preventing residual
  audio artifacts.
- **Client-side ducking**: the frontend ducks (lowers volume of)
  the AI audio track as soon as the user's microphone crosses the
  speech threshold, providing an instant perceptual interruption
  even before the server-side cancel round-trips.
- **LLM interruption context**: when a barge-in truncates the AI's
  reply, the interrupted text and a `[user interrupted]` marker are
  written to a Redis key scoped to the session. The next cognition
  turn prepends this context so the LLM knows what the user cut
  short and can adapt its response accordingly.

## TTS queue decoupling

TTS audio frames from ScaiVoice arrive over a WebSocket. To keep
the WS connection healthy under load, incoming frames are pushed
into an in-memory async queue rather than being written to LiveKit
inline on the WS read loop. A separate playback task drains the
queue and pushes frames to the LiveKit `AudioSource` at the
correct cadence. This decoupling prevents slow LiveKit publishes
from back-pressuring the ScaiVoice WS and triggering timeouts or
missed heartbeats.

## Voice control

Users can set per-user TTS style and speed preferences via
`PUT /v1/users/me/voice-preference` (see
[Voice API](/docs/scaiwave/reference/api/voice#user-voice-preference)).

- **Style** (`voice_style`): free-text instructions forwarded to
  ScaiVoice at session create (e.g. "warm and professional, calm
  pace"). This influences prosody, tone, and emphasis.
- **Speed** (`voice_speed`): a float from 0.5 to 2.0 controlling
  TTS playback speed. Defaults to 1.0 when unset.

Both values are stored in the user's `notification_prefs` JSON and
resolved at session creation time alongside the voice-id resolution
chain (user pref → tenant default → first available).

## Voice-mode system prompt

When a turn is processed in voice mode, a voice-specific system
message is prepended. It instructs the LLM to:

- Keep responses to **2–3 sentences** to match the cadence of
  spoken conversation.
- Handle interruptions gracefully — if a `[user interrupted]`
  marker is present, acknowledge it briefly rather than repeating
  the truncated content.
- Avoid markdown, code blocks, and other visual formatting that
  does not translate to speech.
- Announce tool calls before executing them so the user hears
  activity rather than silence.

This prompt is layered on top of the room's existing persona and
context; it does not replace them.

## Tool support

Voice mode has full tool calling — same tools as text chat (web
search, notes, todos, files, drive, plugins). Up to 5 tool-call
rounds per turn. The system prompt instructs the AI to speak
briefly before calling a tool so the user hears activity.

## Audio format

- **User → bot**: 16 kHz mono PCM (LiveKit resamples from 48kHz
  Opus internally; bot streams to ScaiEcho STT WS)
- **Bot → user**: 48 kHz mono PCM (ScaiVoice TTS output, pushed
  to LiveKit AudioSource)

## Configuration

| Env var | Purpose |
|---|---|
| `SCAIWAVE_VOICE_SESSION_SIGNING_SECRET` | HMAC key for callback tokens (required) |
| `SCAIWAVE_VOICE_SESSION_TOKEN_TTL` | Token lifetime (default 3600s) |

## Deployment

The voice bot is a separate process:

```bash
make voice-bot                               # production
SCAIWAVE_VOICE_BOT_LOG_LEVEL=DEBUG make voice-bot  # debug
```

Requires: NATS, LiveKit, ScaiGrid reachable. Subscribes to
`swp.call.voice.>` on the `SW_CALL_EVENTS` JetStream stream.

## Related

- Tutorial: [Talk to your AI](/docs/scaiwave/tutorials/beginner/talk-to-your-ai)
- API: [Voice endpoints](/docs/scaiwave/reference/api/voice)
- Concepts: [Calls](/docs/scaiwave/concepts/calls) — LiveKit infrastructure