Avatar API reference

The avatar surface is a single WebSocket endpoint. All I/O is JSON; no binary frames; all audio is base64-encoded inside JSON payloads.

1

wss://scaigrid.scailabs.ai/v1/public/scaibot/avatar/ws-interactive?token=<embed token>

Connection#

• Transport: WSS (TLS 1.2+ required; plain ws:// is rejected).
• Auth: per-bot embed token in the ?token= query. Issued from the Embed & Deploy admin page.
• Origin gate: the browser's Origin header must match an entry in the bot's avatar_allowed_origins list. CLI clients without an Origin header pass through.
• Close codes:
  • 4401 INVALID_ACCESS_TOKEN — token missing, revoked, or invalid.
  • 4403 INVALID_ORIGIN — origin not on the bot's allow-list.
  • 4401 AVATAR_NOT_FOUND — token resolved, but the bot doesn't have avatar_enabled=true.

Message envelope#

Every WebSocket frame is JSON. Outer shape:

Client → Server (requests):

1
2
3
4
5
6
7
8
9

{
  "type": "request",
  "request_id": "<UUIDv4>",
  "payload": {
    "action": "...",
    "params": { ... },
    "response_config": { ... }   // optional, see below
  }
}

Server → Client (responses, system, events, errors):

1
2
3
4
5
6

{
  "type": "response|system|error|event",
  "request_id": "<UUIDv4>",     // matches the request; omitted on protocol-level errors
  "timestamp": "2026-06-23T12:34:56.789Z",
  "payload": { ... }
}

Multiple requests can be in flight simultaneously — match server frames to your request via request_id. Server frames may stream out of request order.

Actions#

start_session#

Create a new session. Required before any chat action.

Request params:

Response data:

1
2
3
4
5
6
7

{
  "session_id": "<UUIDv4>",
  "avatar_id": "<bot slug>",
  "avatar_name": "<bot display name>",
  "user_id": "<user id or null>",
  "created_at": "2026-06-23T12:34:56.789Z"
}

resume_session#

Re-attach to an existing session — used when the WebSocket reconnects.

Request params:

text_to_speech#

Make the avatar say a specific piece of text. No LLM involvement — the text passes straight to TTS + lipsync.

Request params:

Response chunks carry data.text, data.voice (base64 PCM block), and data.blendshapes per response_config (below). chunk_info.is_final=true marks the last chunk.

send_text#

Pass user text through the bot's LLM with the bot's persona + the appended avatar-mode rules, synthesize the reply as audio + lipsync, and emit it as a response chunk.

Request params:

The server persists both the user's message and the assistant's reply against the session so subsequent calls to get_history include them. The reply's text + voice + blendshapes ship in a single response chunk with chunk_info.is_final=true. The same response_config block as text_to_speech controls what gets included.

send_voice#

Send a base64-encoded microphone clip. The server transcribes it via ScaiInfer's STT, then delegates to send_text for the LLM + TTS + lipsync round-trip.

Request params:

When the client sets response_config.include_transcription=true, the transcript appears on the response chunk as data.transcription.

avatar_update event — how clients learn the 3D config#

Right after start_session (and resume_session) succeeds, the server emits an avatar_update event carrying the bot's current visual configuration. The client uses this to pick the 3D model URL, the lipsync engine variant, and the idle animation — without an extra HTTP round-trip.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
  "type": "event",
  "timestamp": "2026-06-23T12:34:56.789Z",
  "payload": {
    "event_type": "avatar_update",
    "data": {
      "avatar_id": "support-bot",
      "avatar_name": "Poolnoodle",
      "model_url": "https://cdn.customer.example.com/poolnoodle.glb",
      "blendshape_profile": "audio.lipsync.scailipsync",
      "idle_animation": "idle_loop_01"
    }
  }
}

The event is emitted on every start_session and resume_session success — even when none of the bot's avatar fields are set. Clients use field presence (model_url etc.) to decide what to load, not whether to handle the event.

Compatibility: avatar_update is in the standalone protocol's event_type enum, and EventPayload.data is dict[str, Any], so this carries zero compatibility risk. Clients that don't handle avatar_update drop the event silently per protocol.

get_history#

Page through the session's message history.

Request params:

stop_command#

Cancel an in-flight request mid-stream.

Request params:

The cancelled request emits a single event frame with event_type: "stopped" before its asyncio task settles.

response_config#

Optional block on chat actions that controls what the server emits. Server-side defaults match the standalone ScaiVatars service.

Blendshape format#

Each blendshape frame is a list:

1

["HH:MM:SS.mmm", c0, c1, c2, ..., c67]

• Index 0 is a timecode string (HH:MM:SS.mmm, millisecond precision).
• Indices 1–68 are ARKit-compatible blendshape coefficients (68 floats in [-1.0, 1.0]).
• At the default 60 fps, frames are 16.67 ms apart.

Drive your 3D model's morph target weights from indices 1–68, synchronized to the audio's playback time.

Avatar mode rules#

When the LLM lands (v1.1), every request will carry an appended system-prompt block reminding the model it's a 3D avatar. The block tells the LLM:

• The user SEES its face and HEARS its words; expressions and tone are rendered live.
• Never use Markdown — asterisks get read aloud by the TTS.
• Never reference visual artifacts ("as shown below", "see the diagram") that don't translate to speech.
• It cannot see the user; don't claim to read their expression.
• It can convey emotion through tone; write words you'd want spoken with the right feeling.
• The user can interrupt at any time; get to the answer in the first sentence.

The full block is visible read-only on the Avatar Settings admin page so operators can see exactly what gets appended to their persona.

Error codes#

Admin endpoints#

For programmatic management (the UI uses these too):

1
2
3
4
5

GET  /v1/modules/scaibot/bots/{bot_id}/scaivatars
PUT  /v1/modules/scaibot/bots/{bot_id}/scaivatars
GET  /v1/modules/scaibot/scaivatars/sessions
GET  /v1/modules/scaibot/scaivatars/sessions/{id}
DELETE /v1/modules/scaibot/scaivatars/sessions/{id}

The PUT accepts any subset of: avatar_enabled, avatar_model_url, avatar_blendshape_profile, avatar_idle_animation, avatar_default_response_config, avatar_allowed_origins, avatar_system_prompt.

Embed tokens for the avatar surface come from the existing ScaiBot embed-token endpoints — one token type, three surfaces (chat widget, voice widget, avatar WebSocket):

1
2
3

GET    /v1/modules/scaibot/bots/{bot_id}/embed-tokens
POST   /v1/modules/scaibot/bots/{bot_id}/embed-token
DELETE /v1/modules/scaibot/bots/{bot_id}/embed-token