OpenAI Realtime engine

When to use

Low-latency voice-to-voice with natural barge-in.
Prompts, voice, and turn detection configured inline per session via an ephemeral OpenAI client secret.
You don’t want to run a separate agent runtime.

For Cartesia Line-managed agents, see Cartesia. For backend-driven scripted speech, see Legacy text-echo.

Prerequisites

OpenAI API key with Realtime API access. Only the GA version is supported — the beta is not.
Avaturn API key (dashboard).

1. Mint a client secret

On your backend, exchange your OpenAI API key for a short-lived client secret. Avaturn uses this secret to open the Realtime WebSocket on the user’s behalf.

from openai import AsyncOpenAI

openai = AsyncOpenAI(api_key="<OPENAI_API_KEY>")
secret = await openai.realtime.client_secrets.create(
    expires_after={"seconds": 600, "anchor": "created_at"},
    session={"type": "realtime", "model": "gpt-realtime"},
)
client_secret = secret.value  # ek_...

import OpenAI from "openai";

const openai = new OpenAI({ apiKey: "<OPENAI_API_KEY>" });
const secret = await openai.realtime.clientSecrets.create({
  expires_after: { seconds: 600, anchor: "created_at" },
  session: { type: "realtime", model: "gpt-realtime" },
});
const clientSecret = secret.value; // ek_...

curl https://api.openai.com/v1/realtime/client_secrets \
  -H "Authorization: Bearer <OPENAI_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "expires_after": { "seconds": 600, "anchor": "created_at" },
    "session": { "type": "realtime", "model": "gpt-realtime" }
  }'

Mint a fresh secret per user session. The default lifetime is 600 seconds (max 7200). The secret governs token issuance — an existing WebSocket continues working after the secret expires.

2. Create an Avaturn session

import httpx

async with httpx.AsyncClient() as http:
    r = await http.post(
        "https://api.avaturn.live/api/v1/sessions",
        headers={"Authorization": "Bearer <AVATURN_API_KEY>"},
        json={
            "conversation_engine": {
                "type": "openai-realtime",
                "client_secret": client_secret,
            },
        },
    )
    r.raise_for_status()
    session = r.json()  # { "session_id": "...", "token": "..." }

const r = await fetch("https://api.avaturn.live/api/v1/sessions", {
  method: "POST",
  headers: {
    Authorization: "Bearer <AVATURN_API_KEY>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    conversation_engine: {
      type: "openai-realtime",
      client_secret: clientSecret,
    },
  }),
});
const session = await r.json(); // { session_id, token }

curl -X POST https://api.avaturn.live/api/v1/sessions \
  -H "Authorization: Bearer <AVATURN_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "conversation_engine": {
      "type": "openai-realtime",
      "client_secret": "ek_xxxxxxxxxxxx"
    }
  }'

Response:

session_id — backend handle (terminate, telemetry)
token — short-lived credential for the Web SDK

Optional session fields: avatar_id, background, render_model (avatar render preset, not the LLM), user_absent_timeout (default 60s, min 10), max_duration (default 3600s, max 86400). See the API reference.

3. Connect from the frontend

import { AvaturnHead } from "@avaturn-live/web-sdk";

const root = document.querySelector<HTMLDivElement>("#avaturn-video")!;
const avatar = new AvaturnHead(root, {
  sessionToken: session.token,
  audioSource: true, // required — voice-to-voice
});

await avatar.init();

Configuring the agent

The session object you pass to client_secrets.create() is applied to the WebSocket Avaturn opens on the user’s behalf — full control over instructions, voice, VAD, and transcription.

Instructions and voice

secret = await openai.realtime.client_secrets.create(
    expires_after={"seconds": 600, "anchor": "created_at"},
    session={
        "type": "realtime",
        "model": "gpt-realtime",
        "instructions": "You are a helpful assistant. Be concise and friendly.",
        "audio": {
            "output": {"voice": "marin"},
            "input": {
                "transcription": {"model": "whisper-1"},
                "turn_detection": {
                    "type": "server_vad",
                    "threshold": 0.5,
                    "silence_duration_ms": 500,
                },
            },
        },
    },
)

OpenAI currently recommends marin and cedar voices for best quality. Other supported values: alloy, ash, ballad, coral, echo, sage, shimmer, verse.

User transcripts require audio.input.transcription. Without it, OpenAI doesn’t emit transcription events and Avaturn has nothing to forward to the SDK. Avatar response transcripts (assistant side) flow regardless.

Stored prompts

secret = await openai.realtime.client_secrets.create(
    expires_after={"seconds": 600, "anchor": "created_at"},
    session={
        "type": "realtime",
        "model": "gpt-realtime",
        "prompt": {
            "id": "pmpt_abc123",
            "version": "6",
            "variables": {"company_name": "Acme", "tone": "professional"},
        },
    },
)

Full configuration surface (turn detection variants, transcription, audio params): OpenAI session reference.

Engine behavior

Audio. 24 kHz mono PCM in both directions.
Interruptions. OpenAI server VAD (or semantic VAD, if configured). When the user starts speaking, Avaturn discards in-flight avatar audio.
Transcripts. Assistant transcripts (response.output_audio_transcript.done) flow by default. User transcripts (conversation.item.input_audio_transcription.completed) flow only when audio.input.transcription is configured. Both are forwarded to the SDK via ce_events.realtime.*.
Tools. Tool definitions sent in the session config are parsed by OpenAI, but Avaturn doesn’t surface response.function_call_arguments.* events to the Web SDK nor relay function results back. Tool calls won’t execute end-to-end — avoid them at this layer until proper support lands.
GA only. Beta or mixed beta/GA usage causes a session_lifecycle_error with code openai-realtime-version-mismatch. See beta-to-GA migration.

Session lifecycle

A session ends on any of:

Explicit DELETE /api/v1/sessions/{session_id}
user_absent_timeout elapses with the user disconnected (default 60s)
max_duration cap reached (default 3600s, max 86400s)

async with httpx.AsyncClient() as http:
    await http.delete(
        f"https://api.avaturn.live/api/v1/sessions/{session_id}",
        headers={"Authorization": "Bearer <AVATURN_API_KEY>"},
    )

Call avatar.dispose() on the frontend to tear down the local SDK state. The backend session terminates as described above — dispose() does not directly close it. Don’t try to resume a session after it ends; mint a new client secret and create a fresh session.

Get Started

Conversation Engines

Web SDK

REST API

Legacy

When to use

Prerequisites

1. Mint a client secret

2. Create an Avaturn session

3. Connect from the frontend

Configuring the agent

Instructions and voice

Stored prompts

Engine behavior

Session lifecycle

Reference

​When to use

​Prerequisites

​1. Mint a client secret

​2. Create an Avaturn session

​3. Connect from the frontend

​Configuring the agent

​Instructions and voice

​Stored prompts

​Engine behavior

​Session lifecycle

​Reference

When to use

Prerequisites

1. Mint a client secret

2. Create an Avaturn session

3. Connect from the frontend

Configuring the agent

Instructions and voice

Stored prompts

Engine behavior

Session lifecycle

Reference