Realtime

Implementation target. This guide specifies the intended product surface. The current frontend uses local samples; it does not call an inference service or collect funds. An endpoint is enabled only after its adapter, ownership, usage, and payment behavior are qualified.

Planned API origin. Examples use https://noemad.dev/v1, the intended product API path. The API is not connected in this frontend preview. MODEL_ID_FROM_DISCOVERY and NOEMAD_API_KEY are placeholders, not published credentials or model IDs. A product API key identifies an account; a separate payment-aware transport must handle spending authorization.

One session, several turns

A realtime voice connection carries microphone input, synthesized audio, transcripts, and session events over WebRTC. Establish the session once and keep it open for multiple turns; do not submit a new paid call for every utterance.

The UI needs distinct idle, requesting-permission, connecting, active, muted, interrupted, closing, and closed states. Only request microphone access after the user starts a voice action. Always expose a visible stop control.

Session configuration

{
  "type": "realtime",
  "model": "MODEL_ID_FROM_DISCOVERY",
  "language": "en-US",
  "instructions": "Speak naturally, keep answers concise, and state uncertainty.",
  "audio": {
    "output": {
      "voice": "VOICE_FROM_DISCOVERY"
    }
  },
  "delegation": {
    "type": "client",
    "ack_filler": false
  }
}

Archived language presets include en-US, en-GB, id-ID, ms-MY, ar-SA, cs-CZ, da-DK, nl-NL, fil-PH, zh-CN, ja-JP, ko-KR, es-ES, fr-FR, de-DE, it-IT, mk-MK, fa-IR, pl-PL, pt-BR, ro-RO, ru-RU, sv-SE, th-TH, tr-TR, vi-VN, and hi-IN. Validate the selected tag against the actual adapter.

The source voice list includes alloy, arbor, ash, ballad, breeze, cedar, coral, cove, echo, ember, juniper, maple, marin, sage, shimmer, sol, spruce, vale, and verse. This is an archived list, not live voice availability.

Negotiate through an authorized application route

// Browser-side design sketch. The product session/payment layer is required.
const peer = new RTCPeerConnection();
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
for (const track of stream.getTracks()) peer.addTrack(track, stream);
peer.ontrack = event => { remoteAudio.srcObject = event.streams[0]; };
const events = peer.createDataChannel('session-events');
const offer = await peer.createOffer();
await peer.setLocalDescription(offer);

// Proposed payment-aware application transport, not a published SDK method.
const established = await authorizedCall('/v1/realtime/calls', {
  sdp: peer.localDescription.sdp,
  session
});
await peer.setRemoteDescription({ type: 'answer', sdp: established.sdp });
// Persist the product session reference for explicit cleanup.

The server verifies product identity and the bounded payment authorization before establishing a costed session. Provider keys remain on the server. If direct browser media access needs a token, issue only a short-lived scoped credential.

The response can include an SDP answer, a private session ID, and a single-session control token. Keep control tokens out of logs, URLs, screenshots, and model prompts. Exact response fields must be frozen with the adapter fixtures.

Conversation loop

1. 1
   Request microphone permission after a deliberate start action.
2. 2
   Review the session’s price unit, spending maximum, duration, and expiry.
3. 3
   Negotiate the same authorized session once.
4. 4
   Play remote audio and display transcript/state events from the data channel.
5. 5
   Keep the connection for subsequent turns and show mute/interrupt state clearly.
6. 6
   After terminal failure or expiry, close and reconcile the session before creating another.

Connect a chat model to a live conversation

The application owns conversation history. For a task that must use a selected chat model, its backend sends recent turns to /v1/chat/completions under its own authorized product operation. After validating the answer, the client can append it as speakable context to the open realtime channel.

// Proposed adapter event format; not a universal vendor SDK format.
events.send(JSON.stringify({
  type: 'session.context.append',
  channel: 'speakable',
  content: [{ type: 'input_text', text: verifiedChatAnswer }]
}));

The archived bridge can emit delegation.created and accept delegation.context.append. A model deciding when to delegate does not guarantee a particular chat request will run. Call the chat operation explicitly when the application requires it, and apply a separate approved spending policy.

These events need adapter-specific compatibility testing. Do not assume they match the event format of every realtime SDK. No complete bridge download is linked until a maintained package exists in this project.

Release the server session

DELETE /v1/realtime/sessions/SESSION_ID
Authorization: Bearer NOEMAD_API_KEY
X-Noemad-Realtime-Control-Token: SESSION_CONTROL_TOKEN

Stop every local microphone track, close the data channel and peer connection, and request server release. A page closing or a local microphone stopping is not proof that the remote session ended.

Release should be idempotent for the same owner and session. Persist enough state to recover cleanup after a dropped connection. Publish maximum duration, idle timeout, release-token policy, and any cancellation charge before enabling voice.

Metering and collection

A realtime quote defines the actual billing unit and enforceable session limits. The archived reference used a rate per successfully established session; a different provider might meter duration or tokens and requires a new specification. Failed negotiation must be classified separately from an established session.

Final usage and collection status remain independent of audio playback. If a session or usage outcome is uncertain, reconcile it. The frontend preview never activates the microphone or negotiates a live call.