Browser audio

Let an agent join a browser-based call and talk in it, with raw PCM in both directions.

Let an agent join a browser-based call (Zoom web client, Slack huddle, Google Meet, anything that runs in a tab) and talk in it. Workbench is the pipe: raw PCM out of the call, raw PCM into the call's microphone. Speech recognition, turn-taking and voice come from your agent (a full-duplex realtime voice model works best).

Enable it#

BROWSER_AUDIO_ENABLED=true
BROWSER_AUDIO_MAX_MINUTES=180   # hard cap per call
BROWSER_AUDIO_FIRST_ATTACH_SECONDS=600   # wait for the first reader/uplink

The Docker image ships PulseAudio; nothing runs until the flag is on. With the flag off the tools answer AUDIO_DISABLED. A browser that was already open before you enabled it must restart: browser_audio_start returns BROWSER_RESTART_REQUIRED, and you call it again with restart: true. Open tabs close, logins survive (same profile), and the page reopens in a new tab whose session_id is returned.

Flow#

1. browser_start gives a session_id. Do not open the meeting yet.
2. browser_audio_start { session_id, sample_rate: 24000 } on that blank tab returns { stream_url, clear_url, sample_rate, format, channels, session_id, restarted, headers }. The URLs are absolute, on the server's own origin, and are capability URLs: the secret in the path is the only credential they take (see Capability URLs). Repeating browser_audio_start with the same tab and rate is idempotent and returns the same URLs; a different rate returns AUDIO_BUSY, so stop first, then start again.
3. Open the downlink: GET stream_url with headers (SSE).
4. Open the uplink: POST stream_url with headers, Content-Type: audio/pcm, chunked, and keep it open.
5. browser_navigate to the meeting link. Order matters: meeting pages ask for the microphone as they load, and a request that arrives before the grant is denied and stays denied. If the page was already open, reload it after browser_audio_start. The grant follows the tab across origins.
6. Click through the meeting's join flow (browser_click, browser_screenshot); choose "computer audio" when asked.
7. Pipe: every SSE audio frame goes to your model's input; every model audio delta is written into the uplink body.
8. When your model is interrupted: POST clear_url, which returns { played_ms, cleared_ms }. audio_end_ms = played_ms - <ms of audio you had sent before this response started> is what a realtime API's truncate call needs.
9. Leave the meeting, then browser_audio_stop { session_id }, which returns { played_ms, duration_ms } for the call.

Wire format#

PCM signed 16-bit little-endian, mono, at sample_rate (16000, 24000 or 48000; default 24000) in both directions.

SSE events:

page_left means the tab landed on a meeting app's post-call page (today: Jitsi's /static/close*.html, shown after any hangup, kick or meeting end). An origin change alone does not end the session, because join flows hop origins.

One reader and one uplink at a time: a second concurrent GET gets 409 stream_busy and a second concurrent POST gets 409 uplink_busy. Once a reader has disconnected, a new GET is accepted, so reconnecting after a network blip is safe. The uplink accepts audio faster than real time and queues up to 120 s; past that it stops reading your body until audio plays. The response starts as soon as the uplink is accepted: 200 with Content-Type: application/json, then a newline every 20 s while the call runs, so a proxy waiting for a response (Cloudflare's 524 after 100 s) or dropping an idle one keeps the call. Closing the body plays out the queue, then the response ends with { "played_ms" }. Leading whitespace is valid JSON, so the whole body still parses as one document. Treat the response ending as the uplink being over, not its arrival. Cancelling the request, even after the body is fully sent, drops the queued audio and frees the slot.

Other HTTP responses: 404 audio_not_found for an unknown, revoked or ended capability (never a redirect), 415 unless Content-Type is audio/pcm.

Capability URLs#

The client that holds the call open is often not the agent that called browser_audio_start, and carries no workbench token. So the audio routes take no OAuth: browser_audio_start mints 128 random bits per audio session and puts them in the path (/api/browser/audio/<capability>/stream and /clear). Any Authorization header is ignored.

• The capability lives exactly as long as the audio session: browser_audio_stop, the ended event, closing the tab, a browser restart, max_duration, and page_left, no reader or uplink within BROWSER_AUDIO_FIRST_ATTACH_SECONDS of the start (default 600, room for a human to approve the call), and 60 s with neither connected after that (ended{idle}) all revoke it. The next browser_audio_start mints a new one.
• Treat the URLs as secrets. The server masks the capability in its request logs and records no trace spans for these paths; mask it in your proxy's access logs too (see Reverse proxy settings).
• headers carries only X-Browser-Session, a routing key, not a credential.
• The URLs are built from SERVER_PUBLIC_URL. A client that pins the workbench origin (refusing redirects and other hosts) needs that origin to match its own configured workbench URL exactly: scheme, host and port.

Errors from the tools#

Routing in a cluster#

Send the returned headers (X-Browser-Session) on all three requests so the mesh routes them to the pod running your browser. If a request lands elsewhere the server pipes it to the right pod itself (it needs INTERNAL_MCP_URL and the X-Browser-Session header, since the capability names no user), at the cost of an extra hop. Operator rules, as for every browser request: hash consistently on X-Browser-Session, hash to pod endpoints rather than a ClusterIP (which would re-round-robin), and keep CLUSTER_ENABLED off, since no proxy can route inside a worker pool. Details in Docker: multiple replicas and browser session pod affinity.

Reverse proxy settings#

The uplink is a long request body and the downlink a long response, so a proxy tuned for short requests cuts calls off. In front of the audio endpoints (/api/browser/audio/*) set:

client_max_body_size 0;        # default 1m cuts the uplink after ~22 s at 24 kHz
proxy_request_buffering off;   # stream the body through
proxy_buffering off;           # SSE downlink
proxy_http_version 1.1;
proxy_read_timeout 7200s;      # at least the longest call
proxy_send_timeout 7200s;
access_log off;                # the path is a credential (or log without $request_uri)

ingress-nginx equivalents: nginx.ingress.kubernetes.io/proxy-body-size: "0", proxy-request-buffering: "off", proxy-buffering: "off", proxy-read-timeout: "7200", proxy-send-timeout: "7200". Also see Docker: reverse proxy.

Behind Cloudflare's proxy, the uplink is one request body for the whole call, and Cloudflare caps request bodies by plan (100 MB on Free and Pro). At 24 kHz that is about 35 minutes of agent audio. For longer calls, serve the audio routes from a DNS-only hostname, and set SERVER_PUBLIC_URL to it.

Isolation#

Each user gets their own PulseAudio daemon, so a page in your browser can only enumerate your own devices, never another user's call audio or microphone. The microphone permission is granted to one origin at a time and follows the tab's main-frame navigation. See the field note.

Limits#

• One call per user at a time (AUDIO_BUSY names the tab that has it).
• Audio from your other tabs is mixed into the stream.
• A pod restart drops the connection with no ended event, and the old tab is gone with the browser. Call browser_start (or browser_tabs) for a new session_id, navigate or rejoin the call, then call browser_audio_start again. ended{browser_exit} means chromium itself exited.
• Tell the people in the call they are talking to an agent. Recording and consent rules where you operate are yours to follow.