caption-local/API.md

Caption Local API v1

Loopback-only, multiple trusted producers. Use the local page or a trusted local client. When a room registry is configured (CAPTION_ROOMS_DB plus CAPTION_RELAY_MASTER), API routes require the signed caption_room session cookie issued after opening a room admin link. Rooms are self-service through POST /api/rooms; each room derives its own private source/output relay rooms and capability tokens from the shared master, served by GET /relay-config and GET /api/rooms/<id>/config. See deployment profiles for room and logging setup. Hosted capture can opt into exact origins with CAPTION_ALLOWED_ORIGINS. Setup and browser limitations. Public assets (/, /capture-local.html, /admin.html, /health), room creation, session redemption and /api/public/rooms/<id> need no room cookie. The operator console API (/api/admin/rooms) uses CAPTION_ADMIN_KEY as a Bearer token.

OpenAI-style WAV API

The local server implements a limited subset of the OpenAI transcription request contract and English translation contract:

• GET /v1/models: returns the one loaded local model.
• POST /v1/audio/transcriptions: multipart fields file, model, optional language and response_format (json or text). Default language detection is automatic. JSON responses contain text.
• POST /v1/audio/translations: same fields except language; detects speech language and translates to English.

Use the loaded local model name, or whisper-1 as an explicit compatibility alias for that local model. This does not load OpenAI's hosted Whisper model or switch models per request. Only uncompressed PCM WAV, mono/stereo, 8–48 kHz, 0.01–12 seconds, within a 5 MiB multipart envelope is supported. Convert/chunk other audio before sending it. Uploads have a ten-second deadline and bounded admission.

The adapter uses the native scheduler and retry cache. Optional X-Stream-ID and X-Request-ID retain native retry identity; keep them stable across retries and use distinct stream IDs per producer. Without a stream ID, calls use temporary independent sessions that close afterward. If the client disconnects during inference, the temporary session closes when its worker finishes; the running worker keeps its admission slot until then. Automatic SDK retries without both IDs do not promise idempotency. Native detail errors and HTTP status codes are retained.

Nonzero temperature, prompts, streamed responses, timestamps, diarization, SRT/VTT, compressed files and other provider fields fail explicitly. There is no OpenAI Realtime/WebSocket, Chat/Responses, Google Speech or Gemini endpoint emulation. These providers have different protocols and capabilities. This is speech-to-text and speech-to-English, not text-to-speech.

Example with the official OpenAI Python SDK, targeting only the local server:

from openai import OpenAI
import os
client = OpenAI(base_url="http://127.0.0.1:8765/v1",
                api_key="local-unused")
with open("speech.wav", "rb") as audio:
    result = client.audio.transcriptions.create(
        model="small", file=audio, language="fr")
print(result.text)

No OpenAI account or cloud API key is used. Test with scripts/smoke_compat.py; the dev requirements pin the SDK version. The native PCM endpoint remains the interface for rolling live capture and bilingual output.

Health

GET /health is available only after model load and actual decoder warmup. It returns ready, version, sample_rate (16000), max_seconds (12), multilingual, model, languages, device, compute_type, modes, translation_target, busy, completed, and failed. Precision is the requested CTranslate2 compute mode; internal hardware fallback may differ; actual_compute_type reports the loaded model's compute type. A native watchdog exits if an individual running inference remains busy for more than 90 seconds. Use Docker or systemd for automatic restart.

Stream scheduling

One request may upload, queue or run per stream. Independent streams share FIFO worker admission. Default limits: twelve sessions, one inference worker, twenty seconds queue wait, ten seconds upload time, and twelve seconds audio per chunk. A busy stream receives 429 without buffering a second body. At most twelve bodies can exist at default capacity. Increase workers with --workers; use the same setting for the model and scheduler (the CLI handles this).

Idle sessions and their retry caches expire opportunistically after 120 seconds. DELETE /streams/{stream_id} with X-Caption-Local: 1 closes an idle session and forgets its cached results immediately; a busy session returns 409. Stop and drain before closing. The browser does this automatically on successful Stop. There is no persistent reservation while a stream is silent beyond the idle timeout.

Health additionally reports max_streams, workers, threads_per_worker, beam_size, sessions, running, pending (uploads plus queued requests), queue_timeout, rejected (busy/capacity admission refusals), and queue_timeouts. queue_seconds excludes upload and inference. A cached retry returns the original timing values. Stream IDs are not credentials: local trusted clients can access this API and must use distinct, unguessable IDs to avoid accidental collisions.

Transcribe

POST /transcribe?language=es&mode=both

Required headers:

• Content-Type: application/octet-stream
• X-Caption-Local: 1
• Optional X-Stream-ID: 8–80 ASCII letters, numbers, _ or -; use a random UUID per input stream. Omission uses the shared legacy stream.
• Optional X-Request-ID: 8–80 ASCII letters, numbers, _ or -.

Body: little-endian float32 mono PCM, 16 kHz, normalized to [-1, 1], finite samples, 0.01–12 seconds. No WAV header. Upload time is limited to ten seconds.

language defaults to en and accepts Whisper language codes or auto. mode is transcribe (default), translate (English only), or both. English-only models reject non-English languages and translation modes.

{
  "text": "Bienvenidos al teatro.",
  "transcript": "Bienvenidos al teatro.",
  "translation": "Welcome to the theatre.",
  "language": "es",
  "output_language": "es",
  "translation_language": "en",
  "mode": "both",
  "audio_seconds": 3.0,
  "committed_seconds": 3.0,
  "request_id": "example-12345678",
  "stream_id": "example-stream-123",
  "queue_seconds": 0.2,
  "inference_seconds": 0.8
}

Timings are illustrative. text is the transcript except in translate-only mode, where it is the English translation. language describes source speech; output_language describes text. Unrequested output fields are empty strings. Silence with automatic detection can return language: null.

Rolling windows

Add window=1, final=0, and context_seconds=0.5 to decode an unfinished window. Context is previously committed audio retained at the front, from zero to one second. At least two seconds of new audio are required for an unfinished window.

The decoder retains the last second of unfinished speech and emits only decoded words ending before that cutoff. committed_seconds identifies how far the caller may advance in the supplied audio. Retain 0.5 seconds before this point as context for the next window; append new audio after the old uncommitted tail. Do not advance by the request's whole duration. On a pause or Stop, send final=1 (the default), which commits the complete supplied window. The browser's tested buffer implementation is in static/audio-buffer.js.

For translation, the service first chooses the source-language word boundary and then translates only the newly committed audio. Both outputs therefore correspond to the same committed source interval. Word timestamps are estimates: this method reduces boundary artifacts but cannot guarantee perfect word alignment.

Speaker labels

Start the service with CAPTION_DIARIZATION=1 (or deploy.py run --speakers); /health then reports "speakers": true. Add speakers=1 to a rolling window (window=1, not mode=translate) to receive segments, the committed words grouped by voice in spoken order, plus diarization_seconds:

"segments": [{"speaker": "S1", "label": "Hamlet", "text": "To be, or not to be"}]

Speaker IDs are assigned per room (local without rooms) in order of first appearance and stay stable across streams until reset. speaker is null when no voice could be matched; label is the name assigned for this show, or Speaker N. Audio and voiceprints are never stored.

Room cookie routes manage names:

• GET /api/rooms/<id>/speakers returns {enabled, cast, assignments, detected}.
• PATCH /api/rooms/<id>/speakers accepts cast (up to 50 names, persisted with the room) and/or assignments ({"S1": "Hamlet"}; an empty name clears it).
• POST /api/rooms/<id>/speakers/reset forgets detected voices and names for a new show and keeps the cast.

CAPTION_SPEAKER_THRESHOLD (default 0.5) sets the voice similarity needed to reuse a speaker; CAPTION_MAX_SPEAKERS (default 12) caps speakers per room.

Retry and error behavior

Reuse the same stream ID and request ID with identical audio and options after a lost response. Successful results are kept in memory for up to 120 seconds, with at most 64 entries per stream. Different audio/options with a cached ID returns 409. The cache does not survive server restart and does not guarantee exactly-once delivery to external caption relays. A client disconnect does not unlock a running worker.

Status	Meaning	Client action
400	Invalid audio, language, mode, context or ID	Correct input; retain audio if recoverable
401	Missing/expired room cookie or closed room	Open the latest admin link
403	Wrong/missing local header or foreign Origin	Use the local page/client
408	Upload exceeded ten seconds	Retry the same request ID
409	Reused ID with different input	Fix client ID handling
413 / 415	Oversized audio / wrong content type	Correct the request
429	Stream busy, stream capacity reached, or queue deadline exceeded	Honor Retry-After and retain pending audio
500	Inference failed	Retain audio; inspect server logs, then retry
503	HTTP concurrency limit or restarting service	Back off and retry

The browser retries network errors and 408/429/502/503/504 up to three times after the first attempt, then stops capture with manual recovery controls. It never silently discards a failed request. Response caching is disabled at the HTTP layer; transcripts are not saved to disk by the service.