Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 21 additions & 14 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,9 @@ console.log('Speech echo WebSocket app listening on port 3000');

Both `WebhookResponse` and `Session` support the same chainable verb methods:

`.say(opts)` `.play(opts)` `.gather(opts)` `.dial(opts)` `.llm(opts)` `.s2s(opts)` `.openai_s2s(opts)` `.google_s2s(opts)` `.elevenlabs_s2s(opts)` `.deepgram_s2s(opts)` `.ultravox_s2s(opts)` `.dialogflow(opts)` `.conference(opts)` `.enqueue(opts)` `.dequeue(opts)` `.hangup()` `.pause(opts)` `.redirect(opts)` `.config(opts)` `.tag(opts)` `.dtmf(opts)` `.listen(opts)` `.transcribe(opts)` `.message(opts)` `.stream(opts)` `.agent(opts)` `.dub(opts)` `.alert(opts)` `.answer(opts)` `.leave()` `.sipDecline(opts)` `.sipRefer(opts)` `.sipRequest(opts)`
`.say(opts)` `.play(opts)` `.gather(opts)` `.dial(opts)` `.llm(opts)` `.s2s(opts)` `.openai_s2s(opts)` `.google_s2s(opts)` `.elevenlabs_s2s(opts)` `.deepgram_s2s(opts)` `.ultravox_s2s(opts)` `.dialogflow(opts)` `.room(opts)` `.enqueue(opts)` `.dequeue(opts)` `.hangup()` `.pause(opts)` `.redirect(opts)` `.config(opts)` `.tag(opts)` `.dtmf(opts)` `.stream(opts)` `.transcribe(opts)` `.message(opts)` `.agent(opts)` `.dub(opts)` `.alert(opts)` `.answer(opts)` `.leave()` `.sipDecline(opts)` `.sipRefer(opts)` `.sipRequest(opts)`

Prefer `.room(opts)` and `.stream(opts)`. `.conference(opts)` and `.listen(opts)` remain as backward-compatible synonyms (same options), but new apps should use `room`/`stream`.

All methods accept the same options as the corresponding verb JSON Schema. Methods are chainable — they return `this`.

Expand Down Expand Up @@ -415,9 +417,9 @@ session.on('close', (code, reason) => { /* connection closed */ });
session.on('error', (err) => { /* error */ });
```

## Audio WebSocket (Listen/Stream)
## Audio WebSocket (Stream/Listen)

The `listen` and `stream` verbs open a separate WebSocket connection from jambonz to your application, carrying raw audio. This is independent of the control WebSocket (`ws.jambonz.org`) — it uses the `audio.drachtio.org` subprotocol.
The `stream` verb (and its backward-compatible synonym `listen`) opens a separate WebSocket connection from jambonz to your application, carrying raw audio. This is independent of the control WebSocket (`ws.jambonz.org`) — it uses the `audio.drachtio.org` subprotocol.

### Receiving Audio in the Same Application

Expand All @@ -433,14 +435,14 @@ const makeService = createEndpoint({ server, port: 3000 });
// Control pipe — handles call sessions
const svc = makeService({ path: '/' });

// Audio pipe — receives listen/stream audio
// Audio pipe — receives stream audio
const audioSvc = makeService.audio({ path: '/audio-stream' });

svc.on('session:new', (session) => {
session
.answer()
.say({ text: 'Recording your audio.' })
.listen({
.stream({
url: '/audio-stream', // relative path — jambonz connects back to same server
sampleRate: 16000,
mixType: 'mono',
Expand Down Expand Up @@ -472,15 +474,15 @@ The `stream` object in the `connection` event is an `AudioStream` instance:

**Events**:
- `audio` — L16 PCM binary frame (`Buffer`)
- `dtmf` — `{digit, duration}` (only if `passDtmf: true` on listen verb)
- `dtmf` — `{digit, duration}` (only if `passDtmf: true` on stream verb)
- `playDone` — `{id}` (after non-streaming playAudio completes)
- `mark` — `{name, event}` where event is `'playout'` or `'cleared'`
- `close` — `(code, reason)`
- `error` — `(err)`

### Sending Audio Back (Bidirectional)

The listen verb supports bidirectional audio. There are two modes, controlled by the `bidirectionalAudio.streaming` option on the listen verb.
The stream verb supports bidirectional audio. There are two modes, controlled by the `bidirectionalAudio.streaming` option on the stream verb.

**Non-streaming mode** (`streaming: false`, the default) — send complete audio clips as base64:

Expand All @@ -502,7 +504,7 @@ Up to 10 playAudio commands can be queued simultaneously.
**Streaming mode** (`streaming: true`) — send raw binary PCM frames directly:

```typescript
// In the listen verb config:
// In the stream verb config:
// bidirectionalAudio: { enabled: true, streaming: true, sampleRate: 16000 }

stream.on('audio', (pcm) => {
Expand All @@ -513,16 +515,16 @@ stream.on('audio', (pcm) => {

### Marks (Synchronization Markers)

Marks let you track when streamed audio has been played out to the caller. They work **only with bidirectional streaming mode** — you must enable `bidirectionalAudio: { enabled: true, streaming: true }` on the listen verb.
Marks let you track when streamed audio has been played out to the caller. They work **only with bidirectional streaming mode** — you must enable `bidirectionalAudio: { enabled: true, streaming: true }` on the stream verb.

The pattern is: stream audio via `sendAudio()`, then send a mark. When all the audio sent before the mark finishes playing out, jambonz sends back a mark event with `event: 'playout'`. This is how you know the caller has heard a specific chunk of audio.

```typescript
// Listen verb must enable bidirectional streaming for marks to work
// Stream verb must enable bidirectional streaming for marks to work
session
.listen({
.stream({
url: '/audio',
actionHook: '/listen-done',
actionHook: '/stream-done',
bidirectionalAudio: {
enabled: true,
streaming: true,
Expand Down Expand Up @@ -557,7 +559,7 @@ audioSvc.on('connection', (stream) => {

```typescript
stream.killAudio(); // Stop playback, flush buffer
stream.disconnect(); // Close connection, end listen verb
stream.disconnect(); // Close connection, end stream verb
stream.sendMark('sync-pt'); // Insert synchronization marker
stream.clearMarks(); // Clear all pending markers
stream.close(); // Close the WebSocket
Expand Down Expand Up @@ -710,7 +712,7 @@ Complete working examples are in the `examples/` directory:
- **echo** — Speech echo using gather with actionHook pattern (webhook + WebSocket). The canonical example for understanding actionHook event handling.
- **ivr-menu** — Interactive menu with speech and DTMF input (webhook)
- **dial** — Simple outbound dial to a phone number (webhook)
- **listen-record** — Record audio using the listen verb to stream to a WebSocket (webhook)
- **stream-record** — Record audio using the stream verb to stream to a WebSocket (webhook)
- **voice-agent** — LLM-powered conversational AI with tool calls (webhook + WebSocket)
- **openai-realtime** — OpenAI Realtime API voice agent with function calling (WebSocket)
- **deepgram-voice-agent** — Deepgram Voice Agent API with function calling (WebSocket)
Expand All @@ -719,3 +721,8 @@ Complete working examples are in the `examples/` directory:
- **queue-with-hold** — Call queue with hold music and agent dequeue (webhook + WebSocket)
- **call-recording** — Mid-call recording control via REST API and inject commands (webhook + WebSocket)
- **realtime-translator** — Bridges two parties with real-time speech translation using STT, Google Translate, and TTS dub tracks. Multi-file example with `src/routes/` structure (WebSocket)
- **room-with-stream** — A Room with a nested bidirectional audio stream forked to a WebSocket (WebSocket)
- **stream-then-room** — 1:1 stream, then move the caller + stream into a Room mid-call (WebSocket)
- **s2s-move-to-room** — Ultravox s2s, then move the caller + agent into a Room mid-call (WebSocket)
- **room-say** — injectSay a one-shot TTS announcement heard by the whole Room (WebSocket)
- **room-play-tone** — injectPlay a tone heard by the whole Room (WebSocket)
13 changes: 10 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ const audioSvc = makeService.audio({ path: '/audio-stream' });
svc.on('session:new', (session) => {
session
.say({ text: 'Listening...' })
.listen({
.stream({
url: '/audio-stream', // relative path — jambonz connects back to same server
sampleRate: 8000,
bidirectionalAudio: {
Expand Down Expand Up @@ -369,7 +369,9 @@ import { JambonzClient } from '@jambonz/sdk/client';

Both `WebhookResponse` and WebSocket `Session` support the same chainable verb methods:

`.say()` `.play()` `.gather()` `.dial()` `.llm()` `.conference()` `.enqueue()` `.dequeue()` `.hangup()` `.pause()` `.redirect()` `.config()` `.tag()` `.dtmf()` `.listen()` `.transcribe()` `.message()` `.stream()` `.agent()` `.dub()` `.alert()` `.answer()` `.leave()` `.sipDecline()` `.sipRefer()` `.sipRequest()`
`.say()` `.play()` `.gather()` `.dial()` `.llm()` `.room()` `.enqueue()` `.dequeue()` `.hangup()` `.pause()` `.redirect()` `.config()` `.tag()` `.dtmf()` `.stream()` `.transcribe()` `.message()` `.agent()` `.dub()` `.alert()` `.answer()` `.leave()` `.sipDecline()` `.sipRefer()` `.sipRequest()`

> Prefer `.room()` and `.stream()`. `.conference()` and `.listen()` are still supported as synonyms for backward compatibility.

All methods accept the same options as the corresponding [verb JSON schemas](schema/verbs/) and are chainable.

Expand Down Expand Up @@ -427,13 +429,18 @@ See the [examples/](examples/) directory:
| [echo](examples/echo/) | Webhook + WS | Speech echo using gather with actionHook |
| [ivr-menu](examples/ivr-menu/) | Webhook | Interactive menu with speech and DTMF |
| [dial](examples/dial/) | Webhook | Outbound dial to a phone number |
| [listen-record](examples/listen-record/) | Webhook | Record audio via WebSocket stream |
| [stream-record](examples/stream-record/) | Webhook | Record audio via WebSocket stream |
| [voice-agent](examples/voice-agent/) | Webhook + WS | LLM-powered conversational AI with tool calls |
| [openai-realtime](examples/openai-realtime/) | WebSocket | OpenAI Realtime API voice agent |
| [deepgram-voice-agent](examples/deepgram-voice-agent/) | WebSocket | Deepgram Voice Agent API |
| [llm-streaming](examples/llm-streaming/) | WebSocket | Anthropic LLM with TTS streaming and barge-in |
| [queue-with-hold](examples/queue-with-hold/) | Webhook + WS | Call queue with hold music |
| [call-recording](examples/call-recording/) | Webhook + WS | Mid-call recording control |
| [room-with-stream](examples/room-with-stream/) | WebSocket | A Room with a nested bidirectional audio stream |
| [stream-then-room](examples/stream-then-room/) | WebSocket | 1:1 stream, then move caller + stream into a Room |
| [s2s-move-to-room](examples/s2s-move-to-room/) | WebSocket | Ultravox s2s, then move caller + agent into a Room |
| [room-say](examples/room-say/) | WebSocket | injectSay a one-shot announcement heard by the whole Room |
| [room-play-tone](examples/room-play-tone/) | WebSocket | injectPlay a tone heard by the whole Room |

## Publishing to npm

Expand Down
7 changes: 6 additions & 1 deletion examples/bedrock-agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@ const envVars = {
description: 'ElevenLabs voice id',
default: 'hpp4J3VqNfWAUOO0d1Us',
},
TAVILY_API_KEY: {
type: 'string' as const,
description: 'Tavily API key for the web_search tool',
obscure: true,
},
SYSTEM_PROMPT: {
type: 'string' as const,
description: 'System prompt for the voice agent',
Expand Down Expand Up @@ -136,7 +141,7 @@ function handleSession(session: Session, opts: AgentOptions) {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({
api_key: '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',
api_key: session.data.env_vars?.TAVILY_API_KEY,
query,
max_results: 3,
search_depth: 'basic',
Expand Down
63 changes: 63 additions & 0 deletions examples/room-play-tone/ws-app.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
import http from 'http';
import { createEndpoint } from '@jambonz/sdk/websocket';

/*
* Play a tone (or any file/URL) into a Room, heard by every member, mid-call.
*
* The caller joins a Room. PLAY_DELAY_MS after joining, we call
* session.injectPlay({...}) — the audio is mixed into the Room and heard by all
* members. The optional `id` is echoed back on the play-start / play-done events.
*
* The room verb subscribes to the play lifecycle via statusEvents + statusHook.
*/

const server = http.createServer();
const makeService = createEndpoint({
server,
port: 3000,
envVars: {
ROOM_NAME: { type: 'string', description: 'Room name', default: 'demo-room' },
PLAY_URL: { type: 'string', description: 'audio to play into the Room (file/http url or tone://)', default: 'tone://?freq=880&duration=400' },
PLAY_DELAY_MS: { type: 'number', description: 'ms after joining before playing', default: 5000 },
},
});

const svc = makeService({ path: '/room-play-tone' });

svc.on('session:new', (session) => {
const env = session.data.env_vars || {};
const room = env.ROOM_NAME || 'demo-room';
const url = env.PLAY_URL || 'tone://?freq=880&duration=400';
const playDelayMs = parseInt(env.PLAY_DELAY_MS ?? '5000', 10);

console.log(`Incoming call ${session.callSid} -> Room '${room}', playing a tone in ${playDelayMs}ms`);

let played = false;
session
.on('/room-status', (evt: Record<string, any>) => {
console.log(`room status: ${evt?.event}`, { playId: evt?.play_id, id: evt?.id, reason: evt?.reason });
// play once the caller has joined the Room
if ((evt?.event === 'join' || evt?.event === 'start') && !played) {
played = true;
setTimeout(() => {
console.log('>>> injectPlay into the Room');
session.injectPlay({ url, id: 'tone' });
}, Math.max(0, playDelayMs));
}
})
.on('/room-done', () => session.reply())
.on('close', (code: number) => console.log(`session ${session.callSid} closed: ${code}`))
.on('error', (err: Error) => console.error('session error:', err));

session
.answer()
.room({
name: room,
actionHook: '/room-done',
statusHook: '/room-status',
statusEvents: ['start', 'end', 'join', 'leave', 'play-start', 'play-done'],
})
.send();
});

console.log('room-play-tone listening on port 3000 (path /room-play-tone)');
67 changes: 67 additions & 0 deletions examples/room-say/ws-app.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
import http from 'http';
import { createEndpoint } from '@jambonz/sdk/websocket';

/*
* Speak TTS into a Room, heard by every member, mid-call.
*
* The caller joins a Room. SAY_DELAY_MS after joining, we call
* session.injectSay({...}) — a one-shot announcement synthesized into the Room and
* heard by all members (not just the caller). The optional `id` is echoed back on
* the say-start / say-done events so you can correlate them.
*
* The room verb subscribes to the say lifecycle via statusEvents + statusHook;
* say-start fires when audio begins, say-done when it finishes.
*/

const server = http.createServer();
const makeService = createEndpoint({
server,
port: 3000,
envVars: {
ROOM_NAME: { type: 'string', description: 'Room name', default: 'demo-room' },
SAY_TEXT: { type: 'string', description: 'text to announce into the Room', default: 'Welcome — this announcement is heard by everyone in the room.' },
SAY_VENDOR: { type: 'string', description: 'TTS vendor (optional; account default if unset)', required: false },
SAY_DELAY_MS: { type: 'number', description: 'ms after joining before the announcement', default: 5000 },
},
});

const svc = makeService({ path: '/room-say' });

svc.on('session:new', (session) => {
const env = session.data.env_vars || {};
const room = env.ROOM_NAME || 'demo-room';
const text = env.SAY_TEXT || 'Welcome — this announcement is heard by everyone in the room.';
const synthesizer = env.SAY_VENDOR ? { vendor: env.SAY_VENDOR } : undefined;
const sayDelayMs = parseInt(env.SAY_DELAY_MS ?? '5000', 10);

console.log(`Incoming call ${session.callSid} -> Room '${room}', announcing in ${sayDelayMs}ms`);

let announced = false;
session
.on('/room-status', (evt: Record<string, any>) => {
console.log(`room status: ${evt?.event}`, { sayId: evt?.say_id, id: evt?.id, reason: evt?.reason });
// announce once the caller has joined the Room
if ((evt?.event === 'join' || evt?.event === 'start') && !announced) {
announced = true;
setTimeout(() => {
console.log('>>> injectSay into the Room');
session.injectSay({ text, id: 'announcement', ...(synthesizer && { synthesizer }) });
}, Math.max(0, sayDelayMs));
}
})
.on('/room-done', () => session.reply())
.on('close', (code: number) => console.log(`session ${session.callSid} closed: ${code}`))
.on('error', (err: Error) => console.error('session error:', err));

session
.answer()
.room({
name: room,
actionHook: '/room-done',
statusHook: '/room-status',
statusEvents: ['start', 'end', 'join', 'leave', 'say-start', 'say-done'],
})
.send();
});

console.log('room-say listening on port 3000 (path /room-say)');
64 changes: 64 additions & 0 deletions examples/room-with-stream/ws-app.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import http from 'http';
import { createEndpoint } from '@jambonz/sdk/websocket';

/*
* A Room with a nested audio stream.
*
* The caller is answered straight into a Room whose `stream` property forks the
* room's mixed audio to a WebSocket endpoint. With bidirectionalAudio enabled,
* whatever the endpoint streams back is mixed into the room and heard by every
* member — so the stream socket is effectively a participant in the Room.
*
* (`stream` is the preferred synonym for the nested `listen` property.)
*/

const server = http.createServer();
const makeService = createEndpoint({
server,
port: 3000,
envVars: {
STREAM_WS_URL: { type: 'string', description: 'WebSocket URL to stream the room audio to', required: true },
ROOM_NAME: { type: 'string', description: 'Room name', default: 'demo-room' },
SAMPLE_RATE: { type: 'number', description: 'audio sample rate for the stream', default: 8000 },
},
});

const svc = makeService({ path: '/room-with-stream' });

svc.on('session:new', (session) => {
const env = session.data.env_vars || {};
const streamUrl = env.STREAM_WS_URL;
const room = env.ROOM_NAME || 'demo-room';
const sampleRate = parseInt(env.SAMPLE_RATE ?? '8000', 10);

console.log(`Incoming call ${session.callSid} -> Room '${room}' with a nested audio stream`);
if (!streamUrl) {
console.error('STREAM_WS_URL is not configured as an application environment variable');
return;
}

session
.on('/room-done', () => session.reply()) // room verb completed (caller left)
.on('/stream-event', (evt: Record<string, any>) => console.log('stream event:', evt?.type))
.on('close', (code: number) => console.log(`session ${session.callSid} closed: ${code}`))
.on('error', (err: Error) => console.error('session error:', err));

session
.answer()
.room({
name: room,
beep: true,
actionHook: '/room-done',
// nested stream: fork the room's audio to STREAM_WS_URL; audio streamed back
// is mixed into the room (bidirectional).
stream: {
url: streamUrl,
sampleRate,
bidirectionalAudio: { enabled: true, streaming: true, sampleRate },
actionHook: '/stream-event',
},
})
.send();
});

console.log('room-with-stream listening on port 3000 (path /room-with-stream)');
Loading