ElevenLabsAgent

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:82

ElevenLabs Conversational AI agent — covers STT + LLM + TTS in a single WebSocket.

See

• ElevenLabsAgentConfig for configuration options
• BaseAgentProvider for the shared agent provider base class
• ElevenLabsAgentEvent for events emitted via onAgentEvent

Extends

• BaseAgentProvider

Constructors

Constructor

new ElevenLabsAgent(config?, logger?): ElevenLabsAgent;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:107

Parameters

Returns

ElevenLabsAgent

Overrides

BaseAgentProvider.constructor

Properties

Accessors

isProxyMode

Get Signature

get protected isProxyMode(): boolean;

Defined in: src/providers/base/BaseProvider.ts:286

Whether the provider is in proxy mode.

Returns

boolean

true when proxyUrl is set.

Inherited from

BaseAgentProvider.isProxyMode

Methods

assertAuth()

protected assertAuth(): void;

Defined in: src/providers/base/BaseProvider.ts:272

Validate that auth is configured (either apiKey or proxyUrl).

Returns

void

Remarks

Call this in onInitialize() for any provider that requires external authentication. Native providers (NativeSTT, NativeTTS) and in-browser providers (WebLLM) should NOT call this method.

Throws

ProviderInitializationError Thrown when neither apiKey nor proxyUrl is set.

Inherited from

BaseAgentProvider.assertAuth

assertReady()

protected assertReady(): void;

Defined in: src/providers/base/BaseProvider.ts:255

Guard that throws if the provider has not been initialized.

Returns

void

Remarks

Call at the start of any method that requires the provider to be ready.

Throws

Error Thrown with a descriptive message when initialized is false.

Inherited from

BaseAgentProvider.assertReady

cleanupPendingState()

protected cleanupPendingState(): void;

Defined in: src/providers/base/BaseAgentProvider.ts:611

Clean up all pending promises and timers. Call from your onDispose() or connection close handler.

Returns

void

Inherited from

BaseAgentProvider.cleanupPendingState

clearBufferedAssistantText()

protected clearBufferedAssistantText(): void;

Defined in: src/providers/base/BaseAgentProvider.ts:526

Discard assistant text buffered for a response that is being abandoned.

Returns

void

Remarks

Call this when the user starts a new utterance (barge-in, or the provider’s “user started speaking” event). Text left in the buffer by a response nobody consumed would otherwise be handed to the next turn’s iterator as if it were that turn’s answer.

Inherited from

BaseAgentProvider.clearBufferedAssistantText

configureInputFormat()

configureInputFormat(metadata): void;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:282

Record the input provider’s capture rate.

Parameters

Returns

void

Remarks

The input format belongs to the agent’s dashboard configuration, not to anything this client can negotiate, so a mismatch can only be reported — see the user_input_audio_format check in the initiation metadata.

Overrides

BaseAgentProvider.configureInputFormat

connect()

connect(): Promise<void>;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:171

Open the WebSocket and complete the conversation initiation handshake.

Returns

Promise<void>

Remarks

The connection follows a 2-step handshake:

1. Open → initiation data — a WebSocket is opened to the signed URL, the proxy, or wss://api.elevenlabs.io/v1/convai/conversation (with agent_id, plus xi-api-key in direct mode with an API key). On open, the client sends conversation_initiation_client_data with any configured overrides and dynamic variables.
2. conversation_initiation_metadata — the server announces the conversation ID and audio formats. The output format is parsed and forwarded to the output provider; the connection is now ready.

The handshake must complete within the configured timeout (default 10 s, override via config.timeout).

Idempotent / piggyback behavior: returns immediately when already connected; piggybacks on an in-flight handshake instead of opening a second socket.

Throws

If the handshake does not complete within the timeout.

Throws

If the WebSocket errors or closes before the handshake finishes.

Overrides

BaseAgentProvider.connect

disconnect()

disconnect(): Promise<void>;

Defined in: src/providers/base/BaseAgentProvider.ts:246

No-op — the agent connection is persistent.

Returns

Promise<void>

Remarks

The orchestrator calls stt.disconnect() / tts.disconnect() during turn-taking and barge-in. For agent providers, the connection must not be torn down between turns. The real close happens in dispose().

Inherited from

BaseAgentProvider.disconnect

dispose()

dispose(): Promise<void>;

Defined in: src/providers/base/BaseProvider.ts:154

Clean up resources and dispose of the provider.

Returns

Promise<void>

Remarks

Delegates to the subclass hook onDispose and resets the initialized flag. If the provider is not initialized, the call is a no-op.

Throws

Re-throws any error raised by onDispose.

Inherited from

BaseAgentProvider.dispose

emitAssistantText()

protected emitAssistantText(text): void;

Defined in: src/providers/base/BaseAgentProvider.ts:507

Deliver the assistant’s response text. Resolves the pending generateFromMessages iterator.

Parameters

Returns

void

Remarks

When no iterator has registered yet — the orchestrator’s STT → LLM flow runs asynchronously after emitUserTranscription, so a fast agent can respond first — the text is buffered and consumed by the next iterator for this turn instead of being dropped.

Inherited from

BaseAgentProvider.emitAssistantText

emitAudioChunk()

protected emitAudioChunk(data): void;

Defined in: src/providers/base/BaseAgentProvider.ts:545

Forward a binary audio chunk to the output provider.

Parameters

Returns

void

Remarks

The audio format is re-asserted before every chunk. Barge-in tears down playback via BrowserAudioOutput.stop(), which clears the player’s metadata, and nothing else replays it — without this, raw PCM arriving after the first barge-in has no format to decode with and the rest of the session is silent. emitOutputMetadata is a cheap field assignment on the output side, so re-asserting per chunk costs nothing and removes the race between “playback stopped” and “next chunk”.

Inherited from

BaseAgentProvider.emitAudioChunk

emitOutputMetadata()

protected emitOutputMetadata(): void;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:263

Emit audio format metadata so the output provider can decode PCM.

Returns

void

Nothing — metadata is delivered via the registered metadataCallback.

Remarks

Called after the handshake and again before every audio chunk (because BrowserAudioOutput.stop() clears metadata during barge-in). Must therefore be cheap and side-effect free — subclasses should do nothing but call this.metadataCallback?.(metadata).

Overrides

BaseAgentProvider.emitOutputMetadata

emitUserTranscription()

protected emitUserTranscription(text): void;

Defined in: src/providers/base/BaseAgentProvider.ts:473

Emit a user transcription. Creates a new per-turn audio promise and triggers the orchestrator’s STT -> LLM flow.

Parameters

Returns

void

Inherited from

BaseAgentProvider.emitUserTranscription

finalize()

finalize(): Promise<void>;

Defined in: src/providers/base/BaseAgentProvider.ts:425

Wait for the agent to finish sending all audio for the current turn.

Returns

Promise<void>

A promise that resolves when the current turn’s audio is complete or the 30 s timeout elapses.

Remarks

Resolves when the subclass calls markAudioDone, or after a 30-second safety timeout — whichever comes first. The timeout prevents the orchestrator from hanging indefinitely if the agent never signals completion.

Inherited from

BaseAgentProvider.finalize

generate()

generate(prompt, options?): Promise<AsyncIterable<string, any, any>>;

Defined in: src/providers/base/BaseAgentProvider.ts:317

Generate a response from a single user prompt.

Parameters

Returns

Promise<AsyncIterable<string, any, any>>

An async iterable that yields the assistant’s response as a single text chunk.

Remarks

Wraps the prompt in a user message and delegates to generateFromMessages.

Throws

AbortError if the supplied options.signal is aborted before the server responds.

Inherited from

BaseAgentProvider.generate

generateFromMessages()

generateFromMessages(_messages, _options?): Promise<AsyncIterable<string, any, any>>;

Defined in: src/providers/base/BaseAgentProvider.ts:349

Returns an async iterable that yields the assistant’s response text.

Parameters

Returns

Promise<AsyncIterable<string, any, any>>

An async iterable that yields the assistant’s response as a single text chunk.

Remarks

Agent APIs manage their own conversation context. This method does NOT re-send the message history — the server already has it. Instead, it returns an iterable that blocks until the server sends the assistant’s response text, then yields it as a single chunk.

The returned iterator resolves when the subclass calls emitAssistantText. If the connection closes before a response arrives, the iterator rejects via rejectPendingLLM.

Throws

AbortError if the supplied _options.signal is aborted before the server responds.

Inherited from

BaseAgentProvider.generateFromMessages

getConfig()

getConfig(): BaseProviderConfig;

Defined in: src/providers/base/BaseProvider.ts:187

Get a shallow copy of the current provider configuration.

Returns

BaseProviderConfig

A new object containing all current configuration values.

Inherited from

BaseAgentProvider.getConfig

initialize()

initialize(): Promise<void>;

Defined in: src/providers/base/BaseProvider.ts:127

Initialize the provider, making it ready for use.

Returns

Promise<void>

Remarks

Calls the subclass hook onInitialize. If the provider has already been initialized the call is a no-op.

Throws

ProviderInitializationError Thrown when onInitialize rejects. The original error is wrapped with the provider class name for diagnostics.

Inherited from

BaseAgentProvider.initialize

isAudioReady()

isAudioReady(chunk): boolean;

Defined in: src/providers/base/BaseAgentProvider.ts:458

Check whether an audio chunk is ready for playback.

Parameters

Returns

boolean

true when the chunk contains a non-empty audio buffer.

Inherited from

BaseAgentProvider.isAudioReady

isFinal()

isFinal(result): boolean;

Defined in: src/providers/base/BaseAgentProvider.ts:294

Check whether a transcription is a final result.

Parameters

Returns

boolean

true when the result has isFinal set.

Inherited from

BaseAgentProvider.isFinal

isInterim()

isInterim(_result): boolean;

Defined in: src/providers/base/BaseAgentProvider.ts:284

Check whether a transcription is an interim (non-final) result.

Parameters

Returns

boolean

Always false for agent providers.

Remarks

Agent providers only surface final transcriptions — always returns false.

Inherited from

BaseAgentProvider.isInterim

isPreflight()

isPreflight(_result): boolean;

Defined in: src/providers/base/BaseAgentProvider.ts:271

Check whether a transcription is a speculative preflight result.

Parameters

Returns

boolean

Always false for agent providers.

Remarks

Agent providers do not emit preflight results — always returns false.

Inherited from

BaseAgentProvider.isPreflight

isReady()

isReady(): boolean;

Defined in: src/providers/base/BaseProvider.ts:178

Check whether the provider has been initialized and is ready.

Returns

boolean

true when initialize has completed successfully and dispose has not yet been called.

Inherited from

BaseAgentProvider.isReady

isUtteranceComplete()

isUtteranceComplete(result): boolean;

Defined in: src/providers/base/BaseAgentProvider.ts:258

Check whether a transcription marks the end of an utterance.

Parameters

Returns

boolean

true when the result has utteranceComplete set.

Inherited from

BaseAgentProvider.isUtteranceComplete

isWebSocketConnected()

isWebSocketConnected(): boolean;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:345

Check whether the underlying WebSocket is currently open.

Returns

boolean

true if the WebSocket exists and its readyState is OPEN.

markAudioDone()

protected markAudioDone(): void;

Defined in: src/providers/base/BaseAgentProvider.ts:558

Signal that all audio for the current turn has been sent. Resolves the pending finalize() call.

Returns

void

Inherited from

BaseAgentProvider.markAudioDone

onAgentEvent()

onAgentEvent(callback): void;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:298

Register a callback for ElevenLabs agent events.

Parameters

Returns

void

See

ElevenLabsAgentEvent for the event union

onAudio()

onAudio(callback): void;

Defined in: src/providers/base/BaseAgentProvider.ts:438

Register the audio callback.

Parameters

Returns

void

Inherited from

BaseAgentProvider.onAudio

onConfigUpdate()

protected onConfigUpdate(_config): void;

Defined in: src/providers/base/BaseProvider.ts:242

Hook called after updateConfig merges new values.

Parameters

Returns

void

Remarks

The default implementation is a no-op. Override in subclasses to react to runtime configuration changes (e.g. reconnect with a new API key).

Inherited from

BaseAgentProvider.onConfigUpdate

onDispose()

protected onDispose(): Promise<void>;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:131

Provider-specific disposal logic.

Returns

Promise<void>

Remarks

Subclasses must implement this method to release any resources acquired during onInitialize (e.g. close connections, free memory).

Overrides

BaseAgentProvider.onDispose

onInitialize()

protected onInitialize(): Promise<void>;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:123

Provider-specific initialization logic.

Returns

Promise<void>

Remarks

Subclasses must implement this method to perform any setup required before the provider can be used (e.g. validate credentials, open connections, load models).

Overrides

BaseAgentProvider.onInitialize

onMetadata()

onMetadata(callback): void;

Defined in: src/providers/base/BaseAgentProvider.ts:448

Register the metadata callback.

Parameters

Returns

void

Inherited from

BaseAgentProvider.onMetadata

onTranscription()

onTranscription(callback): void;

Defined in: src/providers/base/BaseAgentProvider.ts:203

Register the transcription callback.

Parameters

Returns

void

Inherited from

BaseAgentProvider.onTranscription

processAudio()

processAudio(chunk): void;

Defined in: src/providers/base/BaseAgentProvider.ts:193

Forward microphone audio to the agent connection.

Parameters

Returns

void

Remarks

Alias for sendAudio — called by the orchestrator’s input queue.

See

sendAudio

Inherited from

BaseAgentProvider.processAudio

processChunk()

processChunk(_text): void;

Defined in: src/providers/base/BaseAgentProvider.ts:411

No-op alias for sendText.

Parameters

Returns

void

Inherited from

BaseAgentProvider.processChunk

rejectPendingLLM()

protected rejectPendingLLM(error): void;

Defined in: src/providers/base/BaseAgentProvider.ts:571

Reject the pending LLM generator with an error.

Parameters

Returns

void

Inherited from

BaseAgentProvider.rejectPendingLLM

resolveApiKey()

protected resolveApiKey(): Promise<string>;

Defined in: src/providers/base/BaseProvider.ts:321

Resolve the API key, calling the factory if apiKey is a function.

Returns

Promise<string>

The resolved API key string, or 'proxy' in proxy mode.

Inherited from

BaseAgentProvider.resolveApiKey

resolveAuthHeader()

protected resolveAuthHeader(defaultAuthType?): Promise<string | undefined>;

Defined in: src/providers/base/BaseProvider.ts:366

Resolve Authorization header value for the configured auth type.

Parameters

Returns

Promise<string | undefined>

The Authorization header value, or undefined in proxy mode.

Remarks

If apiKey is a factory function it is called to get a fresh token. Returns the header value for REST or server-side WebSocket connections:

• 'token' → 'Token <apiKey>'
• 'bearer' → 'Bearer <apiKey>'

Returns undefined in proxy mode.

Inherited from

BaseAgentProvider.resolveAuthHeader

resolveBaseUrl()

protected resolveBaseUrl(defaultUrl?): string | undefined;

Defined in: src/providers/base/BaseProvider.ts:307

Resolve the base URL for this provider.

Parameters

Returns

string | undefined

The resolved URL, or undefined when all sources are unset.

Remarks

Priority: proxyUrl > endpoint > defaultUrl.

For WebSocket providers (this.type === 'websocket'), the proxy URL’s http(s) scheme is automatically converted to ws(s).

When no URL is configured and defaultUrl is undefined, the return value is undefined — this lets SDK-based providers (Anthropic, OpenAI) fall back to their own built-in defaults.

Inherited from

BaseAgentProvider.resolveBaseUrl

resolveWsProtocols()

protected resolveWsProtocols(defaultAuthType?): Promise<string[] | undefined>;

Defined in: src/providers/base/BaseProvider.ts:342

Resolve WebSocket subprotocol for authentication.

Parameters

Returns

Promise<string[] | undefined>

Subprotocol array for new WebSocket(url, protocols), or undefined.

Remarks

If apiKey is a factory function it is called to get a fresh token. Returns the subprotocol array for direct mode based on authType:

• 'token' → ['token', apiKey] (Deepgram default)
• 'bearer' → ['bearer', apiKey] (OAuth/Bearer tokens)

Returns undefined in proxy mode (no client-side auth needed).

Inherited from

BaseAgentProvider.resolveWsProtocols

sendAudio()

sendAudio(chunk): void;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:258

Send a raw audio chunk to the agent for speech recognition.

Parameters

Returns

void

Remarks

No-op when the WebSocket is not open — audio sent before connect completes or after the socket closes is silently dropped.

Overrides

BaseAgentProvider.sendAudio

sendContextualUpdate()

sendContextualUpdate(text): void;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:325

Send background context to the agent without triggering a response.

Parameters

Returns

void

Remarks

Sends a contextual_update frame. The agent factors the context into subsequent responses but does not reply to it directly.

sendText()

sendText(_chunk): void;

Defined in: src/providers/base/BaseAgentProvider.ts:404

No-op — TTS synthesis is handled server-side.

Parameters

Returns

void

Inherited from

BaseAgentProvider.sendText

sendUserActivity()

sendUserActivity(): void;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:336

Signal user activity to reset the agent’s inactivity timeout.

Returns

void

Remarks

Sends a user_activity frame. Useful to keep the conversation alive while the user is interacting silently (typing, reading).

sendUserMessage()

sendUserMessage(text): void;

Defined in: src/providers/agent/elevenlabs/ElevenLabsAgent.ts:311

Inject a text message as if the user typed it.

Parameters

Returns

void

Remarks

Sends a user_message frame. The agent responds exactly as it would to transcribed speech.

startKeepAlive()

protected startKeepAlive(intervalMs?, sendFn): void;

Defined in: src/providers/base/BaseAgentProvider.ts:594

Start an auto keep-alive timer. Call from your handshake completion.

Parameters

Returns

void

Example

this.startKeepAlive(8_000, () => {
  this.ws.send(JSON.stringify({ type: 'KeepAlive' }));
});

Inherited from

BaseAgentProvider.startKeepAlive

stopKeepAlive()

protected stopKeepAlive(): void;

Defined in: src/providers/base/BaseAgentProvider.ts:600

Stop the keep-alive timer.

Returns

void

Inherited from

BaseAgentProvider.stopKeepAlive

updateConfig()

updateConfig(config): void;

Defined in: src/providers/base/BaseProvider.ts:201

Merge partial configuration updates into the current config.

Parameters

Returns

void

Remarks

After merging, the subclass hook onConfigUpdate is called so providers can react to changed values at runtime.

Inherited from

BaseAgentProvider.updateConfig