lee/puter
1
0 Fork 0
mirror of https://github.com/HeyPuter/puter.git synced 2026-11-04 13:03:37 +00:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity

feat(ai): retire assistant API and document realtime beta

Browse Source
This commit is contained in:
404oops committed 2026-10-29 11:24:45 +01:00
1 parent 67c772b2f4
commit 5071cfd802
16 files changed
+190 -417
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/backend/services/realtime/CONVERSATION_API.md
-169
View File
@@ -1,169 +0,0 @@
# Conversation API contract
The SDK exposes the composable session through `puter.ai.convo()`. OpenAI is the
baseline provider. The lower-level `puter.ai.assistant.connect()` transport is
also supported.
## Preparation and connection
```js
const convo = await puter.ai.convo({
model: 'gpt-realtime-2.1-mini',
instructions: 'You are a booking assistant.',
output: ['text', 'audio'],
voice: 'alloy',
});
convo.on('response.output_text.delta', ({ delta }) => renderText(delta));
convo.on('error', handleError);
await convo.connect();
convo.sendText('Hello!');
```
The factory validates options, resolves the model, and authenticates the user.
It does not open a provider connection or request media permissions. Connection
rechecks authentication and credit, applies configuration, and resolves when
the provider acknowledges readiness. Attach listeners before connecting.
The lifecycle is `prepared → connecting → connected → closed`. Concurrent
`connect()` calls share the same attempt. Calling it when connected resolves
without opening another socket. After closing, create a new conversation.
Calling `close()` during connection rejects the connection attempt and releases
its resources. Sending before readiness rejects with `session_not_ready`.
## Optional abstractions on one session
Native OpenAI events are the foundation. Convenience methods use the same
connection and event stream. No provider adapter registry is needed.
```js
convo.send({ type: 'response.create' });
convo.on('response.done', handleResponse);
const unsubscribe = convo.on('*', inspectEvent);
unsubscribe();
await convo.configure({ instructions: 'Keep replies short.' });
convo.sendText('Book a table for Tuesday.');
convo.appendAudio(pcmChunk);
convo.commitAudio();
await convo.interrupt();
convo.close();
```
The public shapes are:
```ts
interface ConvoOptions {
model?: string;
instructions?: string;
output?: ('text' | 'audio')[];
voice?: string;
}
interface ConvoEvent {
type: string;
[field: string]: unknown; // Provider-owned event fields pass through.
}
interface PlaybackPosition {
itemId: string;
contentIndex: number;
playedMs: number;
}
interface ConvoError {
code: string;
message: string;
}
```
Each helper has a specific effect:
- `configure(options)` changes session settings and resolves on acknowledgment.
- `sendText(text)` creates a user message and requests a reply.
- `appendAudio(pcm)` appends PCM16, 24 kHz mono input without ending a turn.
- `commitAudio()` commits input and requests a reply in manual turn mode.
Automatic voice activity detection is the default.
- `interrupt(position?)` cancels the response active when called. Idle calls
resolve immediately. Repeated calls for the same response share one promise.
- `send(event)` sends a native event. Native events remain observable even when
helpers consume them.
- `close()` ends the connection and releases media resources owned by the SDK.
Normalized text, transcript, audio, and turn-completion events supplement native
events. Audio decoding runs only when playback or an audio listener is active.
The backend does not parse frames again or run media helpers.
## Optional browser media
```js
await convo.microphone.start();
await convo.speaker.start({ bargeIn: true });
convo.microphone.mute();
convo.microphone.unmute();
await convo.microphone.stop();
await convo.speaker.stop();
```
Microphone and speaker are independent. Starting either requires a connected
session. Only microphone start requests capture permission. Browser-only helpers
reject with `unsupported_environment` elsewhere. Raw events and audio input
remain available in Node and workers.
Capture uses an AudioWorklet with bounded chunks. Playback has a bounded queue.
No ScriptProcessor, growing transcript buffer, or retained audio history is
part of the SDK. Custom capture and playback can be mixed with built-in helpers.
## Asynchronous interruption
Explicit interruption and provider speech-start events enter the same operation.
The built-in speaker immediately clears playback, then cancellation and playback
history synchronization run asynchronously. The promise resolves after provider
acknowledgment. A disconnect rejects it with a stable connection error.
Interruption captures the response ID. Late deltas from that response remain
visible as native events but never enter built-in playback. Delayed cancellation
must not cancel a newer response. The speaker tracks item ID, content index,
and played milliseconds. Custom players supply that position explicitly.
The `interrupted` event carries `{ type: 'interrupted', source:
'speech' | 'manual', responseId }`. It signals local interruption immediately.
Await the operation before submitting a replacement text turn:
```js
await convo.interrupt();
convo.sendText('Actually, make it Wednesday.');
```
## Latency and bounded work
The relay passes original bytes through after a single bounded JSON parse.
Metering is serialized outside forwarding with bounded retained work. It never
buffers an entire conversation or transcodes audio. Media and convenience
helpers run in the client only when selected.
The implemented transport caps each message at 64 KiB in either direction,
including fragmented messages. Audio producers must stream chunks rather than
upload recordings. Client traffic has a 64 KiB burst and 128 KiB/s sustained
budget, plus 100 messages/s. Upstream traffic allows 200 messages/s and 256 KiB/s with a 256 KiB burst. Message
rates use token buckets with bursts equal to their one-second allowance.
Pending handshake input is bounded by both 32 messages and 64 KiB. Pending
metering is bounded by both 32 events and 256 KiB. Forwarding queues have a
128 KiB high-water mark and a 192 KiB hard admission cap in each direction.
The SDK must check text, PCM chunk, native event size, and outbound buffered
bytes before sending. Configuration and interruption acknowledgment waits need
timeouts and bounded pending operations. Closing rejects all pending operations
and clears timers and listeners. Automatic reconnect and tool execution are
outside this contract.
## SDK contract tests
Use typed fixtures matching the shapes above, with explicit malformed fixtures
for rejection tests. Exercise the built SDK against a test server as well as
unit tests. Cover authentication cancellation, option validation, readiness,
concurrent connection attempts, close during setup, native event preservation,
helper-to-event mappings, interruption races, media cleanup, queue overflow,
and operation timeouts. No production method should exist only for a test.
Cross-session tests must show that flooding one session does not prevent another
from making progress. Timing measurements are evidence for a specific run, not
a claim about fleet capacity. A deployment load test is required before raising
transport limits or enabling media helpers at scale.
src/backend/services/realtime/README.md
+14 -19
View File
@@ -1,8 +1,11 @@
# OpenAI Realtime relay
The relay serves `/puterai/openai/v1/realtime` on the API host. The SDK exposes
it through `puter.ai.convo()` and `puter.ai.assistant.connect()`. The composable conversation
API is specified in [CONVERSATION_API.md](CONVERSATION_API.md).
it through [`puter.ai.convo()`](../../../docs/src/AI/convo.md).
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
Direct clients are documented in the [Realtime WebSocket API reference](../../../docs/src/AI/realtime.md).
## Transport
@@ -20,21 +23,12 @@ The socket opens before authentication completes. Authentication and model
failures arrive as OpenAI-shaped error events followed by a close frame.
```js
const session = await puter.ai.assistant.connect();
session.on('error', ({ error }) => console.error(error.code, error.message));
session.on('response.output_text.delta', ({ delta }) => console.log(delta));
session.send({
type: 'session.update',
session: { type: 'realtime', output_modalities: ['text'] },
});
session.send({
type: 'conversation.item.create',
item: {
type: 'message', role: 'user',
content: [{ type: 'input_text', text: 'Hello!' }],
},
});
session.send({ type: 'response.create' });
const conversation = await puter.ai.convo({ output: ['text'] });
conversation.on('error', (event) => console.error(event));
conversation.on('text', ({ delta }) => console.log(delta));
conversation.on('turn.done', () => conversation.close());
await conversation.connect();
conversation.sendText('Hello!');
```
## Limits and lifecycle
@@ -70,8 +64,9 @@ The provider key resolves through `openai-realtime`, `openai-completion`, then
`realtime.maxConnectionsPerUser` and defaults to 20.
The relay does not run load generators, periodic process-memory sampling, or
per-frame statistics counters. Tests use a local WebSocket fixture, not provider
credentials.
per-frame statistics counters. Unit tests use a local WebSocket fixture.
Optional live integration tests use
`PUTER_TEST_AI_OPENAI_API_KEY`.
## Remaining constraints
src/docs/src/AI.md
+2 -2
View File
@@ -204,8 +204,8 @@ These AI features are supported out of the box when using Puter.js:
- **[`puter.ai.speech2speech()`](/AI/speech2speech/)** - Convert speech in one voice to another voice
- **[`puter.ai.txt2vid()`](/AI/txt2vid/)** - Generate short video clips from text or a reference image with Wan, Seedance, Veo, Kling, MiniMax H3 and other models, or edit a video you supply with Gemini Omni, FLUX 3 or Kling Omni
- **[`puter.ai.speech2txt()`](/AI/speech2txt/)** - Transcribe audio recordings into text
- **[`puter.ai.convo()`](/AI/convo/)** - Prepare a realtime conversation with optional text, microphone, playback, and interruption helpers
- **[`puter.ai.assistant.connect()`](/AI/assistant.connect/)** - Hold a realtime voice and text conversation over OpenAI's Realtime API
- **[Realtime WebSocket API](/AI/realtime/)** - Connect directly without puter.js. This API is in beta and may change abruptly.
- **[`puter.ai.convo()`](/AI/convo/)** - Prepare a beta realtime conversation with optional text, microphone, playback, and interruption helpers
## Examples
src/docs/src/AI/assistant.connect.md
-101
View File
@@ -1,101 +0,0 @@
---
title: puter.ai.assistant.connect()
description: Open a realtime voice and text session with OpenAI's Realtime API.
platforms: [websites, apps, nodejs]
---
Opens a live session with OpenAI's Realtime API through Puter. The session is a WebSocket: you send [OpenAI Realtime client events](https://developers.openai.com/api/docs/guides/realtime) and receive its server events unchanged, so OpenAI's realtime guides apply as written.
Realtime sessions require a paid Puter plan; an account without one receives an `error` event with code `subscription_required` and the session closes. This low-level session does not reconnect; [`puter.ai.convo()`](/AI/convo/) does.
In Node.js this needs a global `WebSocket` (Node.js 22 or later).
## Syntax
```js
puter.ai.assistant.connect(options = {})
```
## Parameters
#### `options` (Object) (optional)
- `model` (String): The realtime model. `gpt-realtime-2.1-mini` (default), `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime` or `gpt-realtime-mini`.
## Return value
A `Promise` that resolves to a session once the socket opens. The session has:
- `send(event)`: send a client event such as `{ type: 'response.create' }`. Throws `{ code: 'session_closed' }` once the session has closed.
- `on(type, handler)`: listen for server events by `type`. `'*'` receives every event, and `'close'` fires once with `{ code, reason }` when the session ends. Returns a function that removes the listener.
- `off(type, handler)`: remove a listener.
- `close()`: end the session.
- `socket`: the underlying `WebSocket`.
The promise rejects with `{ code: 'connection_failed' }` if the socket can't open, with `{ code: 'auth_canceled' }` if the user closes the browser sign-in prompt, and with `{ code: 'auth_required' }` when there is no signed-in user outside the browser.
Errors after the socket opens arrive the way OpenAI sends them: an `error` event whose `error.code` says what went wrong. Puter's own checks use the same shape. Policy errors close with code `1008`. Setup and upstream failures close with `1011`. Server restarts close with `1012`:
| `error.code` | Meaning |
| --------------------- | -------------------------------------------------------------- |
| `invalid_api_key` | The credential was missing or rejected. |
| `model_not_found` | `model` isn't one of the realtime models above. |
| `insufficient_funds` | No usage left, at connect time or after a response. |
| `rate_limit_exceeded` | A connection, message rate, byte rate, or queue limit was reached. |
| `invalid_event` | The client message has an invalid envelope or uses binary input. |
| `connection_timeout` | Session setup took longer than 15 seconds. |
| `provider_unavailable` | The server has no configured realtime provider key. |
| `upstream_error` | The provider rejected the connection or sent an invalid event. |
| `server_restarting` | The server is draining connections. |
Each JSON message is limited to 64 KiB in either direction. Send audio in small chunks. Messages must be JSON objects with a nonempty string `type` of at most 128 characters. Invalid client messages receive `invalid_event` and close with 1008. Oversized client messages close with 1009. Binary input is rejected. Queue and message rate limits are listed in [rate limits and quotas](/rate-limits-and-quotas/).
Usage is charged per token at the model's rate, as each `response.done` arrives.
## Examples
<strong class="example-title">Text conversation</strong>
```js
const session = await puter.ai.assistant.connect();
session.on('response.output_text.delta', (e) => puter.print(e.delta));
session.on('response.done', () => session.close());
session.send({
type: 'session.update',
session: { type: 'realtime', output_modalities: ['text'] },
});
session.send({
type: 'conversation.item.create',
item: {
type: 'message',
role: 'user',
content: [{ type: 'input_text', text: 'Say hi in three words.' }],
},
});
session.send({ type: 'response.create' });
```
<strong class="example-title">Spoken reply</strong>
Audio arrives as base64-encoded 24 kHz 16-bit mono PCM in `response.output_audio.delta` events.
```js
const session = await puter.ai.assistant.connect({ model: 'gpt-realtime-2.1' });
const chunks = [];
session.on('response.output_audio.delta', (e) => chunks.push(e.delta));
session.on('response.output_audio_transcript.done', (e) => puter.print(e.transcript));
session.on('response.done', () => session.close());
session.send({
type: 'conversation.item.create',
item: {
type: 'message',
role: 'user',
content: [{ type: 'input_text', text: 'Tell me a one-line joke.' }],
},
});
session.send({ type: 'response.create' });
```
src/docs/src/AI/convo.md
+7 -1
View File
@@ -6,6 +6,12 @@ platforms: [websites, apps, nodejs, workers]
The conversation API prepares a realtime session, verifies Puter sign-in, and lets you connect when your app is ready. Realtime conversations require a paid Puter plan; usage is billed to the signed-in user. Native OpenAI events and convenience methods use the same session. Microphone capture and speaker playback start when your app requests them.
In Node.js, the SDK requires a global `WebSocket`, available in Node.js 22 or later.
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
To connect without puter.js, see the [Realtime WebSocket API](/AI/realtime/).
## Syntax
```js
@@ -79,7 +85,7 @@ The optional microphone and speaker helpers require a browser with Web Audio sup
| `microphone.mute()` | Disables capture tracks and stops forwarding captured chunks. |
| `microphone.unmute()` | Enables capture tracks and forwarding. |
| `microphone.stop()` | Stops capture tracks and releases capture resources. |
| `speaker.start({ bargeIn: true })` | Starts playback. With barge-in enabled, provider speech-start events interrupt playback and synchronize played audio with conversation history. Starting the speaker requests no microphone access. |
| `speaker.start({ bargeIn: true })` | Starts playback. Barge-in defaults to `true`. With barge-in enabled, provider speech-start events interrupt playback and synchronize played audio with conversation history. Starting the speaker requests no microphone access. |
| `speaker.stop()` | Clears playback and releases its audio context. |
Both start methods require a connected session. Capture and playback are independent, so custom audio input can use the built-in speaker, and built-in capture can use custom playback.
src/docs/src/AI/realtime.md
+97
View File
@@ -0,0 +1,97 @@
---
title: Realtime WebSocket API (Beta)
description: Connect to OpenAI realtime conversations through Puter without puter.js.
platforms: [websites, apps, nodejs, workers]
---
The Realtime WebSocket API lets your client send and receive native OpenAI realtime JSON events through Puter. Puter authenticates the user and charges their account for usage.
**Beta:** This API is incomplete. Its methods, options, events, and behavior may change abruptly, including breaking changes without notice.
For optional text, microphone, playback, and interruption helpers, use [`puter.ai.convo()`](/AI/convo/).
## Connection and authentication
Connect to this URL on the Puter API host:
```text
wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini
```
Self-hosted clients use their deployment's API host. The `model` query parameter defaults to `gpt-realtime-2.1-mini`. The model is fixed for the connection. A session update that changes it is rejected. It also accepts `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime`, and `gpt-realtime-mini`.
Authenticate with a Puter session, app, or full-access API token. Puter supplies the provider credential. Server clients pass this header:
```text
Authorization: Bearer <PUTER_TOKEN>
```
Browser WebSocket clients pass `realtime` and `openai-insecure-api-key.<PUTER_TOKEN>` as subprotocols. The server echoes only `realtime`. The endpoint also accepts an `auth_token` query parameter. Prefer the authorization header where supported because URLs and subprotocols can appear in logs. Authentication takes precedence in this order, header, subprotocol, then query parameter.
The socket can open before authentication and provider setup finish. Listen for errors and closure immediately. Wait for `session.created`, send your session settings, then wait for `session.updated` before submitting a turn.
## Text example
Install the `ws` package and set `PUTER_TOKEN` to your Puter token. Run this example as a Node.js ES module:
```js
import WebSocket from 'ws';
const socket = new WebSocket(
'wss://api.puter.com/puterai/openai/v1/realtime?model=gpt-realtime-2.1-mini',
{ headers: { Authorization: `Bearer ${process.env.PUTER_TOKEN}` } },
);
const send = (event) => socket.send(JSON.stringify(event));
socket.on('message', (data) => {
const event = JSON.parse(data.toString());
if (event.type === 'session.created') {
send({
type: 'session.update',
session: {
type: 'realtime',
output_modalities: ['text'],
instructions: 'Keep replies short.',
},
});
}
if (event.type === 'session.updated') {
send({
type: 'conversation.item.create',
item: {
type: 'message',
role: 'user',
content: [{ type: 'input_text', text: 'Hello!' }],
},
});
send({ type: 'response.create' });
}
if (event.type === 'response.output_text.delta')
process.stdout.write(event.delta);
if (event.type === 'error')
console.error(event.error);
if (event.type === 'response.done')
socket.close();
});
socket.on('error', console.error);
socket.on('close', (code, reason) => console.log(code, reason.toString()));
```
## Events, audio, and interruption
Each message contains one JSON object with a nonempty string `type` of at most 128 characters. Binary messages are rejected. Native events retain their provider fields. The endpoint exposes the WebSocket event protocol. Your client manages capture, playback, acknowledgments, and timeouts.
PCM16 audio configured at 24 kHz mono is sent as base64 chunks in `input_audio_buffer.append` events. Output arrives in `response.output_audio.delta` events. With manual turn detection, commit buffered input with `input_audio_buffer.commit`, then request a reply with `response.create`.
To interrupt, stop local playback immediately and send `response.cancel` with the active `response_id`. If some generated audio was not played, send `conversation.item.truncate` with its `item_id`, `content_index`, and `audio_end_ms` set to the amount actually played. Wait for `response.done` and any requested `conversation.item.truncated` acknowledgment before submitting a replacement turn.
Connection loss ends the session. Reconnecting creates a new conversation. Tool execution and automatic reconnection belong to your client.
## Errors, billing, and limits
API errors arrive as `{ "type": "error", "error": { "code": "...", "message": "..." } }`. Handle socket errors and close frames too. Authentication failures use `invalid_api_key`. Other relay codes include `model_not_found`, `insufficient_funds`, `subscription_required`, `rate_limit_exceeded`, `invalid_event`, `connection_timeout`, `provider_unavailable`, `upstream_error`, and `server_restarting`.
Policy failures generally close with 1008, oversized client messages with 1009, upstream or setup failures with 1011, and server restarts with 1012. Authentication and model failures can arrive after the WebSocket handshake succeeds.
The endpoint requires a paid Puter plan. Usage is charged after completed responses and input transcriptions. Each account can open 20 sessions at once by default. Each JSON message is limited to 64 KiB in either direction, including reassembled fragments. Stream audio in small chunks. All message rates, queue caps, timeouts, and other bounds are published in [rate limits and quotas](/rate-limits-and-quotas/).
src/docs/src/rate-limits-and-quotas.md
+1 -1
View File
@@ -91,7 +91,7 @@ Gemini speech-to-text first counts the audio's tokens, which times out after 30
The OpenAI- and Anthropic-compatible endpoints (`/puterai/openai/v1/*`, `/puterai/anthropic/v1/messages`) require a paid plan; a free account gets `402 subscription_required`. The same models are available to every account through `puter.ai.*` and `/drivers/call`, and the model catalogue endpoints are open to everyone.
Realtime sessions ([`puter.ai.assistant.connect()`](/AI/assistant.connect/), `/puterai/openai/v1/realtime`) require a paid plan, like the other OpenAI-compatible endpoints, and are capped at 20 open at once per account. An account without a plan gets an `error` event with code `subscription_required`. A session over the cap gets `rate_limit_exceeded`. Both close with 1008.
Realtime sessions ([`puter.ai.convo()`](/AI/convo/), `/puterai/openai/v1/realtime`) require a paid plan, like the other OpenAI-compatible endpoints, and are capped at 20 open at once per account. An account without a plan gets an `error` event with code `subscription_required`. A session over the cap gets `rate_limit_exceeded`. Both close with 1008.
| Realtime transport limit | Value | Behavior at the limit |
| --- | --- | --- |
src/docs/src/sidebar.js
+3 -6
View File
@@ -156,12 +156,9 @@ let sidebar = [
path: '/AI/convo',
},
{
title: '<code>assistant.connect()</code>',
page_title: '<code>puter.ai.assistant.connect()</code>',
title_tag: 'puter.ai.assistant.connect()',
icon: '/assets/img/function.svg',
source: '/AI/assistant.connect.md',
path: '/AI/assistant.connect',
title: 'Realtime WebSocket API (Beta)',
source: '/AI/realtime.md',
path: '/AI/realtime',
},
{
title: '<code>speech2speech()</code>',
src/puter-js/index.d.ts
Vendored
+1 -3
View File
@@ -53,7 +53,6 @@ export type {
ConvoSessionOptions,
ConvoPlaybackPosition,
ConvoSpeakerOptions,
RealtimeConnectOptions,
RealtimeServerEvent,
Speech2SpeechOptions,
Speech2TxtOptions,
@@ -69,8 +68,7 @@ export type {
Txt2SpeechOptions,
Txt2VidOptions,
} from './types/modules/ai/types.js';
export type { Assistant, Txt2Speech } from './types/modules/ai/index.js';
export type { RealtimeSession } from './types/modules/ai/assistant.js';
export type { Txt2Speech } from './types/modules/ai/index.js';
// -- puter.apps --
export type {
src/puter-js/src/modules/ai/convo.js
+1 -1
View File
@@ -1,6 +1,6 @@
import { fetchUrl } from '../../lib/networkUtils.js';
import { promptIfUpgradeRequired } from '../../lib/upgradePrompt.js';
import { RealtimeSession, createRealtimeSocket } from './assistant.js';
import { RealtimeSession, createRealtimeSocket } from './convoTransport.js';
import { RealtimeEvents } from './realtimeEvents.js';
import { ConvoMicrophone, ConvoSpeaker } from './convoMedia.js';
src/puter-js/src/modules/ai/assistant.js → src/puter-js/src/modules/ai/convoTransport.js
+1 -60
View File
@@ -1,6 +1,5 @@
import { RealtimeEvents } from './realtimeEvents.js';
/** @typedef {import('./types.js').RealtimeConnectOptions} RealtimeConnectOptions */
/** @typedef {import('./types.js').RealtimeServerEvent} RealtimeServerEvent */
const REALTIME_PATH = '/puterai/openai/v1/realtime';
@@ -9,6 +8,7 @@ const encoder = new TextEncoder();
/**
* One live realtime session. Events in both directions are OpenAI Realtime API
* events, unchanged.
* @internal
*/
export class RealtimeSession extends RealtimeEvents {
/** @type {WebSocket} */
@@ -51,71 +51,12 @@ export class RealtimeSession extends RealtimeEvents {
});
}
/**
* Send a client event, e.g. `{ type: 'response.create' }`.
*
* @param {{ type: string } & Record<string, unknown>} event
* @returns {void}
*/
send(event) {
if (this.socket.readyState !== 1) {
throw {
message: 'Realtime session is not open',
code: 'session_closed',
};
}
this.socket.send(JSON.stringify(event));
}
/** @returns {void} */
close() {
this.socket.close(1000);
}
}
/**
* Open a realtime session against OpenAI's Realtime API through Puter. Resolves
* once the socket is open; the first event to arrive is `session.created`.
*
* @this {import('./index.js').AIModule}
* @param {RealtimeConnectOptions} [options]
* @returns {Promise<RealtimeSession>}
*/
export async function connect(options = {}) {
const { puter } = this;
if (!puter.authToken && puter.env === 'web') {
try {
await puter.ui.authenticateWithPuter();
} catch {
throw { message: 'Authentication canceled', code: 'auth_canceled' };
}
}
if (!puter.authToken) {
throw { message: 'Authentication required', code: 'auth_required' };
}
const socket = createRealtimeSocket(puter, options.model);
const session = new RealtimeSession(socket);
await new Promise((resolve, reject) => {
const onOpen = () => {
socket.removeEventListener('close', onClose);
resolve(undefined);
};
const onClose = (/** @type {CloseEvent} */ e) => {
socket.removeEventListener('open', onOpen);
reject({
message: e.reason || 'Realtime connection failed',
code: 'connection_failed',
});
};
socket.addEventListener('open', onOpen, { once: true });
socket.addEventListener('close', onClose, { once: true });
});
return session;
}
/**
* @param {import('../../index.js').Puter} puter
* @param {string} [model]
src/puter-js/src/modules/ai/assistant.test.js → src/puter-js/src/modules/ai/convoTransport.test.js
+1 -13
View File
@@ -1,5 +1,5 @@
import { describe, expect, it, vi } from 'vitest';
import { RealtimeSession } from './assistant.js';
import { RealtimeSession } from './convoTransport.js';
/** Just enough of a WebSocket to drive the session's listeners. */
const fakeSocket = () => {
@@ -52,16 +52,4 @@ describe('RealtimeSession', () => {
socket.dispatchEvent(Object.assign(new Event('close'), { code: 1008, reason: 'model_not_found' }));
expect(handler).toHaveBeenCalledWith({ type: 'close', code: 1008, reason: 'model_not_found' });
});
it('serializes sent events and refuses once closed', () => {
const socket = fakeSocket();
const session = new RealtimeSession(/** @type {any} */ (socket));
session.send({ type: 'response.create' });
expect(socket.sent).toEqual(['{"type":"response.create"}']);
socket.readyState = 3;
expect(() => session.send({ type: 'response.create' })).toThrow(
expect.objectContaining({ code: 'session_closed' }),
);
});
});
src/puter-js/src/modules/ai/index.js
-11
View File
@@ -1,5 +1,4 @@
import { PuterModule } from '../../lib/PuterModule.js';
import { connect } from './assistant.js';
import { convo } from './convo.js';
import { chat } from './chat.js';
import { txt2img } from './image.js';
@@ -20,11 +19,6 @@ import { txt2vid } from './video.js';
* }} Txt2Speech
*/
/**
* Realtime voice/text sessions over OpenAI's Realtime API.
* @typedef {{ connect: OmitThisParameter<typeof connect> }} Assistant
*/
/**
* The `puter.ai` module.
*
@@ -37,9 +31,6 @@ export class AIModule extends PuterModule {
/** @type {Txt2Speech} */
txt2speech;
/** @type {Assistant} */
assistant;
/**
* SDK-wide switch for response-format normalization. Left unset (the
* default), the release-date policy applies: models released on or
@@ -81,8 +72,6 @@ export class AIModule extends PuterModule {
methods[name] = methods[name].bind(this);
}
this.assistant = { connect: connect.bind(this) };
this.txt2speech = /** @type {Txt2Speech} */ (/** @type {unknown} */ (
Object.assign(txt2speech.bind(this), {
listEngines: listEngines.bind(this),
src/puter-js/src/modules/ai/types.js
-8
View File
@@ -559,14 +559,6 @@
* @property {boolean} [enableLogging] camelCase alias of `enable_logging`.
*/
/**
* Options for `puter.ai.assistant.connect`.
*
* @typedef {Object} RealtimeConnectOptions
* @property {string} [model] OpenAI realtime model, e.g. `gpt-realtime-2.1-mini` (the default),
* `gpt-realtime-2.1`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime` or `gpt-realtime-mini`.
*/
/**
* A server event from OpenAI's Realtime API, passed through unchanged. `type` names the event
* (`session.created`, `response.output_text.delta`, `error`, ...); the synthetic `close` event
src/puter-js/tests/api/realtime/convoLive.test.ts
+58 -3
View File
@@ -1,3 +1,4 @@
import WebSocket from 'ws';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import {
closeNodePuter,
@@ -8,7 +9,6 @@ import type {
PuterSDK,
} from '../harness/types';
import type { Conversation } from '../../../types/modules/ai/convo';
import type { RealtimeSession } from '../../../types/modules/ai/assistant';
import type { RealtimeServerEvent } from '../../../types/modules/ai/types';
import { optionalEnv } from '@heyputer/backend/drivers/integrationTestUtil.ts';
import { setupPuterTestEnv, type PuterTestEnv } from '@heyputer/backend/testUtil.ts';
@@ -19,7 +19,7 @@ const apiKey = optionalEnv('PUTER_TEST_AI_OPENAI_API_KEY');
describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
let env: PuterTestEnv;
let puter: PuterSDK;
const sessions = new Set<RealtimeSession | Conversation>();
const sessions = new Set<Conversation>();
beforeAll(async () => {
env = await setupPuterTestEnv({
@@ -41,7 +41,7 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
}, 60_000);
const waitFor = (
session: RealtimeSession | Conversation,
session: Conversation,
type: string,
): Promise<RealtimeServerEvent> =>
new Promise((resolve, reject) => {
@@ -74,6 +74,61 @@ describe.skipIf(!apiKey)('OpenAI realtime live relay', () => {
});
});
it('streams a native text turn without the SDK', async () => {
const url = new URL('/puterai/openai/v1/realtime', env.apiOrigin);
url.protocol = 'ws:';
url.searchParams.set('model', 'gpt-realtime-2.1-mini');
const socket = new WebSocket(url, {
headers: { Authorization: `Bearer ${env.users.user.token}` },
});
let text = '';
try {
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => reject(new Error('Realtime timeout')), 20_000);
const fail = (error: Error) => {
clearTimeout(timer);
reject(error);
};
socket.on('error', fail);
socket.on('close', () => fail(new Error('Realtime closed before completion')));
socket.on('message', (data) => {
const event = JSON.parse(data.toString());
const send = (payload: Record<string, unknown>) => socket.send(JSON.stringify(payload));
if (event.type === 'session.created') {
send({
type: 'session.update',
session: {
type: 'realtime',
output_modalities: ['text'],
instructions: 'Reply with exactly realtime-ok',
},
});
}
if (event.type === 'session.updated') {
send({
type: 'conversation.item.create',
item: {
type: 'message', role: 'user',
content: [{ type: 'input_text', text: 'Hello!' }],
},
});
send({ type: 'response.create' });
}
if (event.type === 'response.output_text.delta') text += event.delta;
if (event.type === 'error') fail(new Error(`Realtime error: ${event.error?.code}`));
if (event.type === 'response.done') {
clearTimeout(timer);
if (event.response?.status !== 'completed') fail(new Error('Realtime response incomplete'));
else resolve();
}
});
});
expect(text).toBe('realtime-ok');
} finally {
socket.close();
}
}, 30_000);
it.each([undefined, 'gpt-realtime-mini'])(
'streams a text reply through the SDK and records usage (model %s)',
async (model) => {
src/puter-js/tests/api/suites/ai.suite.ts
+4 -19
View File
@@ -1779,24 +1779,9 @@ export default suite('ai', {
}
},
// Keyless: the relay rejects an unknown model before dialing OpenAI, so
// this covers the SDK socket, the subprotocol credential and the
// error-event shape without an upstream.
'assistant.connect surfaces relay errors as OpenAI error events': async (
t,
) => {
const session = await t.puter.ai.assistant.connect({
model: 'not-a-realtime-model',
});
const [error, closed] = await Promise.all([
new Promise<{ error?: { code?: string } }>((resolve) =>
session.on('error', resolve),
),
new Promise<{ code?: unknown }>((resolve) =>
session.on('close', resolve),
),
]);
t.assert.equal(error.error?.code, 'model_not_found');
t.assert.equal(closed.code, 1008);
'convo is the realtime SDK entry point': async (t) => {
t.assert.equal('assistant' in t.puter.ai, false);
t.assert.equal('conversation' in t.puter.ai, false);
t.assert.equal(typeof t.puter.ai.convo, 'function');
},
});