The per-commit cap of 1,280,000 bytes assumed G.711, so a PCM16 session
(48,000 bytes/s, the SDK's format) with transcription on was closed with
1013 after about 27 seconds without a commit: silence under server VAD,
one long utterance, or a long assistant reply.
Unacknowledged audio is now a duration at the provider-confirmed input
format (the lower rate while a requested format is unconfirmed). Without
turn detection it counts all audio since the last commit or clear; with
server VAD only speech from speech_started, plus the buffer after an
explicit commit, since the provider drops silence. Each started 160 s
claims one spare item as its hold instead of closing the session; a
commit, clear or input_audio_buffer_commit_empty returns the holds to the
pool. If the upstream closes first, every claimed item is billed. The
settle no longer commits silence.
The relay test file now expires the in-memory per-IP pre-auth window
after each test; every test connects from loopback and the added tests
pushed the file past 120 upgrades a minute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Input transcription reserved 64 items up front at the costliest model's
price and largest cost factor, about $7 before factors per session. Ordinary
subscribers couldn't enable it, and concurrent sessions were refused while
balance was left.
Each item now holds its own credit at the selected transcription model's
budget and cost factor. Two spare items are reserved before configuration or
audio is forwarded and refilled as the provider commits items, within the
64-item pending cap. A commit with no spare reserves its own item. A model
change releases the spares and reserves new ones at the new price; committed
items keep their hold. When the account can't cover another item, client
reads pause while the session's own holds may still free credit, and the
session is refused with insufficient_funds once it holds none.
Audio awaiting a commit acknowledgement is now tracked separately from pool
capacity and reset by each acknowledged commit or clear. If the upstream
closes before that acknowledgement, the audio bills as one item. Previously
the fallback required an untouched window, so a lost acknowledgement after an
earlier completed item went unbilled; the cumulative byte count could not
simply be reused without charging earlier work twice. The uncommitted-audio
cap is now 160 seconds of audio instead of a per-window total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`reserveAiCredits` checked the balance and then took an unconditional
hold, so concurrent requests could all pass the check before any hold
was visible; two 16.2M µ¢ holds were admitted against a 20M µ¢ balance.
The hold store now takes a hold only if it fits beside the live holds,
in one Lua script on the actor's hold key. A per-user version, bumped
when usage is recorded and when a hold is released, lets the script
refuse a balance read that a settlement has since overtaken; the
service reads the balance again and retries, and skips the version
check on its last attempt rather than refuse an account whose balance
keeps moving. Both keys share the actor's hash tag, so the script runs
on Redis Cluster. The balance read for this check reads purchased
credit consistently.
`reserveCreditsIfAffordable` carries this through the service, and the
AI facade calls it once with the cost factor applied. `reserveCredits`
keeps its unconditional semantics. When the cache is unreachable, the
balance alone decides and nothing is held, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A plan or funds refusal was classified by a plan lookup made after the
decision. A fallback plan is cached for only a few seconds, so when it
ran out in between, the refusal was judged against the recovered plan
and still ended the session with a terminal 1008. The plan is now also
checked just before each decision, and a refusal is retryable if either
check found a fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the billing lookup failed, metering answered with a provisional
fallback plan, so the relay refused the session with
`subscription_required` or, against the fallback's allowance,
`insufficient_funds`, both terminal 1008 closes, and the SDK showed the
upgrade prompt to a paying account. A plan or funds refusal decided on a
provisional plan now sends `internal_error` and closes with 1011, at
connection and mid-session, so the client retries once the lookup
recovers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A credential lookup that threw (a database or cache outage) was treated
as no actor, so the session got `invalid_api_key` and a 1008 close. That
is terminal for the SDK, which then ended an established conversation
that only needed to reconnect. A failed lookup now gets `internal_error`
and closes with 1011, which the SDK retries; missing, invalid and
revoked credentials keep `invalid_api_key` and 1008. Nothing is cached
for the failed lookup, and the cause is logged server-side only.
A session whose usage records pile up behind metering now closes with
1013 instead of 1008, still with `rate_limit_exceeded`. Queued usage is
still recorded and its holds released. An unexpected setup failure now
sends `internal_error` before its 1011 close.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A session refused while the server drains returned before its client
error and close listeners were registered, so a frame that errored the
socket during the close handshake (an oversized one, for example)
emitted an unhandled `error` and ended the process. The listeners now
go on as soon as the session exists, ahead of every refusal.
The 404 answer to an upgrade for another host also ends a raw socket
that has no error listener once Node hands it over; it now has one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the upstream realtime connection closes, for any reason, with a
started response that never sent response.done, the relay now records that
response at its credit hold (8k audio in + 4,096 audio out at the session
model's rates, through the AI cost factor) instead of releasing the hold
uncharged. The hold is released once the charge is recorded, and a warning
alarm is still raised.
Client conversation.item.retrieve events and MCP tools in session.update or
response.create are refused with an invalid_event error and a 1008 close.
* test(ai): end every realtime relay test's upstream before the next
The fake upstream now answers response.create with response.created first,
as OpenAI does, so closing those sessions no longer waits out the settle
window.
* feat(ai): ask realtime clients to reconnect before a restart
Open realtime sessions get a puter.reconnect event when the drain starts,
with close_in_ms set to the drain window, and new sessions are still refused
until the 1012 close at shutdown. A host extension can announce a restart
early by emitting serverRestartAnnounced; that only sends the notice, so a
false alarm leaves the server working normally. A session gets the notice at
most once a minute.
onServerDrainStart now receives the drain duration.
* fix(ai): retry realtime reconnects for about a minute and wait out offline browsers
Five attempts gave up after about 15 seconds, shorter than a server restart.
The conversation makes ten attempts with the backoff capped at 10 seconds,
and while the browser reports being offline it waits for the online event,
up to two minutes per reconnect, instead of spending attempts. close() ends
either wait and removes its listener.
* feat(ai): restore recent conversation context after a realtime reconnect
The provider starts a fresh session on every reconnect, so the conversation
lost all earlier items. The SDK now records finished items it can recreate
from the provider's own events: user text, user speech with a transcript,
assistant replies as text, and function calls with their outputs. After the
new session acknowledges its settings, and before ready fires, it adds back
the newest of them, up to 40 items and 48 KiB, which stays under the relay's
upload burst.
Assistant replies are cut to what the user saw or heard: the deltas shown
before an interruption, and the share of the transcript matching the audio
played before a truncation or a drop. Deleted items, unfinished replies,
images, speech without a transcript and incomplete function calls are left
out, as are function outputs whose call falls outside the window. Restored
copies take over their new item IDs. A refused item is reported once as
context_restore_failed without failing the reconnect.
* feat(ai): move a realtime conversation to a new session on the restart notice
On puter.reconnect the conversation waits until it is idle: no reply
requested, generating or playing, no user turn being spoken or transcribed,
no function call awaiting its output and no acknowledgment pending. Then it
reconnects through the usual path, emitting reconnecting (code 1012) and
ready with reconnected: true, without an error. Until then the session keeps
working, and if the server closes it first the normal 1012 reconnect takes
over. close() at any point stops the wait.
* docs(ai): realtime restart notice, context restore and reconnect limits
* test(ai): realtime reconnects against the local server
The mock provider answers with item events and whole replies like OpenAI.
New cases cover a dropped connection (cut through a TCP proxy so the client
sees 1006), a host-announced restart, and a draining server refusing the
reconnect; the first two check the provider receives the restored items
before the next turn. The browser media test waits for captured audio and
the reply instead of a fixed sleep. The live suite gains a tool and text
context round trip across a restart notice.
* ci: run the live realtime conversation tests with the AI provider suite
They need the OpenAI key, which only this workflow has, and skip themselves
without it.
* fix(ai): open the replacement before leaving a realtime session on the restart notice
The handover closed the current session and then reconnected, so every
notice cost a second or two in which sends failed and microphone audio was
dropped. On a single server that refuses new sessions while it drains, the
retries ran out during the drain and the conversation closed sooner than if
it had stayed. The conversation now opens a replacement session as soon as
the notice arrives and switches only once it is open and the conversation is
idle. A refused replacement is retried with backoff while the current
session keeps working; when the server closes it first, the held replacement
is used right away.
The idle check also missed some in-progress work:
- a reply created while the user was still speaking ended the user's turn;
speech now lasts until its audio is committed or cleared, and a committed
turn waits for the reply server turn detection creates
- only the newest function call was checked; every finished call since the
last user message needs its output, and calls cut off by a cancellation
don't count
- native session.update and conversation.item.create events the provider
hadn't acknowledged could be lost; the switch waits for them, up to the
acknowledgment timeout for items
- only the newest transcription was waited on; every committed item is
- clearing playback didn't recheck the switch
Restored items stay internal: the new session's copies of them no longer
reach listeners. At a reconnect, every queued item the user didn't hear is
cut, not just the one playing. A notice from a failed reconnect attempt is
dropped.
* test(ai): realtime replay after split settings, and a draining server refusing the replacement
The relay accepts the context replay right after two full-size settings
updates. Against the local server, a draining relay refuses the replacement
session while the current one keeps answering.
* ci: build the SDK in the live realtime step so it stands on its own
* fix(ai): a failed realtime reconnect attempt no longer poisons the next
An attempt that dropped after sending its session.update but before the
acknowledgment left that event ID queued, so the next attempt's
acknowledgment was matched to it and timed out, and so on until the
conversation closed. Turn state is now reset after each failed attempt.
* fix(ai): bound the realtime handover's waits and stop retries that can't succeed
- The first replacement opens within a randomized 3 seconds of the notice,
so sessions notified together don't all open at once.
- A replacement refused with a code retrying can't fix, such as
insufficient_funds, ends the attempts; the server's close still falls back
to the usual reconnect.
- A held replacement that gets an error or its own server's notice is closed
and replaced.
- A held replacement used after a drop gets a 3 second acknowledgment
timeout, since its socket may have died with the network.
- close() during the switch closes the held replacement too.
- The reply to a committed turn and each transcription hold the switch for
at most 10 and 30 seconds, like app items already did, and an app item is
released only by an added item of its type, role and call ID.
* docs(ai): the realtime restart notice isn't guaranteed before every restart
* feat(ai): relay OpenAI's Realtime API as puter.ai.assistant
Adds a WebSocket relay at /puterai/openai/v1/realtime on the api host,
one upstream OpenAI connection per client, frames passed through 1:1.
The client socket is accepted before auth so socket.io's 1 s
unhandled-upgrade timeout can't kill slow handshakes; failures arrive as
OpenAI-shaped error events and a 1008 close. Usage is metered per
response.done at the model's token rates, sessions are capped per user
(realtime.maxConnectionsPerUser), and each node logs counters,
event-loop delay and memory every 30 s.
puter.js gets puter.ai.assistant.connect(), returning a session with
send/on/off/close over the same OpenAI events.
realtime.upstreamUrl points the relay at a mock for load tests;
tools/realtime-stress has the mock upstream, a load generator with a
JSON report, and an agent runbook. The relay README lists its quirks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ai): backpressure and upload budget on the realtime relay
Neither direction applied backpressure: a client that stopped reading
grew a queue up to 64 MiB, and a client streaming 16 MiB frames was read
and forwarded as fast as it could send. A few such sockets pinned the
event loop and pushed RSS past 1 GB.
Each side now stops reading the other once 1 MiB is queued toward it,
and a client that keeps the upstream paused for 30 s is closed. Client
uploads are budgeted per session (1 MiB/s sustained, 16 MiB burst,
realtime.maxClientBytesPerSecond), pre-open buffering pauses reads at
1 MiB instead of rejecting at 32 MiB, and clients get a 30 s ping so
half-open sockets release their upstream and slot. Upstream frames use
a zero mask and skip UTF-8 validation, which the relay never needed.
The stats line gains elu. tools/realtime-stress gains abuse.mjs and a
runbook phase for event-loop abuse.
* fix(ai): drain, transcription metering and credential re-check on the realtime relay
Shutdown now drains: a new onServerDrainStart hook (run when SIGTERM
starts the drain, before the listener closes) makes the relay refuse new
sessions with 1012, open ones keep running through the drain window, and
at onServerPrepareShutdown each gets up to 5 s to finish an in-flight
response before closing with 1012 so the client reconnects elsewhere.
Input transcription is metered. The relay reads the type from each text
frame's first 96 bytes and, after forwarding, parses response.done,
input_audio_transcription.completed and session.created/updated (for the
transcription model) on setImmediate, so metering never holds up frames.
Unknown transcription models bill at the highest known rate.
Credentials are re-checked: a successful auth is cached in the kv.js
singleton for 60 s under a hash of the token, and a 15 s sweep
re-authenticates sessions whose entry expired, closing revoked or
suspended ones with invalid_api_key. Lookup errors keep the session and
retry on the next sweep.
* docs(ai): API shape proposal for puter.ai.conversation() across OpenAI, Gemini Live and xAI
* feat(ai): add composable realtime conversations with bounded transport
* fix(ai): realtime billing, subscription gate and conversation reconnects
Relay:
- Cancel and bill a response still generating when the client leaves,
instead of cutting the upstream before response.done.
- Record realtime usage through the AI cost-factor hook.
- Drop the upstream byte and message caps; replies arrive faster than real
time and tripping them closed sessions and skipped billing.
- Price usage as it arrives, before any forwarding decision, so a closing
session never drops a usage event.
- Bill the model the session reports and refuse a session.update that
changes the model.
- Require a paid plan.
SDK (puter.ai.convo):
- Reconnect after a dropped connection with the same settings, keeping
listeners and media; emits reconnecting and ready { reconnected }.
- Errors reject only the operation they name; a failed send rejects only
its own request.
- configure() sends only the given settings and settles on its own ack;
commitAudio follows turn detection set natively.
- interrupt() right after sendText cancels the reply once it starts.
- Microphone: capture at the device rate, drop chunks instead of stopping
on stalls or a congested socket, no timeout on the permission prompt,
permission_denied on refusal.
- Speaker: 300 s playback bound without interrupting, played position
accounts for output latency and gaps between chunks.
- Audio control events keep headroom over queued audio.
- Upgrade prompt on subscription_required / insufficient_funds.
Tests that need the built SDK and Chromium move to the puter.js suite.
* chore: remove stray node_modules symlinks
* feat(ai): retire assistant API and document realtime beta
* test(ai): wait for the matching realtime billing batch
* test(kv): exclude health probes from warm read measurements
* Revert "test(kv): exclude health probes from warm read measurements"
This reverts commit 631003f755.
* fix(ai): upstream message size, null usage, team 2FA and reconnect settings for realtime
Relay:
- Upstream messages may be up to 4 MiB; client messages stay at 64 KiB.
An OpenAI event above 64 KiB (an echoed item near the client limit, a
long reply, a retrieved audio item) closed the upstream before in-flight
responses reported usage.
- An upstream that closes with a response still in flight raises a
warning alarm.
- A null usage counts as none instead of closing the session.
- Seats whose team requires 2FA need it for realtime too, at connect and
on the credential recheck.
SDK (puter.ai.convo):
- A reconnect reapplies every acknowledged setting, native session.update
events included, split to fit the message limit.
- connect() during a reconnect settles with it.
- Reconnect delays are jittered, and closing cancels a pending delay.
The credential-recheck and browser worklet tests now fail when the
behavior they cover breaks.
* feat(ai): credit holds, attempt limits and late-response billing for realtime (#4407)
Relay:
- Each response holds 8,000 audio input and 4,096 audio output tokens of
the account's budget, at the session model's rates, from
response.created until its usage is recorded. A session opens only if
one hold fits.
- Session attempts are limited to 60 per minute per credential, checked
before authentication.
- A response the client requested that starts after the client leaves is
cancelled and billed.
- Input transcription is priced at the costlier of the transcription
models in force at commit and at completion.
- rate_limits.updated isn't forwarded, and provider organization IDs are
removed from error messages.
SDK (puter.ai.convo):
- microphone.stop() during the permission prompt rejects the pending
start() with session_closed.
- speaker.start() rejects with { code, message } for browser exceptions.
- connect() waits 20 seconds for the session, past the server's 15-second
setup.
- Convenience events fill IDs the provider leaves out; types cover
turn.done, error, and client events for send().
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Daniel Salazar <daniel.salazar@puter.com>
404oops