mirror of
https://github.com/HeyPuter/puter.git
synced 2026-11-04 04:53:39 +00:00
c7c3f21e4b
* feat(ai): relay OpenAI's Realtime API as puter.ai.assistant
Adds a WebSocket relay at /puterai/openai/v1/realtime on the api host,
one upstream OpenAI connection per client, frames passed through 1:1.
The client socket is accepted before auth so socket.io's 1 s
unhandled-upgrade timeout can't kill slow handshakes; failures arrive as
OpenAI-shaped error events and a 1008 close. Usage is metered per
response.done at the model's token rates, sessions are capped per user
(realtime.maxConnectionsPerUser), and each node logs counters,
event-loop delay and memory every 30 s.
puter.js gets puter.ai.assistant.connect(), returning a session with
send/on/off/close over the same OpenAI events.
realtime.upstreamUrl points the relay at a mock for load tests;
tools/realtime-stress has the mock upstream, a load generator with a
JSON report, and an agent runbook. The relay README lists its quirks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ai): backpressure and upload budget on the realtime relay
Neither direction applied backpressure: a client that stopped reading
grew a queue up to 64 MiB, and a client streaming 16 MiB frames was read
and forwarded as fast as it could send. A few such sockets pinned the
event loop and pushed RSS past 1 GB.
Each side now stops reading the other once 1 MiB is queued toward it,
and a client that keeps the upstream paused for 30 s is closed. Client
uploads are budgeted per session (1 MiB/s sustained, 16 MiB burst,
realtime.maxClientBytesPerSecond), pre-open buffering pauses reads at
1 MiB instead of rejecting at 32 MiB, and clients get a 30 s ping so
half-open sockets release their upstream and slot. Upstream frames use
a zero mask and skip UTF-8 validation, which the relay never needed.
The stats line gains elu. tools/realtime-stress gains abuse.mjs and a
runbook phase for event-loop abuse.
* fix(ai): drain, transcription metering and credential re-check on the realtime relay
Shutdown now drains: a new onServerDrainStart hook (run when SIGTERM
starts the drain, before the listener closes) makes the relay refuse new
sessions with 1012, open ones keep running through the drain window, and
at onServerPrepareShutdown each gets up to 5 s to finish an in-flight
response before closing with 1012 so the client reconnects elsewhere.
Input transcription is metered. The relay reads the type from each text
frame's first 96 bytes and, after forwarding, parses response.done,
input_audio_transcription.completed and session.created/updated (for the
transcription model) on setImmediate, so metering never holds up frames.
Unknown transcription models bill at the highest known rate.
Credentials are re-checked: a successful auth is cached in the kv.js
singleton for 60 s under a hash of the token, and a 15 s sweep
re-authenticates sessions whose entry expired, closing revoked or
suspended ones with invalid_api_key. Lookup errors keep the session and
retry on the next sweep.
* docs(ai): API shape proposal for puter.ai.conversation() across OpenAI, Gemini Live and xAI
* feat(ai): add composable realtime conversations with bounded transport
* fix(ai): realtime billing, subscription gate and conversation reconnects
Relay:
- Cancel and bill a response still generating when the client leaves,
instead of cutting the upstream before response.done.
- Record realtime usage through the AI cost-factor hook.
- Drop the upstream byte and message caps; replies arrive faster than real
time and tripping them closed sessions and skipped billing.
- Price usage as it arrives, before any forwarding decision, so a closing
session never drops a usage event.
- Bill the model the session reports and refuse a session.update that
changes the model.
- Require a paid plan.
SDK (puter.ai.convo):
- Reconnect after a dropped connection with the same settings, keeping
listeners and media; emits reconnecting and ready { reconnected }.
- Errors reject only the operation they name; a failed send rejects only
its own request.
- configure() sends only the given settings and settles on its own ack;
commitAudio follows turn detection set natively.
- interrupt() right after sendText cancels the reply once it starts.
- Microphone: capture at the device rate, drop chunks instead of stopping
on stalls or a congested socket, no timeout on the permission prompt,
permission_denied on refusal.
- Speaker: 300 s playback bound without interrupting, played position
accounts for output latency and gaps between chunks.
- Audio control events keep headroom over queued audio.
- Upgrade prompt on subscription_required / insufficient_funds.
Tests that need the built SDK and Chromium move to the puter.js suite.
* chore: remove stray node_modules symlinks
* feat(ai): retire assistant API and document realtime beta
* test(ai): wait for the matching realtime billing batch
* test(kv): exclude health probes from warm read measurements
* Revert "test(kv): exclude health probes from warm read measurements"
This reverts commit 631003f755.
* fix(ai): upstream message size, null usage, team 2FA and reconnect settings for realtime
Relay:
- Upstream messages may be up to 4 MiB; client messages stay at 64 KiB.
An OpenAI event above 64 KiB (an echoed item near the client limit, a
long reply, a retrieved audio item) closed the upstream before in-flight
responses reported usage.
- An upstream that closes with a response still in flight raises a
warning alarm.
- A null usage counts as none instead of closing the session.
- Seats whose team requires 2FA need it for realtime too, at connect and
on the credential recheck.
SDK (puter.ai.convo):
- A reconnect reapplies every acknowledged setting, native session.update
events included, split to fit the message limit.
- connect() during a reconnect settles with it.
- Reconnect delays are jittered, and closing cancels a pending delay.
The credential-recheck and browser worklet tests now fail when the
behavior they cover breaks.
* feat(ai): credit holds, attempt limits and late-response billing for realtime (#4407)
Relay:
- Each response holds 8,000 audio input and 4,096 audio output tokens of
the account's budget, at the session model's rates, from
response.created until its usage is recorded. A session opens only if
one hold fits.
- Session attempts are limited to 60 per minute per credential, checked
before authentication.
- A response the client requested that starts after the client leaves is
cancelled and billed.
- Input transcription is priced at the costlier of the transcription
models in force at commit and at completion.
- rate_limits.updated isn't forwarded, and provider organization IDs are
removed from error messages.
SDK (puter.ai.convo):
- microphone.stop() during the permission prompt rejects the pending
start() with session_closed.
- speaker.start() rejects with { code, message } for browser exceptions.
- connect() waits 20 seconds for the session, past the server's 15-second
setup.
- Convenience events fill IDs the provider leaves out; types cover
turn.done, error, and client events for send().
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Daniel Salazar <daniel.salazar@puter.com>
The Open-Source Internet Computer!
« LIVE DEMO »
Puter.com
·
App Store
·
AI Builder
·
Developers
·
X
Puter
Puter is an advanced, open-source, self-hostable internet computer designed to be feature-rich, fast, and highly extensible.
For Users
Puter's goal is to provide you with every app and feature you need to work, create, and play under one roof. From a simple Notepad and Voice Recorder to Spreadsheet and Camera, Puter wants to be the all-in-one solution for your digital life.
For Developers
Puter provides everything you need to build and publish web apps and games. From AI to Cloud Storage and Database to Serverless Workers, Puter has you covered. Puter also helps you get users! Once you build your app, you can publish it on our App Store to reach and monetize users.
Getting Started
💻 Local Development
git clone https://github.com/HeyPuter/puter
cd puter
npm install
npm start
→ This should launch Puter at http://puter.localhost:4100
To run this checkout with Docker, follow Building from source. Create a local docker-compose.override.yml to select the local build; keeping these settings out of docker-compose.yml avoids conflicts when pulling updates and keeps local configuration out of pull requests.
🚀 Self-Hosting
Linux/macOS
curl -fsSL https://puter.com/selfhost | sh
Windows
irm https://puter.com/selfhost?os=windows | iex
→ For more details, see Self-Hosting Puter.
☁️ Puter.com
Puter is available as a hosted service at puter.com.
Support
Connect with the maintainers and community through these channels:
- Bug report or feature request? Please open an issue.
- X (Twitter): x.com/HeyPuter
- Security issues or abuse reports? security@puter.com
- Email maintainers at hi@puter.com
We are always happy to help you with any questions you may have. Don't hesitate to ask!
License
This repository, including all its contents, sub-projects, modules, and components, is licensed under AGPL-3.0 unless explicitly stated otherwise. Third-party libraries included in this repository may be subject to their own licenses.
Translations
- Arabic / العربية
- Armenian / Հայերեն
- Bengali / বাংলা
- Chinese / 中文
- Danish / Dansk
- English
- Farsi / فارسی
- Finnish / Suomi
- French / Français
- German / Deutsch
- Hebrew/ עברית
- Hindi / हिंदी
- Hungarian / Magyar
- Indonesian / Bahasa Indonesia
- Italian / Italiano
- Japanese / 日本語
- Korean / 한국어
- Malay / Bahasa Malaysia
- Malayalam / മലയാളം
- Dutch / Nederlands
- Polish / Polski
- Odia / ଓଡ଼ିଆ
- Portuguese / Português
- Punjabi / ਪੰਜਾਬੀ
- Romanian / Română
- Russian / Русский
- Spanish / Español
- Swedish / Svenska
- Tamil / தமிழ்
- Telugu / తెలుగు
- Thai / ไทย
- Turkish / Türkçe
- Ukrainian / Українська
- Urdu / اردو
- Vietnamese / Tiếng Việt