213 Commits

Include renames

Author	SHA1	Message	Date
Daniel Salazar	d3869b7846	feat(notifications): dismissal across devices, high priority, buttons, schedule updates, docs fixes (#4501) ... - Docs: status(), browser limits, events worker fetch/ack, update() across pages, stored-notification retries, device removal, push delivery, queue coalescing, existing service workers and missing error codes match the code. - puter.notifications.markShown(uid) / dismiss(uid). - Dismissals reach the app's live feed (on('dismiss')); pages close the notification on screen, and a later page load closes ones already dismissed. The desktop closes its own copies on notif.ack. - Live notification postings are forwarded to sessions in other regions. - notify({ priority: 'high' }). - onPush() results accept requireInteraction and actions; clicks carry action. - replaced reaches service-worker onPush() handlers. - puter.events.updateSchedule(scheduleId, changes), with the invoker re-checking a schedule run's guard right before it is sent.	2026-11-05 00:17:27 -07:00
Daniel Salazar	48cd4ea355	fix: oidc toggle (#4494)	2026-11-04 13:13:15 -07:00
Juan Fernando Castro	a9a8df6409	feat(events): fetch can start at the newest notification, and the feed uses it (#4479) ... * feat(events): fetch can start from the newest event GET /events/fetch takes from=end: with no cursor it answers an empty page whose cursor sits at the newest row of the slice, so later reads return only rows created or replaced after it. puter.events.fetch() passes `from` through. Adds a (user_id, audience, app_uid, id) index on notification so the newest row of a slice is one seek. * fix(puter-js): onPush starts its feed from the newest notification A notifications listener started without catch-up asks for from: 'end' and keeps the cursor it gets back instead of reading every stored notification to mark them as seen. A server that ignores `from` answers with the oldest page, and the listener reads through it as before. * fix(events): read the newest notification id from the primary * fix(events): mark the end cursor at the slice's latest replacement The end cursor's replacement mark is the newest (replaced_at, id) in the slice rather than the serving node's clock, since replaced_at is stamped by whichever node wrote it. * docs(events): describe the end cursor's replacement mark	2026-11-04 10:15:14 -05:00
Daniel Salazar	23782dec12	feat(events): inline handlers in the user's own scope for onSchedule and onPersistent (#4487) ... * feat(events): schema for user-scope handlers Add event_user_handlers and event_subscriptions.handler_scope in all three dialects. event_handlers is unchanged. * feat(events): scope the handler and durable subscription stores Handler store methods take a scope (app, or a user's scope of an app) and read the matching table. App-scope behavior, SQL and cache keys are unchanged; user scopes get their own cache keys, a 20-handler cap, and the inline @<sha256> name. Adds user-scope listing, counting, deletion and unbound-handler reads. Durable subscriptions store and return handlerScope, and listByHandler filters by scope and holder. * fix(events): hold the user-scope handler cap under concurrency and drop all scope caches on app delete A create that lands past the cap removes itself and is refused. deleteForApp drops each holder's cached set in batches, here and on peers. Batch scans over user scopes and unbound handlers take limit and cursor. * feat(events): set a CPU limit on events worker deploys * test(events): cover redeploy and resend of an evicted events script * test(events): name the deploy-on-miss test for what it checks * fix(events): keep a deploy-on-miss call inside the delivery lease * feat(workers): cap rented units per plan and refuse only new ones * feat(events): publish handlers into a user scope, with inline create * test(events): cover user-scope publish, inline create and listing * test(workers): name the credentials gate without a vendor * feat(events): run user-scope handlers as the holder A user scope deploys one script per handler under the holder's own identity, throttled per scope, with an eager deploy after publish that skips an upload another holder already made. Delivery and schedule runs address the holder's handler, carry the holder on the invoke headers, batch calls, the miss handler and the rehydrate route, and stay owed while the flag is off. Suspensions of a user scope notify the holder, teardown (app delete, consent revoke, account delete) drops the scope, and a job deletes unbound handlers after a day. * test(events): cover user-scope deploy, address, notices, teardown and the sweep Runs an inline user-scope handler through the local worker runtime as its holder, and covers the holder on the wire, the flag-off path, revoke and account-delete teardown, and the unbound-handler sweep. * test(events): a lagging replica read does not suspend a user-scope schedule run * feat(puter-js): inline user-scope handlers and scope option for handlers and workers * docs(events): document inline and user-scope handlers, with playground examples Cover inline handlers on onSchedule and onPersistent, handlerScope, the scope option on handlers and workers, the user-scope limits and rented units, and the new error codes. Add six playground examples and pin the new limit constants in limits.test.ts. * feat(schedules): carry the handler scope on the schedule view * docs(events): describe user-handler cleanup, schedule handlerScope and suspension notices * fix(events): gate the rented-unit cap on user handlers and let an account session destroy its own user scope * fix(events): keep user-scope rows off app handlers across mixed versions and bound their deploys and reads Cached durable rows carry an explicit scope marker; an unmarked entry with a handler is resolved from the table and rewritten. User-scope rows are parked before credit checks, billing, client delivery and suspension while the flag is off. Delivery-time handler checks and addressing share the replica cache entry. Concurrent eager deploys of one user-scope script upload once. * fix(events): keep a user scope's age across handler churn for rent event_user_handlers.scope_since records when the scope first held a handler. A new row takes the scope's existing minimum, so replacing code does not make the scope look newer to the monthly sweep. The count uses it instead of the oldest remaining handler's created_at. * fix(events): drop cached entries no row backs; document account-session scope destroy * test(jobs): list the user-handler sweep among the system jobs * fix(events): let the user-handlers migration apply on its own * fix(events): fail a cached-row read whose scope lookup fails instead of returning app-scope rows * fix(events): touch a reused user-scope handler before binding so the sweep cannot take it * fix(events): bound the deploy-on-miss wait by the call's deadline	2026-11-03 23:26:04 -07:00
Daniel Salazar	d934a47c5e	feat: jobs and schedules (#4467) ... * feat(db): add nowMsExpr, the database clock in epoch milliseconds, and share the retry backoff * feat(config): type the jobs config as an augmentable IJobsConfig and resolve its namespace in one place * feat(jobs): add JobScheduler for fleet- and region-wide system jobs * refactor(jobs): run core's periodic sweeps as system jobs, and publish the generation bumps of reaped subscriptions * feat(events): seams for schedules run by an extension, with events_disabled where none is installed * feat(notifications): add notification.send_key, the deferred mailbox write and notify({ sendAt }) through the schedules contract * feat(puter.js): schedules, notify({ sendAt }) and notifications.cancel(), and treat a 503 events_disabled as final * docs(events): document schedules, sendAt and cancel(), with recipes * fix(events): stop resuming a holder's later pages once the credit sweep's signal aborts, and keep its checkpoint * fix(events): pre-release fixes for jobs, schedule seams and delayed sends - The delayed-send write is a field on the notifications driver, not a method, so /drivers/call can't reach it. - A delayed send checks for its own earlier write by uid, so it takes no gap lock that every other notification insert waits on. - The notification send_key index builds in place without locking writes. - Reaps re-check expiry and suspension on the primary, so a row resumed since the replica read is kept. - Revocation settles skip schedule rows they can't settle. - An empty handler set read from a replica is re-read on the primary. - One ended notice per settle, not one per page. - Sweeps that run once per fleet take more batches per run. - The global usage rate check keeps its 14-minute minimum window. - shutdown() waits for prepare hooks a drain already started. * fix(puter.js): quota refusals are final, and schedule error codes match the server Schedule, subscription and handler quota 429s no longer go through the generic retry. ScheduleErrorCode lists what the schedule methods throw, and the notify({ sendAt }) overload has type tests. * docs(events): schedule and delayed-send accuracy and style pass Spread and send-time wording match the runner, the per-user runs-at-once and rate-limited stored notification limits are published, and the new pages follow the docs writing rules. * docs(events): add the Change Data at a Set Time recipe A one-time schedule that changes data when it runs, cancelling and moving it, and a repeat-a-few-times variant with maxRuns. * fix(events): delayed sends recheck the app's access when they fire, and a handler set missing the run's handler is re-read on the primary Also stops job polling at drain start and marks replayable notification sends. * feat(events): batch handler invocations to the events dispatcher Invocations that come due within a few milliseconds go out as one /invoke-batch request (up to events.invokeBatchSize, default 60). The dispatcher runs a few at a time and streams a started line and an answer line per call, so a slow handler delays no other call's answer. A call's timeout runs from its started line, so time queued behind other calls is never the handler's; queue time comes out of the connection-wait share instead, and a call still queued past it is never started and fails as Puter's. A lone call still uses /invoke, and a dispatcher without the batch route gets single calls for five minutes before the batch route is tried again.	2026-11-02 23:48:35 -07:00
Daniel Salazar	18fd655645	fix: sec improvements (#4469)	2026-11-02 12:11:38 -07:00
Daniel Salazar	5d34e803ff	fix(events): delivery path fixes for queued deliveries, handler failures and per-invocation database load (#4443) ... * fix(events): queue every row's single delivery before any handler runs A dispatch queued each matched single row's event and then waited on that row's hand-over, which can be a whole handler run, before queueing the next row's. An event matching several rows queued the later ones only after the earlier handlers finished, and lost them if the process stopped in between. A batched filesystem write did the same across its entries. The event is now queued in line and the hand-over runs detached; anything it doesn't finish is retried by the pending sweep. A hand-over that throws no longer raises the 'could not be queued and is lost' alarm, since the event is still queued. * perf(events): address an app's events worker from a cached handler set Every handler invocation read the app's handler set (names and hashes) from the primary database to work out which script to call. The delivery path now reads it through a Redis cache. Publishing, updating or removing a handler, and deleting the app's handlers, drop the entry on this node and every peer, so only a read that raced one of those writes can address the superseded script, for at most 30 seconds. An empty set isn't cached. Deploys and the events-worker listing still read the primary. * perf(events): skip leased queues without a database read, and sweep in parallel The pending sweep read each queue's subscription row from the primary database before finding out whether anything in the queue could be claimed. A queue whose head is in flight or waiting out a retry now goes to the back of the line on a Redis check alone; its row is read once the lease lapses. The backstop re-check of a queued row's access waits for that lapse too, and nothing is handed out in the meantime. The pass also worked through its subscriptions one at a time, so one slow handler run held up every queue behind it. It now works on up to ten at once. * fix(events): don't count a wait for a dispatcher connection as a handler timeout Invocations share one pool of connections to the events dispatcher per process, and the invoke timeout started when a call was made, not when it got a connection. With the pool busy, a call could spend its timeout waiting, be reported as a handler that ran past it, and count toward suspending a subscription whose handler never ran. A call now waits for a free connection before its timeout starts. A call that waits more than half the timeout for one fails as Puter's (unavailable): retried on the usual schedule and not counted, as the documented failure rules already say. Wait and run together stay inside the worker lease. * fix(events): end a background row whose consent a delivery finds withdrawn Withdrawing an app's events:background consent is settled by a listener on the node that handled the revoke. If that settle never ran or failed, the row stayed in service: every delivery found the consent gone, minted no token and was deferred, so the backlog was retried forever and the row was never suspended or reported. When the delivery path's consent check answers no, it now runs the same revocation settle (permission_revoked, backlog purged, holder told, worker session revoked), one at a time per holder and app. A consent check that throws still only defers the delivery. * perf(auth): record no session or user activity for events handler calls Every API call an events handler made authenticated through its worker session and touched the session and the user, so background handler runs wrote session and user activity rows on the primary database (each at most once a minute), and replaced the session's last IP and user agent with the worker runtime's. Calls made with an events handler token no longer touch either. A handler run isn't the user being active, and the handler session expires only when it is revoked, so nothing reads its activity to keep it alive. The handler session's last activity in the session list now reads as when it was created. * fix(events): settle withdrawn consent only on a fresh answer The delivery path's consent check reads through the permission scan cache and the cached user-to-app grants, which can trail a grant, for example on a node that cached the app's grants before the consent was given. A stale no then ended the subscription for good (permission_revoked, backlog purged, holder told, worker session revoked), where before it only deferred the delivery until the caches caught up. Before settling, the no is now asked again with a new `primary` scan option, which skips the scan cache and reads user-to-app grants from the primary database. Only a fresh no settles. A fresh yes, or a check that throws, leaves the delivery deferred. * test(events): assert no forwarded delivery, not no traffic at all The region's own connect announces presence to its peers on the forward queue's next flush. Now that a dispatch returns before the hand-over, the wait for the socket send can outlast that flush, and the test saw the presence announce as forwarding. * fix(events): budget the settles deliveries start for withdrawn consent A settle a delivery starts when it finds background consent withdrawn ends rows for good. If the permission check itself answered no wrongly, every row a delivery reached would be suspended, its backlog purged and its holder told, with nothing to bring it back. Each process now settles at most 20 such pairs a minute. Past that, the rest stay deferred, as a cached no always left them, and a warning alarm says so. A spent budget skips the fresh read too. A pair a fresh read just found consented isn't read again for a minute, so a stale cached no costs one primary read rather than one per delivery. Each settle is logged.	2026-10-31 14:52:06 -07:00
Daniel Salazar	dd144b6b64	feat(notifications): notify through KV shares, group replacement, fewer push prompts, and events/worker fixes (#4432) ... * feat(perms): add a pushNotifications resource to puter.perms.request notifications:push gets a resource like backgroundEvents, so it can be batched with other access behind one prompt. requestPermission() and hasPermission() ask through it. * fix(puter-js): read puter.gui_features once at load The desktop's feature list was read from location.search on each call, so an app whose router rewrote the URL lost it and enablePush() and the other desktop dialogs answered as if the desktop lacked them. * feat(gui): ask for notifications:push when an app calls enablePush without it enable() and enablePush() inside the desktop answered unavailable for an app without the push permission. They now show the permission dialog, whose Allow also starts the browser registration, and answer with the browser's push state once it settles, or off when denied. A browser that blocks notifications or has no push service still answers without a dialog. * fix(kv-share): charge mint and grant budgets only for share calls that change something A share call now spends the per-call mint budget and the per-recipient grant windows after it has looked up which recipients already hold the share. A call whose recipients all hold the share at the requested level and label spends neither. Recipients without the share, including unknown usernames and refused recipients, still spend a grant unit each. Every named recipient spends from a new 1,200/min lookup window before any lookup runs. * perf(events): skip the subject split for rows not made through a share handle kvHandleFromSubject checks the stored subject's kv:kvh- prefix before splitting it, so own-namespace and fs rows return null without parsing. * feat(notifications): resolve enable() on a website before the confirmation push enable() waited up to 30 seconds for the confirmation push, so a prompt the same click started next had lost its activation. On a website it now resolves once the browser answered and the server has the subscription, records the opt-in, and finishes the confirmation in the background. A failed confirmation goes to onPush's onError and the next load's resync tries again. disable() and sign-out stop a confirmation still waiting. The GUI keeps waiting, since it reads the registered device right after. * fix(notifications): show a page's own notifications only while push is on A hidden page with an onPush handler kept showing notifications after disable(), and its receipts kept claiming showsOwn, which held pushes back from the user's other devices. Both now need the user's opt-in, cached in the page, refreshed on enable, disable, sign-out and each sync, and re-read when another tab of the site changes it. * fix(events): leave subscriptions suspended for good out of the durable quota countForHolder no longer counts rows suspended with a reason that never resumes (permission_revoked), in both the pre-insert check and the recount after insert. Rows suspended for a resumable reason still count. The store now owns the list of terminal reasons, and isResumable reads it. listActiveForHolder with includeSuspended returns only rows suspended for a resumable reason, so its quota-sized read still covers every row a revocation has to settle. Each holder keeps at most 500 terminal rows; a subscribe past that deletes the oldest ones. * feat(notifications): add status() to read whether pushes reach this browser status() resolves 'on', 'off', 'blocked' or 'unavailable' without asking anything. On a website it reads the notification permission, this browser's subscription and the user's opt-in. Inside the desktop a new pushStatus IPC, advertised as push-status, answers the browser's push state, 'on' only while the app holds notifications:push. * feat(notifications): disable() inside the desktop revokes the app's push permission disable() did nothing inside the desktop. A new disablePush IPC, advertised as disable-push, revokes the calling app's notifications:push grant through the same route Settings uses, so its notifications stop going out as pushes and a later enable() shows the permission dialog again. * feat(notifications): allow @ in a notification group, up to 128 characters A group is an opaque display tag, hashed for the Web Push topic and used as the notification tag, so it can carry identifiers such as an email address. * docs(notifications): say when a page with an older Puter.js holds a push back * fix(notifications): let disable() win over an enable() still in flight An enable() whose subscription reached the server after a disable() went on to record the opt-in, so the next page load turned pushes back on. It now rejects with push_verification_canceled and records nothing. * docs(notifications): list push_verification_canceled and reword sentences that led with code * fix(events): carry KV values through a share handle only at read or write The delivery re-check now reads the share's level off the live handle along with whether it still holds. A row on a watch share loses its includeValue for that delivery, so its events name the key without the value; the owner's own rows on the handle keep theirs. Decided per delivery, so raising or lowering a share applies to subscriptions already made, and subscribing with includeValue at watch is still accepted. The live-updates recipe now shares at read, which its events need to carry the score. * fix(events): retry worker runs Puter could not complete without counting them Invocations that never reached the handler (transport and dispatcher errors, unmarked answers, failed or throttled deploys, no configured runtime, a bad invoke key or missing token) now report an 'unavailable' outcome. So does a handler that throws after one of its own puter.js calls got a 5xx, 429 or network error from Puter's API, which the events runtime reports as 'platform-error'. Those runs back off like retriable ones, honor the answer's retry-after up to the backoff ceiling, and no longer count toward the consecutive-failure suspension. A run past the invoke timeout still counts. * test(notifications): cover the pushNotifications resource and status() in the API suites * test(gui): Allow on a batch that includes notifications:push turns push on * fix(events): hold a share-handle row's backlog when its share can't be checked The share-handle delivery check now tells a share that is gone (handle not live, not the row's holder, identity gone, grant withdrawn) from one it can't check right now (a lookup that throws, an owner who is suspended). Only the first settles a queued row as permission_revoked and purges its backlog. The second delivers nothing, leaves the row in service with its backlog, and sends it to the back of the sweeper's line, so delivery resumes once the lookup recovers or the owner is unsuspended. A revocation settle skips such a row and leaves it to the delivery check. kvSharedRegionAuthorized returns null when the permission check throws, and that answer is not cached. * feat(events): update a persistent subscription's context in place POST /events/subscriptions/update replaces a durable row's context, with the same validation and 4 KB cap as subscribe, keeping its subId and backlog. The row's owner region cache is rebuilt from the table and the generation bumped so every region delivers the new value. It spends the subscribe call budget. puter.events.update(subId, { context }) wraps it; null clears the context, and a function handler this client registered for the subscription gets the new ctx. * fix(events): serve a fan-out's holders in turns and cap values per delivery #route now orders an event's candidate rows before any cap applies: the keyspace owner's rows first, then the other holders taking turns, one row per holder per round, starting from a random holder each event. The filter evaluation cap, the delivery cap and the gap markers all count in that order. Values no longer drop for every delivery once more than 128 rows match or the filter evaluation stops early. The first 128 deliveries that asked for the value carry it, in serving order, and later ones carry the key alone. EVENTS_KV_VALUE_OMIT_MATCHED_SUBSCRIPTIONS becomes EVENTS_KV_VALUE_DELIVERIES_PER_EVENT. The caps themselves are unchanged. * fix(email): file inbox mail as a delivery, not the request actor's write storeInboxMessage and copyIntoInbox now run their filesystem writes with a platformWrite context flag, and fs event dispatch leaves actingUserId unset under it. Mail filed into an account's inbox from inside that account's own request (an app mailing the signed-in user) is delivered with self: false, the same as external ingress. Handler depth and viaWorker are still carried from the actor. * feat(notifications): raise the default daily app message limit to 1000 The per-day default for `app.message` goes from 300 to 1000 per user and app, matching the daily push cap. The per-minute default stays at 30, and `notificationAppMessageLimits` still overrides both. * feat(notifications): look up an unseen notification by group `notification` gets a nullable `group_key` column, a sha256 hex digest of the group a row was written with, indexed on (user_id, app_uid, group_key). `NotificationStore.create` takes a `group`, and `getUnseenByGroup` returns the newest row an app wrote for a user under that group that is neither shown nor acknowledged. Rows written before the column exist never match. `updateValue` takes `onlyUnseen`, which also skips a shown row, and `refreshCreatedAt`, which dates the row to now. Migrations: sqlite 0098, mysql_mig_52, postgres_mig_41. * feat(notifications): replace an app's unseen message in the same group An `app.message` with a `group` now replaces the app's earlier notification to the same user with that group while it is neither shown nor acknowledged. The row keeps its uid and takes the new value and time. With no group, or once the earlier one was shown or dismissed, it is a new notification as before. A replacement spends a per-minute slot but no per-day slot. If the earlier row is marked between the lookup and the rewrite, the message is written as a new notification and spends a day slot then. `app.message` is groupable, so the replacement goes through `notifyUpdate`. For types kept off the desktop, `notifyUpdate` skips the desktop wire and dispatches the rewrite again under the same uid, so app subscribers and receipts see it, and `notif.written` carries `updated`. The driver's write path takes the recipient id rather than the actor, so rate limits key on the recipient. * feat(workers): report the app a worker runs as workers.create now returns app_uid and app_name for the app its token is bound to (null for a user-scoped worker), and list/get items carry app_name beside app_uid from the app rows they already load. The dev center workers table shows an App column, and the SDK typedefs and docs describe the fields and that a sandboxed worker's notifications and pushes come from its sandbox app. * feat(events): refuse handlers with free variables on REST publish POST /events/handlers/publish and /publishAll now run the SDK's own scanHandlerSource on each source before publishing and answer 400 events_handler_free_variable, naming the identifier. The backend imports the module from src/puter-js; the build marks dist/src/puter-js as an ES module package so the compiled backend can require it. Sources over the per-handler cap are left to the store's size refusal rather than tokenized. * docs(events): say Puter-side retries never suspend and stay within backlog caps * fix(events): redeploy an events worker that answers bad-key A script deployed before the internal secret rotated keeps the old invoke key, so every invocation answered bad-key and nothing redeployed it. A bad-key answer now goes through the same miss handler as a dispatcher miss (the deployer, with its hourly throttle) and is retried once with the deployed header. A throttled or failed redeploy stays 'unavailable' and backs off without counting. * perf(events): check the deploy throttle before reading handler sources EventsWorkerDeployer now reads the app's handler names and hashes first, answers no-handlers or stale from those, and checks the hourly deploy throttle before loading the sources and generating the worker. A throttled retry costs a hash read and a counter bump instead of a full source read and build, and a stale request still does not spend the throttle. The generated set is checked against the script again in case it moved between the two reads. * fix(workers): move a redeployed worker's row to the app it now runs as Redeploying an existing worker under a different app minted the new app's token but left subdomains.app_owner on the old one, so list/get and hot-reload kept naming and minting for the old app. The redeploy now sets app_owner to the binding #resolveWorkerAppBinding authorized (null for a user-scoped worker) through a new SubdomainStore.setAppOwner, since app_owner stays read-only to update patches. * fix(workers): revoke the old binding's worker session on a rebind A redeploy that binds a worker to a different app (or to none) now deletes the owner's worker sessions under that name other than the one for the new binding, through the same deleteWorkerSessionsByName path worker deletion uses (which also announces the revocation and drops access-token grants). SessionStore's deleteWorkerSessions takes an exceptAppUid filter for it, null meaning the user-scoped session. The app_owner move and the revocation now both run only after the new script deployed successfully, so a failed redeploy leaves the live script, its token and the row on the old binding. * feat(kv-share): store a notify flag on share handles kv_share_handles gains a notify column, off by default, so every existing share reads as not allowing notifications (sqlite 0099, mysql_mig_53, postgres_mig_42). The store reads it on every row, including the live handle cache and the holder listing, writes it on mint, and changes it through update(). * feat(kv-share): let the owner turn notifications on for a share POST /kv/shares and puter.kv.share() take notify, off by default. Leaving it out of a re-share keeps the stored value, and turning it on or off counts as a change, so it spends the mint budget. Turning it on for a holder who now refuses the owner's shares is refused, like a raise or a new label. The kv.share.received notice is sent only when a share starts carrying notify: a new share made with it, or an existing one switched on. Its hourly bound and the app-less namespace exclusion are unchanged. The share response, the owner's listing and the holder's listing report notify. * feat(kv-share): resolve the other party of a share for a notification KvShareService.resolveNotifyRecipient(actor, handle) returns the recipient user id, app uid and handle for a notification one party of a share sends the other through its handle. The handle must be live with an unsuspended owner, the caller must be its owner or holder acting from the app whose data it is, the share must have notify on, the holder's grant must still stand, and the recipient must not refuse shares from the sender. An unknown, retired or foreign handle, a caller who is not a party, and a caller in another app all get subject_does_not_exist. The account itself gets forbidden, a share without notify kv_share_notify_not_enabled, and a lookup that fails service_unavailable. Once every check passes it spends one unit from KV_SHARE_NOTIFY_LIMITS (30/min, 500/day per sender, recipient and app), peeking both windows first so a refusal spends nothing. * feat(notifications): hand a replaced notification to listeners again A replacement now records `replaced_at` (epoch ms) on its row, added to the same unreleased migrations as `group_key` (the sqlite file is renamed to 0098_notification-groups.sql). Its live event and its fetched copy both carry `replaced: true` and use that time as `ts`, so the pair versions the uid identically on both paths. `onNotification()` dedupes on uid plus that version: an original counts as version 0 and a replacement as its `ts`. A newer replacement is handed over again with `replaced: true`, while a copy already handed over, a replay, or an older state is dropped. Listeners started with `catchUp: false` still get replacements of what was already there. * fix(notifications): replace only the newest notification in a group The group lookup takes the newest row an app wrote under the group and returns nothing when that row was shown or dismissed, instead of falling back to an older row that is still unseen. The docs state the rule as it is: a grouped notification replaces the newest one in its group unless that was shown or dismissed. Puter.js marks neither, and the only thing that does is a tap on the push inside the Puter desktop, which dismisses it. * feat(notifications): notify the other party of a KV share through its handle `app.message` takes `to: { handle }`. The driver validates the message, then `to` must be an object with only a non-empty string `handle`, else 400. `KvShareService.resolveNotifyRecipient` checks the share and spends the pair limit, and the message is written for the share's other party in the share's app. The recipient's per-app limits and group replacement apply to their own rows, and the push is billed to them through `notif.written`. A call with `to` answers with `{ uid }` only, so nothing about the recipient's row reaches the sender. puter.js gets the `NotificationRecipient` type for `to`. * docs(notifications): document notifying the other side of a KV share notify.md covers `to: { handle }`: who can be reached, the errors, the recipient's limits plus the per-pair share limit, the uid-only return and that the recipient pays for pushes. push.md and rate-limits-and-quotas.md point at the share path. The chat notification recipe sends through each share's handle and keeps the per-member handler only for members who need to reach each other; the alert recipe says when its handler is the right fit. * fix(notifications): reveal nothing to the sender of a notification through a share A call with `to: { handle }` answers with an empty object, and puter.js resolves it with no value: a uid would show whether the recipient's last notification in a group was replaced. `notify()` gets one overload per form, and `ShareNotificationInput` is exported. Over the per-pair share limit or the recipient's own limits, such a call gets the same 429 `too_many_requests` with the same generic message, so the sender can't tell which limit it hit. * fix(notifications): serialize grouped app messages per group Grouped app.message sends take a Redis lock on (recipient, app, group) around the lookup and the replace-or-insert, so concurrent sends to an empty group write one row and spend one day slot. A lock still busy after the wait, or an unreachable Redis, sends unlocked as before. Sends without a group take no lock. * fix(events): attribute Puter API failures to the invocation that saw them The events worker runtime counted Puter API failures on the cached per-token client, so a concurrent invocation on the same token could answer platform-error for its own unrelated throw. A cached client now serves one invocation at a time and records failures and retry-after on that invocation; an invocation overlapping another on the same token gets its own client, and idle clients stay cached per token. * fix(events): drop KV values from a queued backlog once its share is at watch A queue delivery is stored with its value. Every hand-out of a stored entry (backlog drain to a socket or another region, the worker, and a declined delivery passed to the worker) now re-reads the share's level and removes the value at watch. A share that can't be checked still holds the backlog. * fix(events): return notifications replaced behind the fetch cursor The notification fetch cursor now holds the last row id and the last (replaced_at, id) it returned. A read from it merges two indexed reads: rows inserted after the id, and rows up to the id replaced after that mark. A replacement made while a client was away therefore comes back on its next catch-up. Adds idx_notification_replaced on (user_id, audience, app_uid, replaced_at).	2026-10-31 12:21:18 -07:00
Daniel Salazar	4851df716c	fix(kv): KV share lifecycle hardening (#4429) ... * fix(kv): harden the KV share lifecycle - Retire a deleted app's live shares before an origin app is registered again under the same uid. The retire runs before the app row exists and throws on failure, so a failed launch leaves no row and the next one retries. - Record deleted-app retires and uninstall cleanups (retire, then purge) as reserved items before they run. A lease keeps one node on a job; a failed run raises an alarm and backs off, and a run that hits its time budget or is cut short by a restart resumes on the next check. Each job is bounded to handles made before it was recorded. - Uninstalling with deleteData records the job and returns; the retire and purge run off the request. The purge checks whether the app was authorized again before deleting each page, so data written after that is kept. - Take a per-region lock when minting and re-check for a live handle under it, so concurrent shares of one region make one handle. - Retire the handles an account holds before the account is deleted, so the owners' subscriptions through them end instead of staying active. * fix(events): bound share-handle session rows by the per-connection cap The count across a holder's connections also counted connections that had dropped but weren't reaped yet (a crashed node, a deploy, a network blip the server hasn't noticed), so resubscribes after a reconnect could be refused with `events_subscription_limit`. Rows made through a share handle now count toward the 50-per-connection cap like any other, and with the open-connections cap per account that bounds what one holder can put in the owner's keyspace. The handle index stays for ending rows on retire. Rows are added to it with the row itself and taken out when the row is dropped, and members whose row is gone are cleared when a retire reads it. * fix(events): keep a retire's durable settle running if ending sessions fails Ending a retired handle's session subscriptions reads Redis. A failure there is now logged and the durable suspension that follows still runs, so those rows' backlogs are still purged. * perf(events): ask once per handle when a refresh screens rows, and share handle reads A socket refresh asks whether a share handle still stands once per handle, holder and app, rather than once per row: rows through one handle share its grant. Concurrent cache misses on one handle share a single read; a forget drops the shared read so later callers start a fresh one. * fix(permissions): chunk the exact-match subtree delete The holder-scoped subtree delete names every matched grant, so a large subtree could exceed an engine's limit on bound parameters. It now deletes 500 grants per statement. * fix(kv): give each holder a share of a namespace's handle budget One holder could spend a namespace's whole handle budget and starve the others. Each holder now gets at most 3,750 of the namespace's 15,000 units per 10 seconds, and the owner's own calls through a handle spend none. The holder's budget is checked before the grant lookups, so throttled retries cost none; the namespace's is only told to callers the grant admits. Multi-unit charges use a new counted claim that keeps two counters per key instead of a sorted-set member per unit. * fix(kv): refuse handles older than their app and claim cleanup jobs globally - A share handle created before its app row reads as gone: in the KV driver, in the handle cache, on the events subscribe path, and in the delegate implicator. An origin app registered again under a deleted app's uid can't reach its predecessor's shares, whatever created the row. The app-token launch no longer retires inline; it records a background retire and never waits on it or fails because of it. The clock-skew rule is shared with app sessions. - Cleanup job records now carry their lease, attempts and retry time, and a node claims a job by compare-and-set on its record. A job stays with the deployment that recorded or last ran it, and another one adopts it only after it has gone unclaimed well past due. - The job check pages through records from where the last look stopped, so waiting records don't hide due ones. After 10 failed runs a job stops retrying with a warning alarm, and its record expires after 30 days. - A mint that waits out the region lock looks for the share again and returns it, or goes ahead, instead of failing with too_many_requests. - The record-failure alarm dedups; an unfinished run is a log line. * fix(kv): hide shares older than their app from re-shares and listings After an app is registered again under a deleted app's uid, its predecessor's live handles stayed visible: sharing the same region again handed back the old, dead handle, and both sides' listings showed the old handles and labels. findLive and both listing queries now leave out a live handle created before its app row, the same rule handle resolution uses, so a re-share makes a new handle and the listings, totals included, skip the old ones until their retire lands. A new share also retires, grants and all, the handles its owner gave the same holder for the earlier app first, so withdrawing those later can't take the new grant with it. * fix(events): keep a loose cap on subscriptions through one share handle Without a count across connections, a holder could put up to 20,000 rows in the owner's keyspace (50 per connection, 400 connections), each one evaluated on every owner write. The cap is back at 1,000 per holder per handle, far above what connections that dropped without being reaped leave behind. The index is only read row by row once it's past the cap. * fix(kv): harden the share cleanup jobs - Job records are keyed by domain, so a deployment that shares the table but not the database or broadcast (staging beside production) never runs another's jobs. - A job whose record can't be written runs unrecorded instead of not at all, and a failed check for live handles records the job anyway so it retries. Failures that used to be swallowed are logged. - Recording a job again keeps its attempts and backoff; one that gave up gets one more attempt. - A system retire gives back the grant of a live share it doesn't cover inside a withdrawn region, and a share made under a reused uid grants again once its row stands, so a concurrent retire can't leave a live handle without its grant. * fix(kv): don't cache a live handle a retire overtook during its app lookup The live read decided whether it could cache before awaiting the app lookup, so a retire's forget that landed during that lookup was overwritten with the handle as live for up to 60 s. It's now checked after the lookup. * chore(kv): word the job key comment for any install	2026-10-31 12:12:29 -07:00
3 people	8d3a1ccc2b	feat(notifications): remove a notification from the dashboard history (#4412) ... Each notification in the dashboard list gets a remove (✕) control next to mark-as-read. It calls the new POST /notif/delete, which deletes the user's own row and sends `notif.removed` to their other tabs so the entry and any toast for it disappear everywhere. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Daniel Salazar <daniel.salazar@puter.com>	2026-10-30 22:52:27 -07:00
Daniel Salazar	4881264c88	feat(ai): realtime reconnect notice, context restore and retry hardening (#4415) ... * test(ai): end every realtime relay test's upstream before the next The fake upstream now answers response.create with response.created first, as OpenAI does, so closing those sessions no longer waits out the settle window. * feat(ai): ask realtime clients to reconnect before a restart Open realtime sessions get a puter.reconnect event when the drain starts, with close_in_ms set to the drain window, and new sessions are still refused until the 1012 close at shutdown. A host extension can announce a restart early by emitting serverRestartAnnounced; that only sends the notice, so a false alarm leaves the server working normally. A session gets the notice at most once a minute. onServerDrainStart now receives the drain duration. * fix(ai): retry realtime reconnects for about a minute and wait out offline browsers Five attempts gave up after about 15 seconds, shorter than a server restart. The conversation makes ten attempts with the backoff capped at 10 seconds, and while the browser reports being offline it waits for the online event, up to two minutes per reconnect, instead of spending attempts. close() ends either wait and removes its listener. * feat(ai): restore recent conversation context after a realtime reconnect The provider starts a fresh session on every reconnect, so the conversation lost all earlier items. The SDK now records finished items it can recreate from the provider's own events: user text, user speech with a transcript, assistant replies as text, and function calls with their outputs. After the new session acknowledges its settings, and before ready fires, it adds back the newest of them, up to 40 items and 48 KiB, which stays under the relay's upload burst. Assistant replies are cut to what the user saw or heard: the deltas shown before an interruption, and the share of the transcript matching the audio played before a truncation or a drop. Deleted items, unfinished replies, images, speech without a transcript and incomplete function calls are left out, as are function outputs whose call falls outside the window. Restored copies take over their new item IDs. A refused item is reported once as context_restore_failed without failing the reconnect. * feat(ai): move a realtime conversation to a new session on the restart notice On puter.reconnect the conversation waits until it is idle: no reply requested, generating or playing, no user turn being spoken or transcribed, no function call awaiting its output and no acknowledgment pending. Then it reconnects through the usual path, emitting reconnecting (code 1012) and ready with reconnected: true, without an error. Until then the session keeps working, and if the server closes it first the normal 1012 reconnect takes over. close() at any point stops the wait. * docs(ai): realtime restart notice, context restore and reconnect limits * test(ai): realtime reconnects against the local server The mock provider answers with item events and whole replies like OpenAI. New cases cover a dropped connection (cut through a TCP proxy so the client sees 1006), a host-announced restart, and a draining server refusing the reconnect; the first two check the provider receives the restored items before the next turn. The browser media test waits for captured audio and the reply instead of a fixed sleep. The live suite gains a tool and text context round trip across a restart notice. * ci: run the live realtime conversation tests with the AI provider suite They need the OpenAI key, which only this workflow has, and skip themselves without it. * fix(ai): open the replacement before leaving a realtime session on the restart notice The handover closed the current session and then reconnected, so every notice cost a second or two in which sends failed and microphone audio was dropped. On a single server that refuses new sessions while it drains, the retries ran out during the drain and the conversation closed sooner than if it had stayed. The conversation now opens a replacement session as soon as the notice arrives and switches only once it is open and the conversation is idle. A refused replacement is retried with backoff while the current session keeps working; when the server closes it first, the held replacement is used right away. The idle check also missed some in-progress work: - a reply created while the user was still speaking ended the user's turn; speech now lasts until its audio is committed or cleared, and a committed turn waits for the reply server turn detection creates - only the newest function call was checked; every finished call since the last user message needs its output, and calls cut off by a cancellation don't count - native session.update and conversation.item.create events the provider hadn't acknowledged could be lost; the switch waits for them, up to the acknowledgment timeout for items - only the newest transcription was waited on; every committed item is - clearing playback didn't recheck the switch Restored items stay internal: the new session's copies of them no longer reach listeners. At a reconnect, every queued item the user didn't hear is cut, not just the one playing. A notice from a failed reconnect attempt is dropped. * test(ai): realtime replay after split settings, and a draining server refusing the replacement The relay accepts the context replay right after two full-size settings updates. Against the local server, a draining relay refuses the replacement session while the current one keeps answering. * ci: build the SDK in the live realtime step so it stands on its own * fix(ai): a failed realtime reconnect attempt no longer poisons the next An attempt that dropped after sending its session.update but before the acknowledgment left that event ID queued, so the next attempt's acknowledgment was matched to it and timed out, and so on until the conversation closed. Turn state is now reset after each failed attempt. * fix(ai): bound the realtime handover's waits and stop retries that can't succeed - The first replacement opens within a randomized 3 seconds of the notice, so sessions notified together don't all open at once. - A replacement refused with a code retrying can't fix, such as insufficient_funds, ends the attempts; the server's close still falls back to the usual reconnect. - A held replacement that gets an error or its own server's notice is closed and replaced. - A held replacement used after a drop gets a 3 second acknowledgment timeout, since its socket may have died with the network. - close() during the switch closes the held replacement too. - The reply to a committed turn and each transcription hold the switch for at most 10 and 30 seconds, like app items already did, and an app item is released only by an added item of its type, role and call ID. * docs(ai): the realtime restart notice isn't guaranteed before every restart	2026-10-30 18:06:54 -07:00
Daniel Salazar	7401fe0b09	fix(notifications): receipts report what the user saw, sticky push opt-in, safer app pushes (#4413) ... * fix(socket): count sockets that never acknowledge instead of asking them askWithAck sent an acked broadcast to sockets outside the ack room only to learn whether any were there. Those sockets never answer, so every node held the ack callback until they disconnected. Each node now answers a server-side count of them instead; a node that does not answer in time still reads as holding one. * feat(notifications): read focus and own display from receipt answers A receipt answer may carry focused and showsOwn beside visibilityState; both pass through to notif.delivery.settled when they are booleans. An answer with received: false is not a responder, and an ask answered only that way settles as unreceived. * fix(notifications): name an app message's bad field before the size cap An app message over the 16 KB value cap answered notification_value_too_large even when one field was what broke a rule of its own. The fields are checked first now, so the caller gets bad_request naming the field; the total cap still applies after. Character counts skip the code point walk on strings that can't fit. * fix(push-sw): match the server's tags, keep taps, and show pages' notifications - A push is tagged with the server's group, `<app uid>:<group>` or the uid, and alerts again when that replaces one showing. Handlers get the app's own group and the send time in ms from `ts`. - The app's icon stands in when the notification has none. - Default and relative tap URLs resolve against the worker's scope, still only on its origin. - A tap posted to an open window waits briefly for the page to take it; without a click handler there, the window is taken to the URL, or one is opened. - A page that got a notification live while away can ask the worker to show it; the worker holds it back while another window of the origin has focus. * fix(notifications): answer every receipt ask A page now answers each socket that asks about a notification, not only the first, and answers { received: false } when the notification doesn't arrive within the wait or the page holds too many asks, so the server's nodes can let go of them. A received answer carries focused and showsOwn beside visibilityState. * fix(notifications): keep web push to users who turned it on, and show what a push would Subscription: - onPush() renews the subscription on load only for a user who turned push on on this origin with enable(); enable() records that, and disable(), sign-out and push_revoked forget it. One renewal runs per page and worker, however many onPush() calls there are, and push_disabled, push_revoked, push_unavailable and too_many_requests stay quiet there. - puter.auth.signOut() on a website removes this browser's subscription with the leaving user's token and forgets the opt-in and kept taps, without holding up the sign-out. - A subscription another service on the site made is left alone and rejected as push_subscription_conflict; one made with Puter's earlier key is replaced. - A subscription refused with push_endpoint_revoked is replaced once, like push_subscription_invalid. - Another worker holding the scope rejects push_worker_conflict, naming importScripts and the serviceWorker option, instead of waiting out the confirmation push. - enable() rejects user_gesture_required outside a click instead of prompting, and a refused browser subscribe rejects push_subscribe_failed. Live side: - A page that is hidden or unfocused shows every app notification it receives, with or without a handler for its channel, through push-sw.js so a focused window of the app holds it back. It uses the server's tag, the app's icon as a fallback, the generic app line for sensitive ones, and the registration of its own worker. - Page handlers get 5 seconds, as in the worker (push_handler_timeout). - A feed stopped by a sign-out starts again on the next sign-in. - onPush() still delivers live when IndexedDB is unavailable. - onPush() rejects invalid_handler for a handler the worker could not rebuild from its source (methods, bound and built-in functions). - Saving and removing a handler's source for the worker is ordered per key, so a quick unmount and remount keeps it. - A tap posted by push-sw.js is acknowledged only when a click handler takes it. * fix(gui): show app pushes under the app's name and Puter's icon - On Puter's own origin, an app's push takes the app's name as its title, with the app's title and text in the body, and Puter's copy of the app's icon; an icon the app chose is never shown. A sensitive one keeps its generic line. - A push's ts is its timestamp, and a newer push in a group alerts again. - Re-registering this browser counts only apps that push to Puter's devices (puterDevices), so a website's Web Push confirmation alone never registers it. * fix(gui): free the push dialog's button when registering fails late A registration that failed after the dialog's deadline left the button spinning and disabled. It now clears the attempt, settles on a state the browser moved to meanwhile, or offers a retry. * fix(permission): a user's own app grant clears a narrower via on the row An app grant that already existed was skipped, so a row another path recorded with extra.via kept it after the user approved the same permission in Puter. A grant without via now rewrites such a row without it, keeping its other extras; a grant with via still never touches an existing row. * refactor(notifications): export the notification text cleaner Extensions that word notifications can use the same cleaning as app messages. * docs(notifications): describe when pushes go out, taps, worker rules and enable() errors Pushes go out when no open page answers that it's visible and focused or shows the notification itself. Documents the page-shown notification, the sticky opt-in, tap routing, group and sensitive behavior, service worker rules, requirements, every enable() and onPush() error, push caps for unlisted apps and websites, and sending from workers and events handlers. The push playground example is removed (the playground can't serve the site's service worker) and redirects to the guide. The minimal page catches enable() errors and sends after 5 seconds. The recipe no longer sends on every load, and app.message is listed in events fetch and the catch-up recipe. * docs(notifications): per-app browser caps and when push_endpoint_revoked appears * docs(notifications): the page's handler timeout * fix(notifications): leave a site's own service worker alone when enabling pushes Registering /puter-sw.js at a scope another worker holds queued it behind that worker, so enable() rejected with push_worker_conflict but swapped the site's worker out once its tabs closed. Now enable() checks the scope first and rejects without registering. * docs(notifications): a background page shows score alerts itself	2026-10-30 14:43:48 -07:00
Juan Fernando CastroandDaniel Salazar	c00f693ec6	feat(notifications): report how each live notification was received (#4403) ... * feat(notifications): report how each live notif.message was received SocketService.sendWithAck sends an event to rooms or sockets and waits, up to a timeout, for an acknowledgement from each socket that listed the event in its handshake (`auth.acks`); every other socket gets a plain emit. It lists sockets across the cluster with fetchSockets and asks the listed ones through the adapter's broadcast acks, and counts this node alone when the adapter cannot list the others. The emitting region now sends notif.message through it, with the window from `notificationAckTimeoutMs` (default 5000), and emits the node-local `notif.delivery.settled` with the outcome: received, unreceived, no-socket or remote-only (sockets only in a peer region), whether any socket there acknowledges, and each responder's visibilityState. Peer regions keep sending their own copy as before; the GUI event carries `region_delivered` so this region does not send it twice. The desktop and dashboard list notif.message in their handshake and answer its callback once with document.visibilityState. Nothing in core acts on the settled event. * fix(notifications): send live notif.message at once and report what the count covers sendWithAck now emits to the whole room straight away as a broadcast with ack, and lists the room's sockets alongside it only to fill in the counts, so nobody waits on the listing. Whether the adapter can count the cluster's servers is checked once; where it cannot, sends go out per socket on this node with a one-time warning. The result carries `complete`: false when only this node's sockets were counted, either for that reason or because the listing failed or took over 1 s, which is warned about at most once a minute. notif.delivery.settled gains `sendId`, `type`, `complete` and `peerRegions`. An incomplete count with no socket found settles as unreceived rather than no-socket, and the peer-region read runs alongside the send instead of after it. notif.written carries the same `sendId`, minted per send (an update gets a new one) and passed through notif.created to the socket adapter on the fold-in path. The comment on revoking sockets no longer says fetchSockets is unavailable with our Redis client; it works with ioredis cluster. * fix(notifications): keep a failed server count from ending the process socket.io awaits the adapter's serverCount without a catch on every broadcast with an ack, and the redis-streams adapter's count rejects as soon as one cluster node fails to answer. The count on the root namespace's adapter now never rejects: a failure reports the last good count (1 before any), and is warned about at most once a minute. A stale count only lets an ack wait run to its timeout. Whether sendWithAck takes the cluster path is no longer a one-shot check awaited before the emit. Until a count has succeeded, sends go out per socket on this node at once while a count is retried in the background, at most every 30 s (SocketService.COUNT_RETRY_MS); the first success switches to the cluster path for good. * fix(notifications): ask only acknowledging sockets, through an ack room A socket whose handshake lists an event in `acks` now joins `<room>:acks:<event>` for each of its rooms and its own id. sendWithAck sends the ordinary broadcast to the target rooms except those ack rooms, which keeps it on the adapter's stream, and a broadcast with an ack only to the ack rooms. Sockets that never acknowledge are no longer asked, so other nodes hold no pending ack for them and a send completes as soon as every acknowledging socket answered. The per-socket listing that matched socket.data.acks is gone; the ack-capable count comes from the room membership the listing already returns. The per-socket path stays only for a node whose adapter has not counted servers yet. A failed server count during a send now marks its counts incomplete, so a stale count can't make a listing that missed a node read as complete. With no count ever succeeded (an in-memory Redis), the failure is warned once per process instead of once a minute. * fix(notifications): say whether every socket on a live notification acknowledges it A tab on an older bundle beside one that acknowledges made the settle read as ack-capable, so a listener could not tell that the older tab may have shown the notification. notif.delivery.settled now carries everySocketAcks. * feat(notifications): app pages acknowledge the live copy of an app message NotificationService.appReceipt() asks an app's own sockets whether a notification reached a page, and settles it as notif.delivery.settled with appUid, the same outcomes as the desktop's. Callers ask only when the answer decides something, since it is a round trip to every node. - SocketService.askWithAck asks without listing sockets: sockets that don't acknowledge get an acked broadcast too, which can only time out, and that says whether an older client shares the target. - puter.js declares notif.receipt on its filesystem and events sockets. A page answers once one of its notification feeds received that uid, before any handler runs, and never when no feed on the page got it. - Docs describe the push route as "no page of the app received it live within 5 seconds". * fix(notifications): count a socket that acknowledges other events as a current client Every desktop page also holds puter.js's own socket, which never shows notif.message, so "every socket acknowledges notif.message" never held there. sendWithAck now reports `declaring`, the sockets that listed any event in auth.acks, and everySocketAcks on the desktop's settle means no socket here declares nothing. * fix(puter-js): fall back to the Continue dialog when a popup can't be opened Some browsers report user activation but still return null from window.open (iOS home-screen web apps, Safari after awaited work), so permission prompts, signIn, enablePush, and GUI action popups failed without showing anything. PuterDialog.launch() now owns the open-now-or-show-Continue decision for every popup flow and shows the consent dialog when the direct open returns null. On cross-origin-isolated pages signIn leaves settling to the /login/wait relay on both launch paths. * fix(notifications): show a live push when the page's window isn't focused onPush showed the handler's result as a system notification only when the tab was hidden. A visible tab in a window the user left showed nothing. It now also shows when the document doesn't have focus. Inside the Puter desktop the desktop's banner shows the notification, so an app's frame still never does. * build(puter-js): publish push-sw.js exactly as written The build emits dist/push-sw.js next to puter.js, but terser minified it along with the bundle. It is now marked as already minimized, so the file apps import into their service worker is the source unchanged. * fix(notifications): send live notif.message plainly and ask for receipts on request Every live notif.message used to be sent with an ack, list the user's sockets across the cluster (fetchSockets ships each socket's handshake between nodes) and wait out the receipt window, whether or not anything would act on it. notif.message is a plain send again. NotificationService.receipt() asks the desktop's sockets, or with appUid an app's own, the way app receipts already did: an ask with no socket listing, settled as notif.delivery.settled. A caller asks only when the answer changes what it does. - The desktop and dashboard sockets answer notif.receipt from the page's puter.js receipt log and note each live notif.message in it, so every socket on the page answers alike. A page whose puter.js has no log lists no acks. - SocketService.sendWithAck, the socket listing and region_delivered are gone. askWithAck counts this node's sockets when the adapter cannot count servers. - The send id stays on notif.written and the settle, and leaves the wire. * fix(socket): join only the ack room receipts are asked in A socket listing events in auth.acks joined an ack room beside each of its rooms and its own id. Receipts are only asked in the user's room, or the app's for an app socket, so that is the one ack room it joins. * docs(notifications): publish the push limits of 12 a minute and 1,000 a day Pushes are capped per user per app at 12 a minute and 1,000 a day, and at 6 a minute and 200 a day while an app is new. The rate limits table and the notification pages said one a minute and 100 a day. --------- Co-authored-by: Daniel Salazar <daniel.salazar@puter.com>	2026-10-29 18:57:36 -07:00
Juan Castro	406fe4dcfc	Merge juancastro/put-2168-let-users-decide-which-apps-can-send-them-pushes into juancastro/put-2167-notification-written-hook	2026-10-29 08:58:57 -04:00
Daniel Salazar	28be1851f9	fix(workers): bind credentials and cleanup to the owning site (#4362) ... * fix(workers): bind credentials and cleanup to the owning site * test(workers): deploy valid source in the site-bound token test Where workerd is installed the deploy starts the script, and the bare identifier `src` threw at startup as an unhandled rejection.	2026-10-28 17:31:15 -07:00
Juan Castro	59700519a6	Merge juancastro/put-2168-let-users-decide-which-apps-can-send-them-pushes into juancastro/put-2167-notification-written-hook	2026-10-28 09:45:38 -04:00
Juan Castro	bbf375db14	fix(notifications): notif.written hands out a copy with the audience ... Listeners get their own copy of the value, so what the socket carries is what was written. Both emits name the audience, and an update names the row's app when the row or the value says which.	2026-10-27 14:10:56 -04:00
Daniel Salazar	49b38f3482	fix: seat creation (#4387)	2026-10-27 12:49:31 -04:00
Juan Castro	20c63b6282	Merge juancastro/put-2168-let-users-decide-which-apps-can-send-them-pushes into juancastro/put-2167-notification-written-hook	2026-10-27 10:44:19 -04:00
Daniel Salazar	2dfd93687f	perf(database): defer primary fallback for user lookups (#4359)	2026-10-26 21:37:06 -07:00
Daniel Salazar	52da7b571c	perf(permission): read a large holder's user-to-user grants per permission (#4349) ... A user-to-user grant check loaded the holder's whole row set, from one Redis blob or one SELECT, and filtered it in JS. Every KV share handle a user receives adds a row, so the cost of each check grew with incoming shares. Holders with up to 200 grants keep the whole-set cache. Past that the cache holds a marker and each check reads only the permissions it names, off a new (holder_user_id, permission) index. Matches stay exact in JS, since MySQL compares `permission` case-insensitively. The batched multi-holder read shares the same chunked query. Migrations: mysql_mig_51, postgres_mig_40, sqlite 0097.	2026-10-26 20:51:46 -07:00
Juan Castro	fceaee6372	feat(notifications): node-local notif.written hook ... NotificationService emits notif.written after each non-silent write and after an in-place rewrite, on the node that wrote the row, so an extension can act on new notifications without a second bus round trip. It stays off the published event stream; notif.created carries the notification itself.	2026-10-26 16:34:18 -04:00
Daniel Salazar	f850155411	KV share handles: hardening, higher limits, batch share, uninstall data deletion, account-deletion cleanup (#4344) ... * fix(kv): sign share handle ids and drop retired handles from every node's cache New handle ids carry a short HMAC (kvh-<uuid>.<mac>); an id whose mac does not verify is refused before any lookup. Unsigned ids keep working but no longer fall back to the primary on a replica miss. Retiring or changing a handle now broadcasts outer.pubsub.kv.share.forget so every node drops its cached row. Event subscribes and delivery rechecks refuse a handle whose owner is suspended, as the driver does. * fix(kv): an app its user removed stops reaching shares An app acting for a user exercises a share grant only while the user still has the app authorized: at least one user-to-app permission row, which every app token path leaves and revoke-user-app '*' removes. The share itself stays; authorizing the app again restores access. The answer rides the permission cache, keyed by the app-under-user generation the revoke bumps, so live subscriptions go quiet too. * feat(kv): list keys through a watch-level share handle list with as: 'keys' through a handle needs only watch, which already learns every key written in the region. Values and entries still need read. * feat(kv): filter share listings by app and by state GET /kv/shares and GET /kv/shares/with-me take an optional appUid: an app uid, or null for the app-less store. An account session may name any app; an app is refused any namespace but its own. GET /kv/shares also takes state=live|revoked|all, all by default. * feat(puter.js): appUid and state options on the KV share listings listShared() and listSharedByMe() pass appUid (null for the app-less store), and listSharedByMe() passes state. Docs cover the new options and that a watch share can list keys without their values. * feat(kv): indexes and store counts for share-handle quotas and retention Adds covering indexes for the owner's live count, the owner-holder pair, app-filtered listings and retired rows, plus store methods to count a pair's live handles and delete rows retired before a cutoff. The live handle cache holds up to 50,000 entries; the miss cache stays at 10,000. * feat(kv): higher share-handle caps, a per-holder cap, and sharing with several recipients Live share handles per account go to 25,000 paid / 5,000 free, and per app to 10,000 / 2,000. One owner may hold out 256 live handles to one holder on every plan, which bounds the rows a revoke or unshare walks. Re-sharing an existing region takes no slot and is never refused. POST /kv/shares takes recipients (1 to 50) as an alternative to recipient: one region, level and label for all. Caps and a new per-recipient grant budget (300/min, 2,000/hour) are checked for the whole batch up front; failures about one recipient are reported on its item. The single-recipient request and response are unchanged and run through the same code path. * feat(kv): delete share handles 30 days after they stop A daily job deletes retired share-handle rows older than 30 days, in batches of 500. One node runs it per day; it stops on shutdown. * feat(puter.js): share a KV prefix with several users in one call puter.kv.share() takes an array of recipients, positionally or as recipients in the options form, and resolves to one item per recipient: the share, or an error with its message and code. The single-recipient forms are unchanged. * feat(kv): delete an app's data on uninstall, and close account-deletion gaps Uninstall: - POST /auth/revoke-user-app takes an optional `deleteData` boolean, only with `permission: '*'` (400 otherwise). AuthService.uninstallApp revokes every grant, deletes the app's worker sessions, then awaits `app.uninstalled` { userId, userUuid, appUid, deleteData }. - With deleteData, KvShareService retires the handles the user shares out of that app's namespace (as the platform, in batches) and then empties the namespace in the background, 5,000 items per step, stopping on shutdown. Handles shared with the user are not touched. - SystemKVStore.purgeNamespace(namespace, { maxItems }) deletes up to maxItems items of one `v1:<uuid>:<appUid>` namespace, reading keys only, with no mutation events or usage, and reports `done` once it is empty. It refuses the system namespace and anything else. flush shares its page-delete path. - puter.perms.revokeApp(appUid, '*', { deleteData: true }); the desktop's uninstall dialog has an unchecked "Also delete this app's data" option. Account deletion: - EventsService settles each of a deleted account's apps on `user.delete` (they cascade without `app.changed`) and drops the owner's subscription keys. - The flat permission entries the account held are purged in pages. Suspension: - `user.suspended` / `user.unsuspended` events; the share-handle store drops cached rows owned by that user here and on other nodes. * fix(permission): await a peer's flat-permission deletes so emitAndWait sees them applied * fix(kv): strongly consistent purge pages, and an uninstall purge stops once the app is authorized again A stale empty page could end a purge as done while items remained. The background purge after an uninstall now checks before each step that the user hasn't authorized the app again, so data written after that is kept. * fix(kv): flush keeps eventually consistent page reads; only a purge reads strongly consistent * fix(user): emit user.suspended and user.unsuspended when the user store changes an account's suspension	2026-10-26 09:43:55 -07:00
Daniel Salazar	589a98b12c	feat(kv): share KV prefixes with other users at watch/read/write, and bound what apps can be granted (#4331) ... * feat(kv): share key-value prefixes with other users, and bound what apps can be granted - KV share handles move out of Events into KvShareService, served at /kv/shares (the /events/kv-handles routes stay as an alias). - Shares have watch, read and write levels, encoded in the permission root (kv-share, kv-share-read, kv-share-write) with write implying read and watch. - Every puter.kv method takes optConfig.handle; keys are relative to the shared prefix, cursors are sealed, and private entries stay hidden. - New puter.kv.share, unshare, listShared and listSharedByMe; optional label; kv.share.received notice in the app's mailbox only; recipient blocks apply. - puter.kv.set gains onlyIfAbsent. - fs and kv events gain viaWorker, and kv events report self from the writer. - /auth/grant-user-app only accepts canonical permissions from a fixed set of families; the consent dialog, puter.perms and docs follow. - Migrations: kv_share_handles gains revoked_by, label, and grantee and app indexes (mysql_mig_49, postgres_mig_38, sqlite 0095). * fix(kv): never name a private key in a cross-app or handle cursor, and bound flat-permission scans - Listings of another namespace (cross-app or through a handle) resume after the last entry they returned, and their cursors are sealed, bound to the namespace or handle, and padded to one length. Old plain cross-app cursors are still read when they name the target namespace; other namespaces get 400. - PermissionStore scans flat permissions a few roots at a time and skips roots nested under another root in the same call.	2026-10-25 08:16:19 -07:00
Neal Shah	03e483e372	PUT-2252 (#4311)	2026-10-23 17:33:59 -04:00
Daniel Salazar	4729d733bb	feat(aws): DynamoDB, Polly and Textract can use the default credential chain (#4307) ... `useCredentialChain: true` in `dynamo.aws`, `providers['aws-polly']` or `providers['aws-textract']` takes credentials from the default AWS chain instead of static keys, as `s3.s3Config` already does. One provider is shared and refreshes temporary credentials itself.	2026-10-22 21:31:23 -07:00
Daniel Salazar	4968a69d45	fix(s3): keep credential-chain credentials an hour from expiry when signing (#4299) ... Presigned URLs die with the credentials that signed them, and the SDK only refreshes temporary credentials five minutes before expiry, so a URL signed late in a credential's life could expire long before its requested lifetime (up to an hour). The chain provider now reports expiry an hour early and force-refreshes credentials that are already inside that window.	2026-10-22 00:47:48 -07:00
Daniel Salazar	9ff224e22f	perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259) ... Pipeline - CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s. - `req.ua` is parsed on first read instead of on every request. - JSON `rawBody` reuses the parsed buffer instead of copying it. - `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call. - SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under load is unchanged. Auth and permissions - Token-to-actor auth starts the user, session and app lookups together and checks them in the same order: fewer sequential Redis round trips per authed request. - get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select after its own insert and doesn't wait on the cache write for replica rows. - getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer hands out a session that was never written. - whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged. - OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight per URL, and the Apple key is parsed once. - gui-token and sync-cookie use `req.actor.user` instead of re-reading the user. - listSessions batches app lookups. App icon probes run together, and the original-icon probe skips the lineage fallback. - Default-granted permissions are answered from config before the scan cache, and exploded options are deduped: one less Redis read per /drivers/call. - ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node first, then the ancestors concurrently on a miss. Filesystem - Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about 1.3 s to about 40 ms per 100 new files. - MySQL single-file inserts return the row built from the written values, falling back to the primary read: one less primary round trip per new file. - Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row fallback. - storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries `userUuid`: quota checks and /df no longer serialize the listener reads. - readdir cache backfill is one pipeline instead of 3xN SETEX. - Profile reads skip the lineage fallback; the own profile is read alongside the user lookup. - Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently. - Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream upstream error. - mkdir looks up target and parent together. - Upload completion reuses the session from the access check (status is still re-read before completing), and multipart signing reads it once. - Moving a shared file to Trash skips two lookups that always returned the entry itself. Drivers and events - Subscription resolvers run concurrently and are folded in registration order. - SubdomainDriver and WorkerDriver overlap independent lookups. - The events "anyone subscribed?" check is one Redis pipeline when the region is warm. - Handler publish overlaps the count with the size check. - Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays cross-region invalidations to the others. Sites, share and AI - Hosted sites resolve the owner first, then the app and root together. Site config loads only on 404s. The hosted-app query selects only the columns it uses and moves into AppStore. - shareFlags asks about the plan only when an item is shared by link only. - Share listings run the count, teams and plan reads alongside their page query. - Claude attachment cleanup no longer blocks the response. - New util `awaitLater` for promises started early and awaited later.	2026-10-17 12:03:35 -07:00
Juan Fernando Castro	e32bfa839d	fix: count one app open per account per app per hour (#4152) ... * fix: count one app open per account per app per day `app_opens` is read two ways at once: `open_count` is a row count and `user_count` is distinct accounts. A repeated open moves the first without the first moving the second, and the incentive report prices opens. An open is now marked as it is recorded and skipped for the rest of the day, before both the row and the event ClickHouse ingests, so the two stores agree. The marker is set only when absent, so the first open of the day wins a race; an unreachable marker counts the open, since counting twice beats not counting. * fix: count one app open per account per app per day, at the count Dropped the row-suppression approach. `app_opens` is not only a counter: `getRecentAppOpens` orders the desktop's recent list by `MAX(_id)`, and `app.opened` is the socket event other tabs reorder on. Suppressing the repeat row froze both at the first open of the day. Every open is recorded as before. `open_count` counts each account once per day instead, which is the figure `user_count` is read against. `countDistinctUserDays` folds the pair into one number because `COUNT(DISTINCT a, b)` is MySQL-only, and `dayBucket` covers MySQL's decimal division. ClickHouse has uniqExact over the tuple. Note the figure's meaning changes at deploy: a period spanning it mixes rows with account-days, so a comparison across that point is not one. * fix: count one app open per account per app per hour Review feedback: a day is too long a window for the repeat to collapse into. The period is now an argument rather than a baked-in 86400, held in one constant on AppStore beside the other store policy numbers, so the SQL and ClickHouse sites cannot drift and a future change is one line. The fold `COUNT(DISTINCT user * N + bucket)` is only injective while the bucket index stays under N. Hourly buckets are 24x the day index and pass the old 1000000 in 2084, so N moves to 10000000. Buckets are clock hours, not a rolling window, so two opens a minute apart either side of the hour still count twice — tested, and the same property the day version had at UTC midnight. The detailed-stats fixture gave its repeat open a 60s offset, which under hourly buckets would have counted twice whenever the suite ran in the first minute of an hour. It now shares a timestamp with the open it repeats.	2026-10-14 11:54:17 -04:00
Daniel Salazar	ff815f730d	fix: harden token delivery, session revokes, app identity and fs scoping (#4229) ... - relay sign-in popups take the opener origin from the referrer only - session revokes read the primary, batch per user, and tombstone caches in every region - wisp relay tokens bound to the minting session; verify rechecks the account - hosted origins keep app identity with the site owner; deleted app uids recorded in deleted_apps - subdomain cache tombstoned on delete, update, reclaim and app/account cascades - email lookups require an exact address match after the SQL lookup - workers deploy only from files the deployer owns - share_app_data:false enforced at ACL time, plus copy/move/rename/share/recursive listing - kv remove returns no value without read; kv:expire op grant usable - fs errors name the entry instead of the owner's path - shared-by-me drops rows a delegate no longer has authority over - private apps can't move index_url onto another user's origin - GUI: default apps saved and launched by uid; save picker accepts only copy/move; permission dialog waits for focus; Publish as Worker hidden on others' files	2026-10-14 06:56:44 -07:00
404oops	f9cde23937	Merge branch 'filipkujundzic/put-1954-elevenlabs-tts-passes-voice-names-through-as-voice-ids-so' into filipkujundzic/put-1907-ai-streaming-buffering-and-error-shape-cleanup ... # Conflicts: # src/backend/drivers/ai-chat/utils/inlineImages.test.ts # src/backend/drivers/ai-chat/utils/inlineImages.ts # src/backend/drivers/util/fileInput.ts	2026-10-12 12:39:35 +02:00
Daniel Salazar	d8f9aff3dd	fix: follow-ups to #4161–#4203 (rate limits, broadcast, AI routing and billing, fs subtree walks) (#4205) ... * fix(rate-limit): count the newest KV rate rows first The KV backend's rate check, peek and concurrent acquire list the key's rows with a limit + 1 cap that the store applies before dropping expired rows, so unswept rows from past windows (oldest keys) could fill the fetch and the live count read as zero. They now list newest first. * fix(broadcast): split webhook flushes under a byte budget, per-peer backlogs Flushes go out in bodies of at most `broadcast.outboundMaxBodyBytes` (default 4 MiB) instead of one POST, and each peer keeps its own backlog of what it missed, capped by count and four bodies' worth of bytes, oldest dropped first, so one failing peer can no longer grow every peer's body past their 50 MB limit. * fix(ai-chat): classify dropped upstream connections as provider failures A connection reset or refusal (SDK APIConnectionError, Mistral's ConnectionError, an undici reset on the cause) carries no status, so a chain of them fell through to 500 internal_error. Attempts now record connectionFailed; a chain of only resets and 5xx answers 400 upstream_provider_unavailable, and a reset counts as an upstream signal in a mixed chain. * fix(ai-chat): keep SDK retries on Anthropic count_tokens The Claude client is built with SDK retries off so the driver can fall back, but count_tokens has no fallback route. It now passes sdkRetryOptions, like the OpenAI moderation call, so a transient 429 or reset is retried. * fix(ai-chat): mark a route unhealthy for 60 s after a transient failure With SDK retries off, one 5xx, 429, timeout or dropped connection kept a route marked for 10 minutes, sending that model's traffic to resellers. Those failures now mark the route for 60 s; bad credentials, an exhausted account and other route-level failures keep 10 minutes. * fix(metering): don't memoise spent addons inside the force-exact window The spent-addons memo could cache a pre-grant record read from a lagging replica right after a credits-changed announcement, so overage skipped the new credit and requireCredits kept answering 402 for up to 15 s. A spent record is now only cached once the actor's force-exact window has ended. * fix(share): send the digest entries a flush claimed when another take fails Digest entries are taken concurrently, and a single rejected take threw away every entry the other takes had already removed from KV. The flush now logs the failed takes and sends what it claimed; an entry whose take failed stays queued for a later flush. * fix(fs): announce an overwriting move's replaced entry only when it took its name /move and v2 /fs/move looked up the entry at the destination name before moving and always announced its removal. A move into someone else's Trash renames instead of replacing, so the owner's desktop hid a live entry and legacy /move reported it as `overwritten`. Both routes now pass the target on only when the moved entry kept the name it was looked up by. * fix(ai): name the vendor in upstream error fields from media providers upstreamFetch now takes the provider id plus a display label, and UpstreamResponseError carries the id. translateProviderError copies a string `provider` off the error into `fields`, and the BytePlus video and Cloudflare image errors set one, so upstream_auth_failed and upstream_credits_exhausted alarms say which vendor failed again. * fix(teams): delete a provisioned seat whose home folders could not be created provisionUser logs a failed default-folder creation and still returns the account, so provisionAccount went on to add the membership, issue a password, mail the seat and emit team.account.created for an account with no home. It now cascade-deletes the new user and returns 500 internal_error before any of that. * fix(ai-chat): load model catalogs without holding up server start onServerStart awaited every gateway catalog (up to its 10s fetch timeout) while HTTP was already listening, so drivers started after it had no providers yet and chat requests found an empty model map. The first build now runs in the background; complete(), models() and token counting wait on it instead of failing. * fix(ai-chat): serve direct models while gateway catalogs load at boot The first model-map build now publishes a fresh map as each provider's catalog lands, so static catalogs are routable at once. complete() and token counting wait for the full first build only when the model, or the provider asked for, isn't in the map yet. Refreshes still swap once. * fix(ai-chat): split gateway-billed cost across input and output A completion billed at the cost OpenRouter, Infron or Neuralwatt reports was quoted to `ai.prompt.cost-calculated` as all input and no output. The ledger still records one `billedUsage` line; the reported per-key costs now spread that bill over the usage keys at the catalog's relative rates, in whole µ¢ that sum to it. Neuralwatt energy units take no share. * fix(ai-image): hold Replicate credits for the whole prediction The output and input-surcharge holds took the default 10-minute TTL, the same as the prediction window alone, so a slow prediction could outlive them once creation and input measurement were added. They now last 15 minutes, like the video job holds. * feat(ai-chat): report what a completion was charged as usage.cost Chat usage (the non-stream result and the stream's final usage line) now carries `cost`: the µ¢ the ledger records for the request after the cost factor, from the request's recorded charges and the gate's factor lookup. `usd_cents` is unchanged. A hooked recording now copies the usage object first, so keys the driver adds later never land in the ledger. * fix(ai): charge streamed speech once its first byte arrives Per-character TTS and the voice changer recorded their charge when the upstream's response headers arrived, so a body that failed before any audio was billed in full. A shared relay now charges on the first byte, or when the caller hangs up, and charges nothing for an upstream that fails or ends before sending audio. * fix(ai): release credit holds only after the request's usage lands A hold was released as soon as its operation returned, while the usage it stood for was still being written (behind the cost-factor lookup), so a concurrent request briefly saw neither. AI metering now tracks the request's usage writes and its holds (chat included) release once those settle, without delaying the response; a streamed charge chains its own. A failed write still releases the hold, and the hold's TTL backstops it. * test(ai-chat): wait for a stream's credit hold to come back after its usage lands * perf(fs): page subtree delete and copy by parent link, not by depth over the path range Each page of a recursive delete or copy sorted the whole remaining subtree by a computed depth, so a walk cost grew with the square of its size. It now pages the children of up to 100 sibling directories at a time by parent_uid, depth first, keyset on (parent_uid, id), so a page reads only its own rows; rows with no stored path are reached too. SQLite gets the parent_uid index the others have. * fix(fs): find children by parent_uid in the empty check and supplementary-name lookup Rows the old backend wrote carry parent_uid and no parent_id, so a non-recursive delete of a directory holding only such rows passed the empty check and orphaned them, and on MySQL the supplementary-character child lookup missed them, leaving their subtrees out of listings, counts and renames. Both now key on parent_uid; the empty check still probes parent_id too, which the delete cascade follows. * docs(fs): drop the single-POST note from the entry broadcast cap	2026-10-12 01:10:54 -07:00
Daniel Salazar	ecef154630	refactor(services): page fs subtree walks, gate share fan-out, one fs ACL helper (#4190) ... * refactor(acl): read ancestor chains from FSEntryStore and share one fs ACL helper getAncestorChain (split a path, batch-read the entries) moves to FSEntryStore, with getAncestors for callers that want the rows. Services read it from the store instead of calling FSService sideways; FSService.getAncestorChain stays for drivers and extensions. ACLService gains fsDescriptor (memoized chain), assertFsAccess, assertFsCreate and refusal, plus the pure aclRefusal mapping. The eight hand-rolled descriptor and safe-error copies in FSService, ShareService, events/authorization, FSController, legacyFsHelpers, LegacyFSController, AuthController and WebDAVController use them. Each route keeps its status, legacyCode and message: fs routes answer access_denied on a 403, the legacy routes still hide refusals from apps as 404, share and events keep forbidden and their own 404 text. * perf(fs): page recursive deletes and batch their event dispatch A recursive delete read every descendant into memory, and an empty-check for a non-recursive one did the same. It now walks the subtree a page at a time in descending path order (children before the directory holding them, so the parent_id cascade never takes a row before it is handled), deleting objects and rows and publishing each page as it goes. A non-recursive delete probes with one LIMIT 1 query. Removals publish once per entry, after the row is gone (the same change as the open fix for the double emit), with one surviving-ancestor chain read at most once for the whole subtree. Events-API dispatch for a batch runs as one background task at bounded concurrency instead of a fire-and-forget promise per entry. FSEntryStore gains hasDescendants and listDescendantsInOrder (keyset paged by path, then id). * perf(fs): copy directories a level at a time A directory copy walked every descendant in memory and copied them one by one: an insert per directory and, per file, an S3 CopyObject, a batchCreateEntries of one and its read-back. It now pages the source in ascending path order (parents before children) and takes each page a depth at a time: rows for directories, links and empty files at bounded concurrency, then the depth's S3 copies at bounded concurrency and one batchCreateEntries for all of them. A single-file copy goes through the same path. A failed S3 copy still fails the call (a ghost file still answers 404), but only after the objects that did copy have their rows, so nothing is left in the bucket unreferenced. Copied entries publish per depth in one dispatch. * refactor(share): build masked share paths in one place sharePathMask exports maskedRoot(owner, uuid, name) and maskUnder(root, realPath). ShareService's maskedPathVia goes, its maskedSelfPath and shareDeepLink's maskedSharePath build on them, and the masker, maskUnderAnchor and learnShareRoots use maskedRoot. Output is byte-for-byte the same; tests pin it. * fix(share): send old_path on a recipient's item.moved The desktop's item.moved handler reads old_path; ShareService sent the former masked path only as from_path. Both are sent now. * perf(share): skip the audience lookup for owners with nothing shared Every fs write, create, rename, move and delete looked for share recipients: an ancestor walk plus two share queries, for an owner who in nearly every case has shared nothing. #sharesReaching now reads the chain through FSEntryStore.getAncestors (no third copy of the walk) and asks a per-owner "has anything shared" flag before the share queries. The flag is the per-user cache events already keeps for subscriptions (SubscriptionCache): one ShareStore.ownerHasShares read per owner per TTL, coalesced across a burst. Creating a share (user, team, claimed invite) or moving a subtree to a new owner bumps it locally and over outer.pubsub.share.ownerShared; the TTL covers a process that misses the bump. A failed lookup reads as shared, so the full lookup decides. * perf(share): batch the per-row reads behind share listings Share listings looked things up one row at a time: - teams by id: TeamStore.getByIdsIncludingDeleted (the single-id read calls it) for listSharedByMe, listSharesOf and #revokeGroupSharesBy - team shares per node and ancestor: ShareStore.listGroupOnFsentries, one query for the chain, in the same node-then-ancestor order - #liveGroupGrants: PermissionStore.queryIssuerGroupPermsByPrefixes reads every issuer/group pair under every prefix in one query - #revokeGroupSharesBy: the five canManagePermission checks per row run together instead of one after another - ShareNotificationService #unblocked: users via getByIds and blocks via UserBlockStore.blockersAmong, two queries instead of two per member - digest flush: entries are taken at bounded concurrency rather than one at a time * perf(events): ask background consent once per holder and app on a revoke settle Withdrawing events:background re-checked consent row by row, resolving the holder's actor and asking the permission service for every one of their background rows. Rows of one holder and app stand on one consent, so it is asked once and shared. * refactor(events): drop the registry's unused push fields notify, defaultDelivery, PushMessage, NotifyProjection and pushProjection were read only by registry.test. Every subject was unpushable and broadcast by default; the subscription's delivery class stays in DeliveryClass, which the stores use. * refactor(share): retire trashed shares through onEntryOwnerChanged onEntryTrashed was a copy of onEntryOwnerChanged: the same subtree share sweep, grant retire and row delete. * fix(fs): page subtree walks by depth, not path MySQL sorts a PAD SPACE string on its first max_sort_length (1024) bytes, so under utf8mb4_unicode_ci paths past about 512 characters tie and fall back to id. A file moved under a newer directory then sorted after it: the delete walk reached the directory first and the parent_id cascade took the file without its object delete or event, and the keyset skipped or repeated rows. Pages now order by segment count, then id, which no collation can reorder. * fix(share): cache an owner as shared when the bump arrives A bump only dropped the cached answer, so the next lookup could read a replica that hadn't applied the new share yet and cache "nothing shared" for another TTL. The bump now records "shared" outright. * fix(fs): probe a directory for children by parent_id, from the primary * fix(fs): keep the copied-size check in the batched directory copy A copied file records the size CopyObject actually duplicated; growth past the source row is charged against the allowance, and a copy that no longer fits is deleted and refused, as a single-file copy already does.	2026-10-11 21:54:13 -07:00
Daniel Salazar	c815dc82a1	refactor(core-util): shared bounded map, cheaper route lifecycle, DDB/DB/rate-limit/gate dedup (#4185) ... * refactor(util): add BoundedTtlMap for core caches and log throttles One size-capped, LRU-evicting map with an optional TTL and a once-per-window shouldEmit helper. Replaces the hand-rolled maps in the DynamoDB throttle and repair loggers (the repair one was unbounded), the Slack repeat throttle, the auth probe's reauth log throttle, and the card-verification status memo. * perf(http): skip route lifecycle work when nothing listens The per-route lifecycle middleware built a payload, opened an event.emitAndWait span and hooked finish/close on every request, listeners or not. Each phase now checks hasListeners first. EventClient's three copies of the key-prefix walk become one memoized #matchKeys. * perf(ddb): write batch chunks concurrently and retry the TTL sweep batchPut/batchDel sent their 25-item chunks one after another; they now run up to four at a time through runWithConcurrencyLimit, and a failed chunk stops the ones not yet started. batchGet and the batch write share one #sendWithUnprocessedRetry instead of two copies of the backoff loop, and the TTL sweep goes through #batchWrite, so the unprocessed deletes it used to drop are retried. The AWS and local clients are built by one #bindClient. * refactor(db): share migrations and tryHardRead in the base client MySQL and Postgres each carried a copy of the migration-path runner and of the replica-racing tryHardRead; both now live on AbstractDatabaseClient (applyMigrationPaths, and tryHardRead gated on hasReadReplica), and SQLite drops its own tryHardRead override. Removes MySQL reinitPrimary, reinitReplica, isRetriableError and readWithRetry and the base requireRead, which had no callers outside tests. SQLite prepared statements and the Postgres SQL rewrite are memoized per query text in a bounded map. * refactor(rate-limit): route every entry point through one charge() The nine rate/peek/acquire entry points each repeated key and limit resolution, the backend call and the fail-open catch. They now share charge(), routeKey() and guardSlot(); log lines are unchanged. Driver concurrency slots release at most once like the other slot handles, and checkRateLimit takes { failOpen: false } for limits that must hold when the backend is down. * refactor(util): add secretsEqual for constant-time secret comparison One HMAC-then-timingSafeEqual helper, safe for inputs of any length, so callers stop hand-rolling the length guard. File signature verification uses it. * refactor(fs): infer signed-file MIME types with mime-types mimeFromName looks extensions up in mime-types instead of a 35-entry inline table, keeping the two types the table reported differently (ico, wav). The purpose-labelled signing key is derived once per secret instead of on every signature, where signFile derived it twice. * refactor(http): build route gates from one gate() helper Each gate repeated a try/catch -> next(err) wrapper or a .then(next, next) chain around an assertion. gate(check) does that once: a throw or rejection goes to next(err), and a synchronous check still calls next synchronously. The credits, subscription and reputation gate files held nothing but such a wrapper; their factories move into gates.ts and the files go. * perf(hosting): build the puter-site hosting config once createPuterSiteMiddleware rebuilt the normalized hosting-domain config on every hosted-site request; it is derived from static config, like the domain lists the middleware already builds at construction. * fix(db): tryHardRead's replica leg doesn't fail over to the primary MySQL's read() retries on the primary after a load-shed or transient replica error, so routing tryHardRead through it read the primary twice on that path. The replica leg now reads the replica alone. * fix(ddb): settle in-flight batch chunks before rethrowing A failed chunk stopped unstarted chunks but rejected while others were still retrying, so writes could land after the caller had handled the error and invalidated its cache.	2026-10-11 21:53:33 -07:00
Daniel Salazar	906d6b7886	fix(email): one canonicalization and blocklist for every address path (#4176) ... Share invites, share notifications and app feedback checked addresses through EmailClient.clean/validate, a second copy of the rules in util/email.ts that stripped dots on iCloud, stripped `+tags` on every domain, and matched the blocklist case-sensitively. They now use util/email.ts: isBlockedEmail for the blocklist, abuseKey for the per-address notification budget, and cleanEmail for the invite digest key. EmailClient keeps transport and templates only. isBlockedEmail also lowercases the configured suffixes, so a mixed-case blocklist entry still matches.	2026-10-11 18:24:28 -07:00
Daniel Salazar	cf9df8719a	fix(alarm): forget idle alarms and cap how many the client holds (#4171) ... AlarmClient kept every alarm id it had seen for the life of the process, and ids that embed a user, key or invoice grew it without bound. The registry is now ordered by last occurrence: alarms idle for 24 hours are dropped on the next create, and past 1000 the least recently raised goes first. Clearing or evicting an alarm only removes its short-id alias when the alias still points at it.	2026-10-11 17:54:26 -07:00
Daniel Salazar	b67acdbf6f	fix(stores): return written rows without reading them from a lagging replica (#4166) ... * fix(stores): read rows back from the primary after writing them ShareStore (create, upsertPending, upsertActive, upsertActiveGroup, applyPending, upsertAnyone), AppStore (create, createFromOrigin, update, claimOwnership), NotificationStore.create, SubdomainStore.update and NotificationDriver.create returned the row they had just written by reading it back through `db.read`, which goes to the replica. Under replica lag that came back null or stale. Those read-backs now go to the primary, through a `primary` option on the getters. AppStore caches the primary copy on create, and claimOwnership no longer reads through getById, which cached the replica's pre-claim row on a miss. * fix(stores): skip the notify read-back; read OIDC link conflicts from the primary NotificationStore gains `insert`, which writes and returns the uid; `create` is now `insert` plus the primary read-back. NotificationService .notify ignores the row, so it uses `insert` and no longer pays a primary round trip per recipient. NotificationDriver.create still reads the row back from the primary. OIDCStore.link reads the conflicting row from the primary after a unique violation, so a replica that has not seen another account's link can no longer make the call report success. * perf(stores): return written rows instead of reading them back ShareStore.create, NotificationStore.create, AppStore.create and AppStore.createFromOrigin build the row they return from the values they inserted and the insert id, instead of reading it back from the primary. The DB-defaulted creation timestamp is now written explicitly from unix seconds (`timestampFromUnixSql`), so the returned value matches what a read returns (`timestampValue`); postgres INSERTs add RETURNING id. The AppStore rows fill every column the INSERT leaves to the schema and are cached as before. ShareStore.upsertPending's re-invite branch selects the full pending row in its dedupe read and returns it with the new mode and data; it falls back to the primary only when the UPDATE matched nothing. The remaining primary read-backs stay and say why: the share upserts and the new-invite insert (an upsert's surviving uid/created_at and the entry-owner subquery are only known to the primary), applyPending and claimOwnership (only an id or uid in hand), AppStore.update and SubdomainStore.update (the base row is a replica copy that would be cached), NotificationDriver.create (notify returns only the uid) and the OIDC link conflict.	2026-10-11 17:51:38 -07:00
Juan Fernando Castro	50bcdc9c6c	fix: let a revocation be seen by the reads and the nodes that outlive it (#4151) ... * fix: let a revocation be seen by the reads and the nodes that outlive it Four places where a revoke lands on the primary and the next reader can still be told the old answer. The u2a cache key was dropped but not re-filled, so the next miss read a replica that may still carry the row and cached it for the TTL -- and a revoke's own settling pass is usually that miss. It re-fills from the primary, as the u2u side already does. A revoked session dropped its keys the same way. It now leaves the revoked row in the cache, read from the primary, so there is nothing for a lagging replica to re-introduce. A link share is its own authority rather than an index over a permission, so a replica still carrying a withdrawn row answers with it. For a short window after one is withdrawn, those reads go to the primary. Socket eviction rode a local event, so it only reached the node that did the revoking. It is announced on the fan-out channel too, which is where the node actually holding the connection hears it. * refactor: carry the revocation on outer.* instead of outer.pubsub.* Eviction runs through `io.in(room).disconnectSockets()`, which goes via the socket.io Redis adapter a cluster shares, so one node handling the event disconnects that user on every node in it. Fanning out to siblings had each of them repeat the same cluster-wide disconnect. The adapter's Redis is per-cluster, so peer clusters still need the hop -- `outer.*` reaches them and skips the sibling fan-out. The local `auth.sessions.revoked` goes with it: `outer.*` fires on the emitting node too, and nothing else subscribed.	2026-10-09 16:36:26 -04:00
Juan Fernando Castro	7de8360a6e	Sharing cleanup: bounded outbound listings, and what see does not reach (#4118) ... * fix: keep a file name to one line in a subject, and the inbound total to the account **A name reaches the digest subject as the user typed it.** Subjects compile with `noEscape` on purpose -- a header is not HTML, and escaping would show entities to the reader -- and the comment there says free-form values collapse whitespace upstream. A file name does not: `#assertUsableName` refuses only an empty name, a slash, `.` and `..`, so a name may carry newlines. It is collapsed to one line and bounded before it goes in. **`includeTotal` answered an app with the account's figure.** The page is filtered by what the caller can reach -- an app sees only what its user handed it -- while `countByHolder` counts every inbound share. The docs already call `total` an approximation for the filtering; they did not cover an app reading the size of everything else. It is answered only for a caller acting as the account now, and the page documents that. The team-share fixture built its actor as `{ user }` with no `effectiveApp`, which reads as an app rather than a session. A session acts as the account, so it says so. * fix: stop `see` reaching inside a folder, and bound mode changes **A `see`-only folder recipient was pushed every child's name, size and type.** The fan-out picked holders by reach -- any live grant on a reaching root -- and `see` is a grant. But `see` says a node is there; `list` is what says what is inside it, and the payload carries exactly what `list` would: name, path, `is_dir`, type, size, modified. Descendants now go only to a holder whose mode reaches inside. A change to the shared node itself still goes to everyone who holds it, which is what `see` covers. **Mode changes skipped the daily quota entirely.** Not spending share budget is right -- changing a mode is not new reach, and the existing test says so -- but unbounded it lets one account write a pair as fast as it can ask. They have their own budget now, ten times the share limit by default and configurable, so the exemption stays an exemption from *that* budget rather than from any. * fix: let a half-finished downgrade be finished, and stop Trash answering **A mode downgrade that failed between its two halves could not be repaired.** `setUserUser` grants the new mode and then revokes the superseded ones. If the revokes did not all land, the higher grant stood -- and the retry returned early, because the mode it was asked for was now present. "Nothing to do" now means this mode is the only one standing, so asking again is what finishes it. The same two halves, and the same early return, were in the group variant. **Trashing someone else's item answered for what is in their Trash.** Moving into the owner's Trash is deliberately authorized without write on it -- that is what lets a recipient delete from a shared folder -- but a name collision there came back as `An entry already exists at /owner/Trash/<name>`. The mover picks the name, so that is a question they can ask about a folder they cannot list. It dedupes instead, which is also what the GUI wants: trashing should not fail over a name. * perf: record the entry's owner on the share row The outbound listings answer "shares on things I own that someone else issued" by joining `share` to `fsentries` and filtering on the owner there. Neither side of that join bounds the work: driven from `fsentries` it walks every file the user owns, driven from `share` it walks every share until it has a page. For a user who owns a lot and delegates nothing, that is the whole table for an empty result. `#outboundAppsSql` is worse than the ticket says -- it has no LIMIT at all, so every call aggregates the join across both tables. The owner now rides on the row, indexed with `id`, so all three read it directly. An entry cannot change owner -- a move out of the owner's tree is refused -- so it is written once, with the row. Set in SQL from `fsentry_id` rather than threaded through callers: four insert sites, each of which already has the id, and none of which can now forget. The risk in denormalising is a write path that does: a row with no owner recorded is a share that silently stops appearing in its owner's listing. The new test covers exactly that -- it fails when one insert is made to write NULL. Verified against a real boot: a pre-existing share row, on a database that genuinely predates the column, comes out with the owner its entry has. * fix: the review's findings on this branch **The Trash collision still had both halves open.** `overwrite` was checked before the trash case, so a caller setting it -- it is wire-supplied -- could still delete whatever it collided with in the owner's Trash, and the 409 survived for anyone who did not. The trash case is decided first now. Dedupe alone did not close it either: a name that changes only when there is a collision answers the same question the refusal did, and the mover sees that name even though the path is masked. Trashing someone else's item renames it every time, so the result carries no signal. `original_name` already records the real name, which is what a restore reads. Two tests pinned the old name; they say what it is now. **The owner column had no fallback.** Reads went only through it, so a row written by anything that predates it -- a pod mid-roll -- would never appear in its owner's listing again. The three reads now also cover `IS NULL` rows through the old join, indexed on the null and empty once none remain. **The grant became conditional on a replica read.** `if (!already)` skipped it on a stale row while the revokes below stayed unconditional, which could leave the holder with nothing where the old early return left their access alone. It grants unconditionally again; only the early return changed. **The re-issue budget covered one path of four.** Team, link and invite re-issues reserved nothing, so three ways round it were left open. They share one `#reserveRemodeQuota` now. Two docblocks had been pushed off the functions they describe, and two comments still described the join that is gone. * chore: one-line the comments added here * test: measure the prefix walk by what it keys on, not by the clock `checks many long paths without re-reading every prefix of each` gave the walk 1500ms of wall clock. It measured nothing useful: the shape it guards against — keying each step on the accumulated prefix text rather than on the parent id — costs 29ms against the current 8ms on that input, so both ends sit far under the bound. Under coverage the same test ran 1511ms and failed, which is the only thing the number could ever report on. `PathPrefixIds` now counts the characters it keys on, and the walk is asserted on that: proportional to the input, and twice the depth costs twice as much rather than four times. Keying on the accumulated text fails both. The original case keeps its real assertion, that a request this size is refused. * fix: keep the recorded entry owner current, and the trash guard on the retry A move into someone else's tree re-owns the whole subtree, which the migration comment claimed could not happen. The shares on it are re-pointed at the new owner, or they drop out of the listing the column keys and stay in the old owner's. The unique-violation retry re-entered the collision resolver without the trash flag, which is the one call site the guard was not threaded through.	2026-10-09 16:14:23 -04:00
Juan Fernando Castro	cf574e4b34	Merge pull request #4110 from HeyPuter/juancastro/icloud-dot-significance ... Treat dots as significant on Apple's mail domains	2026-10-07 17:11:59 -04:00
Daniel Salazar	2e275a8f6b	fix: godmode installation (#4116) ... * fix: godmode installation * fix: godmode classifier * fix: vitest	2026-10-07 13:16:10 -07:00
Juan Castro	0555ea6479	Merge branch 'main' into juancastro/icloud-dot-significance	2026-10-07 11:01:12 -04:00
Juan Castro	4b5f509066	fix: only rewrite rows the old rule actually wrote, and fix the invites too ... Two findings from review, both of which contradict what the last commit claimed. **The backfill could merge keys, not only split them.** I asserted it "can only split rows apart, so it cannot collide". Not true, and reproducible: a row predating the `clean_email` column has a NULL there, so its index key is `lower(email)` -- filling it in folds `+tag`, and `john+news@me.com` lands on a key `john@me.com` already owns. A legacy row holding an un-lowercased `clean_email` collides the same way once lowercased. Either throws, and the two engines that re-run every file on every boot would then refuse to start until someone operated on the database by hand. Both updates now fire only where the stored value is exactly what the old rule produced and differs from the correct one. That makes "splits only" true: the sole change is dots coming back. NULLs and legacy values are left alone. The same predicate makes them write-idempotent. Postgres has no no-op-update suppression, so without it every boot rewrote a tuple per Apple row forever. **The pending-invite residue misdelivers rather than just going missing.** `claimPendingShares` looks the row up by the canonical address, so whoever confirms `jsmith@icloud.com` claims the invites addressed to `j.smith@icloud.com` -- someone else's file, handed over on confirmation. I had written this off in the PR as unfindable invites that "expire on their own"; `share` has no expiry column and the lookup has no time predicate, so they are claimable indefinitely. Fixed here rather than later: the typed address is kept as `invitedAddress` whenever it differed from the canonical, which is exactly the collapsed rows, so the invites recompute from it under the same guard. Verified by seeding a database at the old version with the two rows that used to abort the migration, and booting: it reaches version 87, repairs the collapsed row, and leaves the NULL and legacy rows untouched. Replaying the statements changes zero rows.	2026-10-07 10:23:30 -04:00
Juan Castro	ece8ad5f1d	fix: register the Apple-domain backfill with the sqlite runner ... The sqlite migration list is hand-maintained, so the file alone did nothing — a fresh boot still reported 87 files and stopped at version 86. MySQL and Postgres read their directories, so only this one needed the entry. Caught by booting a local instance rather than by the suite: the client tests pin `CURRENT_SCHEMA_VERSION`, which fails once the entry exists but says nothing while the file is simply unregistered. Verified against a seeded database at the old version. `John.Smith@iCloud.com` went from `johnsmith@icloud.com` to `john.smith@icloud.com`, `a.b+tag@me.com` from `ab@me.com` to `a.b@me.com` — subaddressing still folded — and a gmail row was left alone.	2026-10-07 09:45:26 -04:00
Juan Castro	cf817b0f0a	fix: treat dots as significant on Apple's mail domains ... `cleanEmail()` applied gmail's `dots_dont_matter` rule to icloud.com, me.com and mac.com. Apple allocates the exact string: `j.smith@icloud.com` and `jsmith@icloud.com` are two mailboxes with two owners. Collapsing them made one canonical address stand for both. What that canonical address feeds: - `findEmailOwner` falls back to it, so an OIDC sign-in whose verified claim is the dotted address resolves to the account holding the undotted one, and `linkProviderToUser` has nothing left to object to -- the provider verified a real mailbox and the account's address is confirmed. - The unique index on COALESCE(clean_email, lower(email)) reads it, so the rightful owner of a dotted Apple address is refused signup because the undotted account already "owns" it. - Pending share invites are stored and claimed on it. Subaddressing is untouched: Apple does fold `+tag`, so that rule stays. Gmail keeps both rules -- there the fold is correct. A migration recomputes `clean_email` for those domains on all three engines, since rows written before this still hold the collapsed value and the index reads the stored one. Verified the SQL against the function rather than by inspection: same five inputs through `cleanEmail()` and through sqlite give byte-identical output. It can only split rows apart, never merge them, so it cannot collide with the unique index. Not changed: `findUserByEmail` still matches on the canonical address. The cross-match it documents is correct where the provider really does fold -- `foo.bar+tag@gmail.com` is the same inbox as `foobar@gmail.com`. Requiring an exact match would break that for no benefit here; the fix is for the data to say what the provider does.	2026-10-07 09:17:03 -04:00
Juan Castro	6efb96d560	Merge branch 'juancastro/flat-permission-view-scope' into juancastro/permissions-access-control-cleanup	2026-10-07 09:00:40 -04:00
Daniel Salazar	11e48c68a9	fix: verification for sms (#4101)	2026-10-06 22:03:07 -07:00
Daniel Salazar	464f758a97	fix: verification for sms (#4100)	2026-10-06 22:00:14 -07:00
Daniel Salazar	d2cdde5147	fix: verification for sms (#4098)	2026-10-06 19:36:18 -07:00
Juan Castro	ad1be70d06	Merge branch 'juancastro/flat-permission-view-scope' into juancastro/permissions-access-control-cleanup	2026-10-06 19:44:37 -04:00