85 Commits

Include renames

Author	SHA1	Message	Date
Daniel Salazar	dd144b6b64	feat(notifications): notify through KV shares, group replacement, fewer push prompts, and events/worker fixes (#4432) ... * feat(perms): add a pushNotifications resource to puter.perms.request notifications:push gets a resource like backgroundEvents, so it can be batched with other access behind one prompt. requestPermission() and hasPermission() ask through it. * fix(puter-js): read puter.gui_features once at load The desktop's feature list was read from location.search on each call, so an app whose router rewrote the URL lost it and enablePush() and the other desktop dialogs answered as if the desktop lacked them. * feat(gui): ask for notifications:push when an app calls enablePush without it enable() and enablePush() inside the desktop answered unavailable for an app without the push permission. They now show the permission dialog, whose Allow also starts the browser registration, and answer with the browser's push state once it settles, or off when denied. A browser that blocks notifications or has no push service still answers without a dialog. * fix(kv-share): charge mint and grant budgets only for share calls that change something A share call now spends the per-call mint budget and the per-recipient grant windows after it has looked up which recipients already hold the share. A call whose recipients all hold the share at the requested level and label spends neither. Recipients without the share, including unknown usernames and refused recipients, still spend a grant unit each. Every named recipient spends from a new 1,200/min lookup window before any lookup runs. * perf(events): skip the subject split for rows not made through a share handle kvHandleFromSubject checks the stored subject's kv:kvh- prefix before splitting it, so own-namespace and fs rows return null without parsing. * feat(notifications): resolve enable() on a website before the confirmation push enable() waited up to 30 seconds for the confirmation push, so a prompt the same click started next had lost its activation. On a website it now resolves once the browser answered and the server has the subscription, records the opt-in, and finishes the confirmation in the background. A failed confirmation goes to onPush's onError and the next load's resync tries again. disable() and sign-out stop a confirmation still waiting. The GUI keeps waiting, since it reads the registered device right after. * fix(notifications): show a page's own notifications only while push is on A hidden page with an onPush handler kept showing notifications after disable(), and its receipts kept claiming showsOwn, which held pushes back from the user's other devices. Both now need the user's opt-in, cached in the page, refreshed on enable, disable, sign-out and each sync, and re-read when another tab of the site changes it. * fix(events): leave subscriptions suspended for good out of the durable quota countForHolder no longer counts rows suspended with a reason that never resumes (permission_revoked), in both the pre-insert check and the recount after insert. Rows suspended for a resumable reason still count. The store now owns the list of terminal reasons, and isResumable reads it. listActiveForHolder with includeSuspended returns only rows suspended for a resumable reason, so its quota-sized read still covers every row a revocation has to settle. Each holder keeps at most 500 terminal rows; a subscribe past that deletes the oldest ones. * feat(notifications): add status() to read whether pushes reach this browser status() resolves 'on', 'off', 'blocked' or 'unavailable' without asking anything. On a website it reads the notification permission, this browser's subscription and the user's opt-in. Inside the desktop a new pushStatus IPC, advertised as push-status, answers the browser's push state, 'on' only while the app holds notifications:push. * feat(notifications): disable() inside the desktop revokes the app's push permission disable() did nothing inside the desktop. A new disablePush IPC, advertised as disable-push, revokes the calling app's notifications:push grant through the same route Settings uses, so its notifications stop going out as pushes and a later enable() shows the permission dialog again. * feat(notifications): allow @ in a notification group, up to 128 characters A group is an opaque display tag, hashed for the Web Push topic and used as the notification tag, so it can carry identifiers such as an email address. * docs(notifications): say when a page with an older Puter.js holds a push back * fix(notifications): let disable() win over an enable() still in flight An enable() whose subscription reached the server after a disable() went on to record the opt-in, so the next page load turned pushes back on. It now rejects with push_verification_canceled and records nothing. * docs(notifications): list push_verification_canceled and reword sentences that led with code * fix(events): carry KV values through a share handle only at read or write The delivery re-check now reads the share's level off the live handle along with whether it still holds. A row on a watch share loses its includeValue for that delivery, so its events name the key without the value; the owner's own rows on the handle keep theirs. Decided per delivery, so raising or lowering a share applies to subscriptions already made, and subscribing with includeValue at watch is still accepted. The live-updates recipe now shares at read, which its events need to carry the score. * fix(events): retry worker runs Puter could not complete without counting them Invocations that never reached the handler (transport and dispatcher errors, unmarked answers, failed or throttled deploys, no configured runtime, a bad invoke key or missing token) now report an 'unavailable' outcome. So does a handler that throws after one of its own puter.js calls got a 5xx, 429 or network error from Puter's API, which the events runtime reports as 'platform-error'. Those runs back off like retriable ones, honor the answer's retry-after up to the backoff ceiling, and no longer count toward the consecutive-failure suspension. A run past the invoke timeout still counts. * test(notifications): cover the pushNotifications resource and status() in the API suites * test(gui): Allow on a batch that includes notifications:push turns push on * fix(events): hold a share-handle row's backlog when its share can't be checked The share-handle delivery check now tells a share that is gone (handle not live, not the row's holder, identity gone, grant withdrawn) from one it can't check right now (a lookup that throws, an owner who is suspended). Only the first settles a queued row as permission_revoked and purges its backlog. The second delivers nothing, leaves the row in service with its backlog, and sends it to the back of the sweeper's line, so delivery resumes once the lookup recovers or the owner is unsuspended. A revocation settle skips such a row and leaves it to the delivery check. kvSharedRegionAuthorized returns null when the permission check throws, and that answer is not cached. * feat(events): update a persistent subscription's context in place POST /events/subscriptions/update replaces a durable row's context, with the same validation and 4 KB cap as subscribe, keeping its subId and backlog. The row's owner region cache is rebuilt from the table and the generation bumped so every region delivers the new value. It spends the subscribe call budget. puter.events.update(subId, { context }) wraps it; null clears the context, and a function handler this client registered for the subscription gets the new ctx. * fix(events): serve a fan-out's holders in turns and cap values per delivery #route now orders an event's candidate rows before any cap applies: the keyspace owner's rows first, then the other holders taking turns, one row per holder per round, starting from a random holder each event. The filter evaluation cap, the delivery cap and the gap markers all count in that order. Values no longer drop for every delivery once more than 128 rows match or the filter evaluation stops early. The first 128 deliveries that asked for the value carry it, in serving order, and later ones carry the key alone. EVENTS_KV_VALUE_OMIT_MATCHED_SUBSCRIPTIONS becomes EVENTS_KV_VALUE_DELIVERIES_PER_EVENT. The caps themselves are unchanged. * fix(email): file inbox mail as a delivery, not the request actor's write storeInboxMessage and copyIntoInbox now run their filesystem writes with a platformWrite context flag, and fs event dispatch leaves actingUserId unset under it. Mail filed into an account's inbox from inside that account's own request (an app mailing the signed-in user) is delivered with self: false, the same as external ingress. Handler depth and viaWorker are still carried from the actor. * feat(notifications): raise the default daily app message limit to 1000 The per-day default for `app.message` goes from 300 to 1000 per user and app, matching the daily push cap. The per-minute default stays at 30, and `notificationAppMessageLimits` still overrides both. * feat(notifications): look up an unseen notification by group `notification` gets a nullable `group_key` column, a sha256 hex digest of the group a row was written with, indexed on (user_id, app_uid, group_key). `NotificationStore.create` takes a `group`, and `getUnseenByGroup` returns the newest row an app wrote for a user under that group that is neither shown nor acknowledged. Rows written before the column exist never match. `updateValue` takes `onlyUnseen`, which also skips a shown row, and `refreshCreatedAt`, which dates the row to now. Migrations: sqlite 0098, mysql_mig_52, postgres_mig_41. * feat(notifications): replace an app's unseen message in the same group An `app.message` with a `group` now replaces the app's earlier notification to the same user with that group while it is neither shown nor acknowledged. The row keeps its uid and takes the new value and time. With no group, or once the earlier one was shown or dismissed, it is a new notification as before. A replacement spends a per-minute slot but no per-day slot. If the earlier row is marked between the lookup and the rewrite, the message is written as a new notification and spends a day slot then. `app.message` is groupable, so the replacement goes through `notifyUpdate`. For types kept off the desktop, `notifyUpdate` skips the desktop wire and dispatches the rewrite again under the same uid, so app subscribers and receipts see it, and `notif.written` carries `updated`. The driver's write path takes the recipient id rather than the actor, so rate limits key on the recipient. * feat(workers): report the app a worker runs as workers.create now returns app_uid and app_name for the app its token is bound to (null for a user-scoped worker), and list/get items carry app_name beside app_uid from the app rows they already load. The dev center workers table shows an App column, and the SDK typedefs and docs describe the fields and that a sandboxed worker's notifications and pushes come from its sandbox app. * feat(events): refuse handlers with free variables on REST publish POST /events/handlers/publish and /publishAll now run the SDK's own scanHandlerSource on each source before publishing and answer 400 events_handler_free_variable, naming the identifier. The backend imports the module from src/puter-js; the build marks dist/src/puter-js as an ES module package so the compiled backend can require it. Sources over the per-handler cap are left to the store's size refusal rather than tokenized. * docs(events): say Puter-side retries never suspend and stay within backlog caps * fix(events): redeploy an events worker that answers bad-key A script deployed before the internal secret rotated keeps the old invoke key, so every invocation answered bad-key and nothing redeployed it. A bad-key answer now goes through the same miss handler as a dispatcher miss (the deployer, with its hourly throttle) and is retried once with the deployed header. A throttled or failed redeploy stays 'unavailable' and backs off without counting. * perf(events): check the deploy throttle before reading handler sources EventsWorkerDeployer now reads the app's handler names and hashes first, answers no-handlers or stale from those, and checks the hourly deploy throttle before loading the sources and generating the worker. A throttled retry costs a hash read and a counter bump instead of a full source read and build, and a stale request still does not spend the throttle. The generated set is checked against the script again in case it moved between the two reads. * fix(workers): move a redeployed worker's row to the app it now runs as Redeploying an existing worker under a different app minted the new app's token but left subdomains.app_owner on the old one, so list/get and hot-reload kept naming and minting for the old app. The redeploy now sets app_owner to the binding #resolveWorkerAppBinding authorized (null for a user-scoped worker) through a new SubdomainStore.setAppOwner, since app_owner stays read-only to update patches. * fix(workers): revoke the old binding's worker session on a rebind A redeploy that binds a worker to a different app (or to none) now deletes the owner's worker sessions under that name other than the one for the new binding, through the same deleteWorkerSessionsByName path worker deletion uses (which also announces the revocation and drops access-token grants). SessionStore's deleteWorkerSessions takes an exceptAppUid filter for it, null meaning the user-scoped session. The app_owner move and the revocation now both run only after the new script deployed successfully, so a failed redeploy leaves the live script, its token and the row on the old binding. * feat(kv-share): store a notify flag on share handles kv_share_handles gains a notify column, off by default, so every existing share reads as not allowing notifications (sqlite 0099, mysql_mig_53, postgres_mig_42). The store reads it on every row, including the live handle cache and the holder listing, writes it on mint, and changes it through update(). * feat(kv-share): let the owner turn notifications on for a share POST /kv/shares and puter.kv.share() take notify, off by default. Leaving it out of a re-share keeps the stored value, and turning it on or off counts as a change, so it spends the mint budget. Turning it on for a holder who now refuses the owner's shares is refused, like a raise or a new label. The kv.share.received notice is sent only when a share starts carrying notify: a new share made with it, or an existing one switched on. Its hourly bound and the app-less namespace exclusion are unchanged. The share response, the owner's listing and the holder's listing report notify. * feat(kv-share): resolve the other party of a share for a notification KvShareService.resolveNotifyRecipient(actor, handle) returns the recipient user id, app uid and handle for a notification one party of a share sends the other through its handle. The handle must be live with an unsuspended owner, the caller must be its owner or holder acting from the app whose data it is, the share must have notify on, the holder's grant must still stand, and the recipient must not refuse shares from the sender. An unknown, retired or foreign handle, a caller who is not a party, and a caller in another app all get subject_does_not_exist. The account itself gets forbidden, a share without notify kv_share_notify_not_enabled, and a lookup that fails service_unavailable. Once every check passes it spends one unit from KV_SHARE_NOTIFY_LIMITS (30/min, 500/day per sender, recipient and app), peeking both windows first so a refusal spends nothing. * feat(notifications): hand a replaced notification to listeners again A replacement now records `replaced_at` (epoch ms) on its row, added to the same unreleased migrations as `group_key` (the sqlite file is renamed to 0098_notification-groups.sql). Its live event and its fetched copy both carry `replaced: true` and use that time as `ts`, so the pair versions the uid identically on both paths. `onNotification()` dedupes on uid plus that version: an original counts as version 0 and a replacement as its `ts`. A newer replacement is handed over again with `replaced: true`, while a copy already handed over, a replay, or an older state is dropped. Listeners started with `catchUp: false` still get replacements of what was already there. * fix(notifications): replace only the newest notification in a group The group lookup takes the newest row an app wrote under the group and returns nothing when that row was shown or dismissed, instead of falling back to an older row that is still unseen. The docs state the rule as it is: a grouped notification replaces the newest one in its group unless that was shown or dismissed. Puter.js marks neither, and the only thing that does is a tap on the push inside the Puter desktop, which dismisses it. * feat(notifications): notify the other party of a KV share through its handle `app.message` takes `to: { handle }`. The driver validates the message, then `to` must be an object with only a non-empty string `handle`, else 400. `KvShareService.resolveNotifyRecipient` checks the share and spends the pair limit, and the message is written for the share's other party in the share's app. The recipient's per-app limits and group replacement apply to their own rows, and the push is billed to them through `notif.written`. A call with `to` answers with `{ uid }` only, so nothing about the recipient's row reaches the sender. puter.js gets the `NotificationRecipient` type for `to`. * docs(notifications): document notifying the other side of a KV share notify.md covers `to: { handle }`: who can be reached, the errors, the recipient's limits plus the per-pair share limit, the uid-only return and that the recipient pays for pushes. push.md and rate-limits-and-quotas.md point at the share path. The chat notification recipe sends through each share's handle and keeps the per-member handler only for members who need to reach each other; the alert recipe says when its handler is the right fit. * fix(notifications): reveal nothing to the sender of a notification through a share A call with `to: { handle }` answers with an empty object, and puter.js resolves it with no value: a uid would show whether the recipient's last notification in a group was replaced. `notify()` gets one overload per form, and `ShareNotificationInput` is exported. Over the per-pair share limit or the recipient's own limits, such a call gets the same 429 `too_many_requests` with the same generic message, so the sender can't tell which limit it hit. * fix(notifications): serialize grouped app messages per group Grouped app.message sends take a Redis lock on (recipient, app, group) around the lookup and the replace-or-insert, so concurrent sends to an empty group write one row and spend one day slot. A lock still busy after the wait, or an unreachable Redis, sends unlocked as before. Sends without a group take no lock. * fix(events): attribute Puter API failures to the invocation that saw them The events worker runtime counted Puter API failures on the cached per-token client, so a concurrent invocation on the same token could answer platform-error for its own unrelated throw. A cached client now serves one invocation at a time and records failures and retry-after on that invocation; an invocation overlapping another on the same token gets its own client, and idle clients stay cached per token. * fix(events): drop KV values from a queued backlog once its share is at watch A queue delivery is stored with its value. Every hand-out of a stored entry (backlog drain to a socket or another region, the worker, and a declined delivery passed to the worker) now re-reads the share's level and removes the value at watch. A share that can't be checked still holds the backlog. * fix(events): return notifications replaced behind the fetch cursor The notification fetch cursor now holds the last row id and the last (replaced_at, id) it returned. A read from it merges two indexed reads: rows inserted after the id, and rows up to the id replaced after that mark. A replacement made while a client was away therefore comes back on its next catch-up. Adds idx_notification_replaced on (user_id, audience, app_uid, replaced_at).	2026-10-31 12:21:18 -07:00
Daniel Salazar	4851df716c	fix(kv): KV share lifecycle hardening (#4429) ... * fix(kv): harden the KV share lifecycle - Retire a deleted app's live shares before an origin app is registered again under the same uid. The retire runs before the app row exists and throws on failure, so a failed launch leaves no row and the next one retries. - Record deleted-app retires and uninstall cleanups (retire, then purge) as reserved items before they run. A lease keeps one node on a job; a failed run raises an alarm and backs off, and a run that hits its time budget or is cut short by a restart resumes on the next check. Each job is bounded to handles made before it was recorded. - Uninstalling with deleteData records the job and returns; the retire and purge run off the request. The purge checks whether the app was authorized again before deleting each page, so data written after that is kept. - Take a per-region lock when minting and re-check for a live handle under it, so concurrent shares of one region make one handle. - Retire the handles an account holds before the account is deleted, so the owners' subscriptions through them end instead of staying active. * fix(events): bound share-handle session rows by the per-connection cap The count across a holder's connections also counted connections that had dropped but weren't reaped yet (a crashed node, a deploy, a network blip the server hasn't noticed), so resubscribes after a reconnect could be refused with `events_subscription_limit`. Rows made through a share handle now count toward the 50-per-connection cap like any other, and with the open-connections cap per account that bounds what one holder can put in the owner's keyspace. The handle index stays for ending rows on retire. Rows are added to it with the row itself and taken out when the row is dropped, and members whose row is gone are cleared when a retire reads it. * fix(events): keep a retire's durable settle running if ending sessions fails Ending a retired handle's session subscriptions reads Redis. A failure there is now logged and the durable suspension that follows still runs, so those rows' backlogs are still purged. * perf(events): ask once per handle when a refresh screens rows, and share handle reads A socket refresh asks whether a share handle still stands once per handle, holder and app, rather than once per row: rows through one handle share its grant. Concurrent cache misses on one handle share a single read; a forget drops the shared read so later callers start a fresh one. * fix(permissions): chunk the exact-match subtree delete The holder-scoped subtree delete names every matched grant, so a large subtree could exceed an engine's limit on bound parameters. It now deletes 500 grants per statement. * fix(kv): give each holder a share of a namespace's handle budget One holder could spend a namespace's whole handle budget and starve the others. Each holder now gets at most 3,750 of the namespace's 15,000 units per 10 seconds, and the owner's own calls through a handle spend none. The holder's budget is checked before the grant lookups, so throttled retries cost none; the namespace's is only told to callers the grant admits. Multi-unit charges use a new counted claim that keeps two counters per key instead of a sorted-set member per unit. * fix(kv): refuse handles older than their app and claim cleanup jobs globally - A share handle created before its app row reads as gone: in the KV driver, in the handle cache, on the events subscribe path, and in the delegate implicator. An origin app registered again under a deleted app's uid can't reach its predecessor's shares, whatever created the row. The app-token launch no longer retires inline; it records a background retire and never waits on it or fails because of it. The clock-skew rule is shared with app sessions. - Cleanup job records now carry their lease, attempts and retry time, and a node claims a job by compare-and-set on its record. A job stays with the deployment that recorded or last ran it, and another one adopts it only after it has gone unclaimed well past due. - The job check pages through records from where the last look stopped, so waiting records don't hide due ones. After 10 failed runs a job stops retrying with a warning alarm, and its record expires after 30 days. - A mint that waits out the region lock looks for the share again and returns it, or goes ahead, instead of failing with too_many_requests. - The record-failure alarm dedups; an unfinished run is a log line. * fix(kv): hide shares older than their app from re-shares and listings After an app is registered again under a deleted app's uid, its predecessor's live handles stayed visible: sharing the same region again handed back the old, dead handle, and both sides' listings showed the old handles and labels. findLive and both listing queries now leave out a live handle created before its app row, the same rule handle resolution uses, so a re-share makes a new handle and the listings, totals included, skip the old ones until their retire lands. A new share also retires, grants and all, the handles its owner gave the same holder for the earlier app first, so withdrawing those later can't take the new grant with it. * fix(events): keep a loose cap on subscriptions through one share handle Without a count across connections, a holder could put up to 20,000 rows in the owner's keyspace (50 per connection, 400 connections), each one evaluated on every owner write. The cap is back at 1,000 per holder per handle, far above what connections that dropped without being reaped leave behind. The index is only read row by row once it's past the cap. * fix(kv): harden the share cleanup jobs - Job records are keyed by domain, so a deployment that shares the table but not the database or broadcast (staging beside production) never runs another's jobs. - A job whose record can't be written runs unrecorded instead of not at all, and a failed check for live handles records the job anyway so it retries. Failures that used to be swallowed are logged. - Recording a job again keeps its attempts and backoff; one that gave up gets one more attempt. - A system retire gives back the grant of a live share it doesn't cover inside a withdrawn region, and a share made under a reused uid grants again once its row stands, so a concurrent retire can't leave a live handle without its grant. * fix(kv): don't cache a live handle a retire overtook during its app lookup The live read decided whether it could cache before awaiting the app lookup, so a retire's forget that landed during that lookup was overwritten with the handle as live for up to 60 s. It's now checked after the lookup. * chore(kv): word the job key comment for any install	2026-10-31 12:12:29 -07:00
Daniel Salazar	2dfd93687f	perf(database): defer primary fallback for user lookups (#4359)	2026-10-26 21:37:06 -07:00
Daniel Salazar	52da7b571c	perf(permission): read a large holder's user-to-user grants per permission (#4349) ... A user-to-user grant check loaded the holder's whole row set, from one Redis blob or one SELECT, and filtered it in JS. Every KV share handle a user receives adds a row, so the cost of each check grew with incoming shares. Holders with up to 200 grants keep the whole-set cache. Past that the cache holds a marker and each check reads only the permissions it names, off a new (holder_user_id, permission) index. Matches stay exact in JS, since MySQL compares `permission` case-insensitively. The batched multi-holder read shares the same chunked query. Migrations: mysql_mig_51, postgres_mig_40, sqlite 0097.	2026-10-26 20:51:46 -07:00
Daniel Salazar	f850155411	KV share handles: hardening, higher limits, batch share, uninstall data deletion, account-deletion cleanup (#4344) ... * fix(kv): sign share handle ids and drop retired handles from every node's cache New handle ids carry a short HMAC (kvh-<uuid>.<mac>); an id whose mac does not verify is refused before any lookup. Unsigned ids keep working but no longer fall back to the primary on a replica miss. Retiring or changing a handle now broadcasts outer.pubsub.kv.share.forget so every node drops its cached row. Event subscribes and delivery rechecks refuse a handle whose owner is suspended, as the driver does. * fix(kv): an app its user removed stops reaching shares An app acting for a user exercises a share grant only while the user still has the app authorized: at least one user-to-app permission row, which every app token path leaves and revoke-user-app '*' removes. The share itself stays; authorizing the app again restores access. The answer rides the permission cache, keyed by the app-under-user generation the revoke bumps, so live subscriptions go quiet too. * feat(kv): list keys through a watch-level share handle list with as: 'keys' through a handle needs only watch, which already learns every key written in the region. Values and entries still need read. * feat(kv): filter share listings by app and by state GET /kv/shares and GET /kv/shares/with-me take an optional appUid: an app uid, or null for the app-less store. An account session may name any app; an app is refused any namespace but its own. GET /kv/shares also takes state=live|revoked|all, all by default. * feat(puter.js): appUid and state options on the KV share listings listShared() and listSharedByMe() pass appUid (null for the app-less store), and listSharedByMe() passes state. Docs cover the new options and that a watch share can list keys without their values. * feat(kv): indexes and store counts for share-handle quotas and retention Adds covering indexes for the owner's live count, the owner-holder pair, app-filtered listings and retired rows, plus store methods to count a pair's live handles and delete rows retired before a cutoff. The live handle cache holds up to 50,000 entries; the miss cache stays at 10,000. * feat(kv): higher share-handle caps, a per-holder cap, and sharing with several recipients Live share handles per account go to 25,000 paid / 5,000 free, and per app to 10,000 / 2,000. One owner may hold out 256 live handles to one holder on every plan, which bounds the rows a revoke or unshare walks. Re-sharing an existing region takes no slot and is never refused. POST /kv/shares takes recipients (1 to 50) as an alternative to recipient: one region, level and label for all. Caps and a new per-recipient grant budget (300/min, 2,000/hour) are checked for the whole batch up front; failures about one recipient are reported on its item. The single-recipient request and response are unchanged and run through the same code path. * feat(kv): delete share handles 30 days after they stop A daily job deletes retired share-handle rows older than 30 days, in batches of 500. One node runs it per day; it stops on shutdown. * feat(puter.js): share a KV prefix with several users in one call puter.kv.share() takes an array of recipients, positionally or as recipients in the options form, and resolves to one item per recipient: the share, or an error with its message and code. The single-recipient forms are unchanged. * feat(kv): delete an app's data on uninstall, and close account-deletion gaps Uninstall: - POST /auth/revoke-user-app takes an optional `deleteData` boolean, only with `permission: '*'` (400 otherwise). AuthService.uninstallApp revokes every grant, deletes the app's worker sessions, then awaits `app.uninstalled` { userId, userUuid, appUid, deleteData }. - With deleteData, KvShareService retires the handles the user shares out of that app's namespace (as the platform, in batches) and then empties the namespace in the background, 5,000 items per step, stopping on shutdown. Handles shared with the user are not touched. - SystemKVStore.purgeNamespace(namespace, { maxItems }) deletes up to maxItems items of one `v1:<uuid>:<appUid>` namespace, reading keys only, with no mutation events or usage, and reports `done` once it is empty. It refuses the system namespace and anything else. flush shares its page-delete path. - puter.perms.revokeApp(appUid, '*', { deleteData: true }); the desktop's uninstall dialog has an unchecked "Also delete this app's data" option. Account deletion: - EventsService settles each of a deleted account's apps on `user.delete` (they cascade without `app.changed`) and drops the owner's subscription keys. - The flat permission entries the account held are purged in pages. Suspension: - `user.suspended` / `user.unsuspended` events; the share-handle store drops cached rows owned by that user here and on other nodes. * fix(permission): await a peer's flat-permission deletes so emitAndWait sees them applied * fix(kv): strongly consistent purge pages, and an uninstall purge stops once the app is authorized again A stale empty page could end a purge as done while items remained. The background purge after an uninstall now checks before each step that the user hasn't authorized the app again, so data written after that is kept. * fix(kv): flush keeps eventually consistent page reads; only a purge reads strongly consistent * fix(user): emit user.suspended and user.unsuspended when the user store changes an account's suspension	2026-10-26 09:43:55 -07:00
Daniel Salazar	589a98b12c	feat(kv): share KV prefixes with other users at watch/read/write, and bound what apps can be granted (#4331) ... * feat(kv): share key-value prefixes with other users, and bound what apps can be granted - KV share handles move out of Events into KvShareService, served at /kv/shares (the /events/kv-handles routes stay as an alias). - Shares have watch, read and write levels, encoded in the permission root (kv-share, kv-share-read, kv-share-write) with write implying read and watch. - Every puter.kv method takes optConfig.handle; keys are relative to the shared prefix, cursors are sealed, and private entries stay hidden. - New puter.kv.share, unshare, listShared and listSharedByMe; optional label; kv.share.received notice in the app's mailbox only; recipient blocks apply. - puter.kv.set gains onlyIfAbsent. - fs and kv events gain viaWorker, and kv events report self from the writer. - /auth/grant-user-app only accepts canonical permissions from a fixed set of families; the consent dialog, puter.perms and docs follow. - Migrations: kv_share_handles gains revoked_by, label, and grantee and app indexes (mysql_mig_49, postgres_mig_38, sqlite 0095). * fix(kv): never name a private key in a cross-app or handle cursor, and bound flat-permission scans - Listings of another namespace (cross-app or through a handle) resume after the last entry they returned, and their cursors are sealed, bound to the namespace or handle, and padded to one length. Old plain cross-app cursors are still read when they name the target namespace; other namespaces get 400. - PermissionStore scans flat permissions a few roots at a time and skips roots nested under another root in the same call.	2026-10-25 08:16:19 -07:00
Daniel Salazar	9ff224e22f	perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259) ... Pipeline - CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s. - `req.ua` is parsed on first read instead of on every request. - JSON `rawBody` reuses the parsed buffer instead of copying it. - `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call. - SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under load is unchanged. Auth and permissions - Token-to-actor auth starts the user, session and app lookups together and checks them in the same order: fewer sequential Redis round trips per authed request. - get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select after its own insert and doesn't wait on the cache write for replica rows. - getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer hands out a session that was never written. - whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged. - OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight per URL, and the Apple key is parsed once. - gui-token and sync-cookie use `req.actor.user` instead of re-reading the user. - listSessions batches app lookups. App icon probes run together, and the original-icon probe skips the lineage fallback. - Default-granted permissions are answered from config before the scan cache, and exploded options are deduped: one less Redis read per /drivers/call. - ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node first, then the ancestors concurrently on a miss. Filesystem - Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about 1.3 s to about 40 ms per 100 new files. - MySQL single-file inserts return the row built from the written values, falling back to the primary read: one less primary round trip per new file. - Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row fallback. - storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries `userUuid`: quota checks and /df no longer serialize the listener reads. - readdir cache backfill is one pipeline instead of 3xN SETEX. - Profile reads skip the lineage fallback; the own profile is read alongside the user lookup. - Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently. - Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream upstream error. - mkdir looks up target and parent together. - Upload completion reuses the session from the access check (status is still re-read before completing), and multipart signing reads it once. - Moving a shared file to Trash skips two lookups that always returned the entry itself. Drivers and events - Subscription resolvers run concurrently and are folded in registration order. - SubdomainDriver and WorkerDriver overlap independent lookups. - The events "anyone subscribed?" check is one Redis pipeline when the region is warm. - Handler publish overlaps the count with the size check. - Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays cross-region invalidations to the others. Sites, share and AI - Hosted sites resolve the owner first, then the app and root together. Site config loads only on 404s. The hosted-app query selects only the columns it uses and moves into AppStore. - shareFlags asks about the plan only when an item is shared by link only. - Share listings run the count, teams and plan reads alongside their page query. - Claude attachment cleanup no longer blocks the response. - New util `awaitLater` for promises started early and awaited later.	2026-10-17 12:03:35 -07:00
Juan Fernando Castro	e32bfa839d	fix: count one app open per account per app per hour (#4152) ... * fix: count one app open per account per app per day `app_opens` is read two ways at once: `open_count` is a row count and `user_count` is distinct accounts. A repeated open moves the first without the first moving the second, and the incentive report prices opens. An open is now marked as it is recorded and skipped for the rest of the day, before both the row and the event ClickHouse ingests, so the two stores agree. The marker is set only when absent, so the first open of the day wins a race; an unreachable marker counts the open, since counting twice beats not counting. * fix: count one app open per account per app per day, at the count Dropped the row-suppression approach. `app_opens` is not only a counter: `getRecentAppOpens` orders the desktop's recent list by `MAX(_id)`, and `app.opened` is the socket event other tabs reorder on. Suppressing the repeat row froze both at the first open of the day. Every open is recorded as before. `open_count` counts each account once per day instead, which is the figure `user_count` is read against. `countDistinctUserDays` folds the pair into one number because `COUNT(DISTINCT a, b)` is MySQL-only, and `dayBucket` covers MySQL's decimal division. ClickHouse has uniqExact over the tuple. Note the figure's meaning changes at deploy: a period spanning it mixes rows with account-days, so a comparison across that point is not one. * fix: count one app open per account per app per hour Review feedback: a day is too long a window for the repeat to collapse into. The period is now an argument rather than a baked-in 86400, held in one constant on AppStore beside the other store policy numbers, so the SQL and ClickHouse sites cannot drift and a future change is one line. The fold `COUNT(DISTINCT user * N + bucket)` is only injective while the bucket index stays under N. Hourly buckets are 24x the day index and pass the old 1000000 in 2084, so N moves to 10000000. Buckets are clock hours, not a rolling window, so two opens a minute apart either side of the hour still count twice — tested, and the same property the day version had at UTC midnight. The detailed-stats fixture gave its repeat open a 60s offset, which under hourly buckets would have counted twice whenever the suite ran in the first minute of an hour. It now shares a timestamp with the open it repeats.	2026-10-14 11:54:17 -04:00
Daniel Salazar	ff815f730d	fix: harden token delivery, session revokes, app identity and fs scoping (#4229) ... - relay sign-in popups take the opener origin from the referrer only - session revokes read the primary, batch per user, and tombstone caches in every region - wisp relay tokens bound to the minting session; verify rechecks the account - hosted origins keep app identity with the site owner; deleted app uids recorded in deleted_apps - subdomain cache tombstoned on delete, update, reclaim and app/account cascades - email lookups require an exact address match after the SQL lookup - workers deploy only from files the deployer owns - share_app_data:false enforced at ACL time, plus copy/move/rename/share/recursive listing - kv remove returns no value without read; kv:expire op grant usable - fs errors name the entry instead of the owner's path - shared-by-me drops rows a delegate no longer has authority over - private apps can't move index_url onto another user's origin - GUI: default apps saved and launched by uid; save picker accepts only copy/move; permission dialog waits for focus; Publish as Worker hidden on others' files	2026-10-14 06:56:44 -07:00
Daniel Salazar	d8f9aff3dd	fix: follow-ups to #4161–#4203 (rate limits, broadcast, AI routing and billing, fs subtree walks) (#4205) ... * fix(rate-limit): count the newest KV rate rows first The KV backend's rate check, peek and concurrent acquire list the key's rows with a limit + 1 cap that the store applies before dropping expired rows, so unswept rows from past windows (oldest keys) could fill the fetch and the live count read as zero. They now list newest first. * fix(broadcast): split webhook flushes under a byte budget, per-peer backlogs Flushes go out in bodies of at most `broadcast.outboundMaxBodyBytes` (default 4 MiB) instead of one POST, and each peer keeps its own backlog of what it missed, capped by count and four bodies' worth of bytes, oldest dropped first, so one failing peer can no longer grow every peer's body past their 50 MB limit. * fix(ai-chat): classify dropped upstream connections as provider failures A connection reset or refusal (SDK APIConnectionError, Mistral's ConnectionError, an undici reset on the cause) carries no status, so a chain of them fell through to 500 internal_error. Attempts now record connectionFailed; a chain of only resets and 5xx answers 400 upstream_provider_unavailable, and a reset counts as an upstream signal in a mixed chain. * fix(ai-chat): keep SDK retries on Anthropic count_tokens The Claude client is built with SDK retries off so the driver can fall back, but count_tokens has no fallback route. It now passes sdkRetryOptions, like the OpenAI moderation call, so a transient 429 or reset is retried. * fix(ai-chat): mark a route unhealthy for 60 s after a transient failure With SDK retries off, one 5xx, 429, timeout or dropped connection kept a route marked for 10 minutes, sending that model's traffic to resellers. Those failures now mark the route for 60 s; bad credentials, an exhausted account and other route-level failures keep 10 minutes. * fix(metering): don't memoise spent addons inside the force-exact window The spent-addons memo could cache a pre-grant record read from a lagging replica right after a credits-changed announcement, so overage skipped the new credit and requireCredits kept answering 402 for up to 15 s. A spent record is now only cached once the actor's force-exact window has ended. * fix(share): send the digest entries a flush claimed when another take fails Digest entries are taken concurrently, and a single rejected take threw away every entry the other takes had already removed from KV. The flush now logs the failed takes and sends what it claimed; an entry whose take failed stays queued for a later flush. * fix(fs): announce an overwriting move's replaced entry only when it took its name /move and v2 /fs/move looked up the entry at the destination name before moving and always announced its removal. A move into someone else's Trash renames instead of replacing, so the owner's desktop hid a live entry and legacy /move reported it as `overwritten`. Both routes now pass the target on only when the moved entry kept the name it was looked up by. * fix(ai): name the vendor in upstream error fields from media providers upstreamFetch now takes the provider id plus a display label, and UpstreamResponseError carries the id. translateProviderError copies a string `provider` off the error into `fields`, and the BytePlus video and Cloudflare image errors set one, so upstream_auth_failed and upstream_credits_exhausted alarms say which vendor failed again. * fix(teams): delete a provisioned seat whose home folders could not be created provisionUser logs a failed default-folder creation and still returns the account, so provisionAccount went on to add the membership, issue a password, mail the seat and emit team.account.created for an account with no home. It now cascade-deletes the new user and returns 500 internal_error before any of that. * fix(ai-chat): load model catalogs without holding up server start onServerStart awaited every gateway catalog (up to its 10s fetch timeout) while HTTP was already listening, so drivers started after it had no providers yet and chat requests found an empty model map. The first build now runs in the background; complete(), models() and token counting wait on it instead of failing. * fix(ai-chat): serve direct models while gateway catalogs load at boot The first model-map build now publishes a fresh map as each provider's catalog lands, so static catalogs are routable at once. complete() and token counting wait for the full first build only when the model, or the provider asked for, isn't in the map yet. Refreshes still swap once. * fix(ai-chat): split gateway-billed cost across input and output A completion billed at the cost OpenRouter, Infron or Neuralwatt reports was quoted to `ai.prompt.cost-calculated` as all input and no output. The ledger still records one `billedUsage` line; the reported per-key costs now spread that bill over the usage keys at the catalog's relative rates, in whole µ¢ that sum to it. Neuralwatt energy units take no share. * fix(ai-image): hold Replicate credits for the whole prediction The output and input-surcharge holds took the default 10-minute TTL, the same as the prediction window alone, so a slow prediction could outlive them once creation and input measurement were added. They now last 15 minutes, like the video job holds. * feat(ai-chat): report what a completion was charged as usage.cost Chat usage (the non-stream result and the stream's final usage line) now carries `cost`: the µ¢ the ledger records for the request after the cost factor, from the request's recorded charges and the gate's factor lookup. `usd_cents` is unchanged. A hooked recording now copies the usage object first, so keys the driver adds later never land in the ledger. * fix(ai): charge streamed speech once its first byte arrives Per-character TTS and the voice changer recorded their charge when the upstream's response headers arrived, so a body that failed before any audio was billed in full. A shared relay now charges on the first byte, or when the caller hangs up, and charges nothing for an upstream that fails or ends before sending audio. * fix(ai): release credit holds only after the request's usage lands A hold was released as soon as its operation returned, while the usage it stood for was still being written (behind the cost-factor lookup), so a concurrent request briefly saw neither. AI metering now tracks the request's usage writes and its holds (chat included) release once those settle, without delaying the response; a streamed charge chains its own. A failed write still releases the hold, and the hold's TTL backstops it. * test(ai-chat): wait for a stream's credit hold to come back after its usage lands * perf(fs): page subtree delete and copy by parent link, not by depth over the path range Each page of a recursive delete or copy sorted the whole remaining subtree by a computed depth, so a walk cost grew with the square of its size. It now pages the children of up to 100 sibling directories at a time by parent_uid, depth first, keyset on (parent_uid, id), so a page reads only its own rows; rows with no stored path are reached too. SQLite gets the parent_uid index the others have. * fix(fs): find children by parent_uid in the empty check and supplementary-name lookup Rows the old backend wrote carry parent_uid and no parent_id, so a non-recursive delete of a directory holding only such rows passed the empty check and orphaned them, and on MySQL the supplementary-character child lookup missed them, leaving their subtrees out of listings, counts and renames. Both now key on parent_uid; the empty check still probes parent_id too, which the delete cascade follows. * docs(fs): drop the single-POST note from the entry broadcast cap	2026-10-12 01:10:54 -07:00
Daniel Salazar	c815dc82a1	refactor(core-util): shared bounded map, cheaper route lifecycle, DDB/DB/rate-limit/gate dedup (#4185) ... * refactor(util): add BoundedTtlMap for core caches and log throttles One size-capped, LRU-evicting map with an optional TTL and a once-per-window shouldEmit helper. Replaces the hand-rolled maps in the DynamoDB throttle and repair loggers (the repair one was unbounded), the Slack repeat throttle, the auth probe's reauth log throttle, and the card-verification status memo. * perf(http): skip route lifecycle work when nothing listens The per-route lifecycle middleware built a payload, opened an event.emitAndWait span and hooked finish/close on every request, listeners or not. Each phase now checks hasListeners first. EventClient's three copies of the key-prefix walk become one memoized #matchKeys. * perf(ddb): write batch chunks concurrently and retry the TTL sweep batchPut/batchDel sent their 25-item chunks one after another; they now run up to four at a time through runWithConcurrencyLimit, and a failed chunk stops the ones not yet started. batchGet and the batch write share one #sendWithUnprocessedRetry instead of two copies of the backoff loop, and the TTL sweep goes through #batchWrite, so the unprocessed deletes it used to drop are retried. The AWS and local clients are built by one #bindClient. * refactor(db): share migrations and tryHardRead in the base client MySQL and Postgres each carried a copy of the migration-path runner and of the replica-racing tryHardRead; both now live on AbstractDatabaseClient (applyMigrationPaths, and tryHardRead gated on hasReadReplica), and SQLite drops its own tryHardRead override. Removes MySQL reinitPrimary, reinitReplica, isRetriableError and readWithRetry and the base requireRead, which had no callers outside tests. SQLite prepared statements and the Postgres SQL rewrite are memoized per query text in a bounded map. * refactor(rate-limit): route every entry point through one charge() The nine rate/peek/acquire entry points each repeated key and limit resolution, the backend call and the fail-open catch. They now share charge(), routeKey() and guardSlot(); log lines are unchanged. Driver concurrency slots release at most once like the other slot handles, and checkRateLimit takes { failOpen: false } for limits that must hold when the backend is down. * refactor(util): add secretsEqual for constant-time secret comparison One HMAC-then-timingSafeEqual helper, safe for inputs of any length, so callers stop hand-rolling the length guard. File signature verification uses it. * refactor(fs): infer signed-file MIME types with mime-types mimeFromName looks extensions up in mime-types instead of a 35-entry inline table, keeping the two types the table reported differently (ico, wav). The purpose-labelled signing key is derived once per secret instead of on every signature, where signFile derived it twice. * refactor(http): build route gates from one gate() helper Each gate repeated a try/catch -> next(err) wrapper or a .then(next, next) chain around an assertion. gate(check) does that once: a throw or rejection goes to next(err), and a synchronous check still calls next synchronously. The credits, subscription and reputation gate files held nothing but such a wrapper; their factories move into gates.ts and the files go. * perf(hosting): build the puter-site hosting config once createPuterSiteMiddleware rebuilt the normalized hosting-domain config on every hosted-site request; it is derived from static config, like the domain lists the middleware already builds at construction. * fix(db): tryHardRead's replica leg doesn't fail over to the primary MySQL's read() retries on the primary after a load-shed or transient replica error, so routing tryHardRead through it read the primary twice on that path. The replica leg now reads the replica alone. * fix(ddb): settle in-flight batch chunks before rethrowing A failed chunk stopped unstarted chunks but rejected while others were still retrying, so writes could land after the caller had handled the error and invalidated its cache.	2026-10-11 21:53:33 -07:00
Daniel Salazar	b67acdbf6f	fix(stores): return written rows without reading them from a lagging replica (#4166) ... * fix(stores): read rows back from the primary after writing them ShareStore (create, upsertPending, upsertActive, upsertActiveGroup, applyPending, upsertAnyone), AppStore (create, createFromOrigin, update, claimOwnership), NotificationStore.create, SubdomainStore.update and NotificationDriver.create returned the row they had just written by reading it back through `db.read`, which goes to the replica. Under replica lag that came back null or stale. Those read-backs now go to the primary, through a `primary` option on the getters. AppStore caches the primary copy on create, and claimOwnership no longer reads through getById, which cached the replica's pre-claim row on a miss. * fix(stores): skip the notify read-back; read OIDC link conflicts from the primary NotificationStore gains `insert`, which writes and returns the uid; `create` is now `insert` plus the primary read-back. NotificationService .notify ignores the row, so it uses `insert` and no longer pays a primary round trip per recipient. NotificationDriver.create still reads the row back from the primary. OIDCStore.link reads the conflicting row from the primary after a unique violation, so a replica that has not seen another account's link can no longer make the call report success. * perf(stores): return written rows instead of reading them back ShareStore.create, NotificationStore.create, AppStore.create and AppStore.createFromOrigin build the row they return from the values they inserted and the insert id, instead of reading it back from the primary. The DB-defaulted creation timestamp is now written explicitly from unix seconds (`timestampFromUnixSql`), so the returned value matches what a read returns (`timestampValue`); postgres INSERTs add RETURNING id. The AppStore rows fill every column the INSERT leaves to the schema and are cached as before. ShareStore.upsertPending's re-invite branch selects the full pending row in its dedupe read and returns it with the new mode and data; it falls back to the primary only when the UPDATE matched nothing. The remaining primary read-backs stay and say why: the share upserts and the new-invite insert (an upsert's surviving uid/created_at and the entry-owner subquery are only known to the primary), applyPending and claimOwnership (only an id or uid in hand), AppStore.update and SubdomainStore.update (the base row is a replica copy that would be cached), NotificationDriver.create (notify returns only the uid) and the OIDC link conflict.	2026-10-11 17:51:38 -07:00
Juan Fernando Castro	7de8360a6e	Sharing cleanup: bounded outbound listings, and what see does not reach (#4118) ... * fix: keep a file name to one line in a subject, and the inbound total to the account **A name reaches the digest subject as the user typed it.** Subjects compile with `noEscape` on purpose -- a header is not HTML, and escaping would show entities to the reader -- and the comment there says free-form values collapse whitespace upstream. A file name does not: `#assertUsableName` refuses only an empty name, a slash, `.` and `..`, so a name may carry newlines. It is collapsed to one line and bounded before it goes in. **`includeTotal` answered an app with the account's figure.** The page is filtered by what the caller can reach -- an app sees only what its user handed it -- while `countByHolder` counts every inbound share. The docs already call `total` an approximation for the filtering; they did not cover an app reading the size of everything else. It is answered only for a caller acting as the account now, and the page documents that. The team-share fixture built its actor as `{ user }` with no `effectiveApp`, which reads as an app rather than a session. A session acts as the account, so it says so. * fix: stop `see` reaching inside a folder, and bound mode changes **A `see`-only folder recipient was pushed every child's name, size and type.** The fan-out picked holders by reach -- any live grant on a reaching root -- and `see` is a grant. But `see` says a node is there; `list` is what says what is inside it, and the payload carries exactly what `list` would: name, path, `is_dir`, type, size, modified. Descendants now go only to a holder whose mode reaches inside. A change to the shared node itself still goes to everyone who holds it, which is what `see` covers. **Mode changes skipped the daily quota entirely.** Not spending share budget is right -- changing a mode is not new reach, and the existing test says so -- but unbounded it lets one account write a pair as fast as it can ask. They have their own budget now, ten times the share limit by default and configurable, so the exemption stays an exemption from *that* budget rather than from any. * fix: let a half-finished downgrade be finished, and stop Trash answering **A mode downgrade that failed between its two halves could not be repaired.** `setUserUser` grants the new mode and then revokes the superseded ones. If the revokes did not all land, the higher grant stood -- and the retry returned early, because the mode it was asked for was now present. "Nothing to do" now means this mode is the only one standing, so asking again is what finishes it. The same two halves, and the same early return, were in the group variant. **Trashing someone else's item answered for what is in their Trash.** Moving into the owner's Trash is deliberately authorized without write on it -- that is what lets a recipient delete from a shared folder -- but a name collision there came back as `An entry already exists at /owner/Trash/<name>`. The mover picks the name, so that is a question they can ask about a folder they cannot list. It dedupes instead, which is also what the GUI wants: trashing should not fail over a name. * perf: record the entry's owner on the share row The outbound listings answer "shares on things I own that someone else issued" by joining `share` to `fsentries` and filtering on the owner there. Neither side of that join bounds the work: driven from `fsentries` it walks every file the user owns, driven from `share` it walks every share until it has a page. For a user who owns a lot and delegates nothing, that is the whole table for an empty result. `#outboundAppsSql` is worse than the ticket says -- it has no LIMIT at all, so every call aggregates the join across both tables. The owner now rides on the row, indexed with `id`, so all three read it directly. An entry cannot change owner -- a move out of the owner's tree is refused -- so it is written once, with the row. Set in SQL from `fsentry_id` rather than threaded through callers: four insert sites, each of which already has the id, and none of which can now forget. The risk in denormalising is a write path that does: a row with no owner recorded is a share that silently stops appearing in its owner's listing. The new test covers exactly that -- it fails when one insert is made to write NULL. Verified against a real boot: a pre-existing share row, on a database that genuinely predates the column, comes out with the owner its entry has. * fix: the review's findings on this branch **The Trash collision still had both halves open.** `overwrite` was checked before the trash case, so a caller setting it -- it is wire-supplied -- could still delete whatever it collided with in the owner's Trash, and the 409 survived for anyone who did not. The trash case is decided first now. Dedupe alone did not close it either: a name that changes only when there is a collision answers the same question the refusal did, and the mover sees that name even though the path is masked. Trashing someone else's item renames it every time, so the result carries no signal. `original_name` already records the real name, which is what a restore reads. Two tests pinned the old name; they say what it is now. **The owner column had no fallback.** Reads went only through it, so a row written by anything that predates it -- a pod mid-roll -- would never appear in its owner's listing again. The three reads now also cover `IS NULL` rows through the old join, indexed on the null and empty once none remain. **The grant became conditional on a replica read.** `if (!already)` skipped it on a stale row while the revokes below stayed unconditional, which could leave the holder with nothing where the old early return left their access alone. It grants unconditionally again; only the early return changed. **The re-issue budget covered one path of four.** Team, link and invite re-issues reserved nothing, so three ways round it were left open. They share one `#reserveRemodeQuota` now. Two docblocks had been pushed off the functions they describe, and two comments still described the join that is gone. * chore: one-line the comments added here * test: measure the prefix walk by what it keys on, not by the clock `checks many long paths without re-reading every prefix of each` gave the walk 1500ms of wall clock. It measured nothing useful: the shape it guards against — keying each step on the accumulated prefix text rather than on the parent id — costs 29ms against the current 8ms on that input, so both ends sit far under the bound. Under coverage the same test ran 1511ms and failed, which is the only thing the number could ever report on. `PathPrefixIds` now counts the characters it keys on, and the walk is asserted on that: proportional to the input, and twice the depth costs twice as much rather than four times. Keying on the accumulated text fails both. The original case keeps its real assertion, that a request this size is refused. * fix: keep the recorded entry owner current, and the trash guard on the retry A move into someone else's tree re-owns the whole subtree, which the migration comment claimed could not happen. The shares on it are re-pointed at the new owner, or they drop out of the listing the column keys and stay in the old owner's. The unique-violation retry re-entered the collision resolver without the trash flag, which is the one call site the guard was not threaded through.	2026-10-09 16:14:23 -04:00
Juan Castro	4b5f509066	fix: only rewrite rows the old rule actually wrote, and fix the invites too ... Two findings from review, both of which contradict what the last commit claimed. **The backfill could merge keys, not only split them.** I asserted it "can only split rows apart, so it cannot collide". Not true, and reproducible: a row predating the `clean_email` column has a NULL there, so its index key is `lower(email)` -- filling it in folds `+tag`, and `john+news@me.com` lands on a key `john@me.com` already owns. A legacy row holding an un-lowercased `clean_email` collides the same way once lowercased. Either throws, and the two engines that re-run every file on every boot would then refuse to start until someone operated on the database by hand. Both updates now fire only where the stored value is exactly what the old rule produced and differs from the correct one. That makes "splits only" true: the sole change is dots coming back. NULLs and legacy values are left alone. The same predicate makes them write-idempotent. Postgres has no no-op-update suppression, so without it every boot rewrote a tuple per Apple row forever. **The pending-invite residue misdelivers rather than just going missing.** `claimPendingShares` looks the row up by the canonical address, so whoever confirms `jsmith@icloud.com` claims the invites addressed to `j.smith@icloud.com` -- someone else's file, handed over on confirmation. I had written this off in the PR as unfindable invites that "expire on their own"; `share` has no expiry column and the lookup has no time predicate, so they are claimable indefinitely. Fixed here rather than later: the typed address is kept as `invitedAddress` whenever it differed from the canonical, which is exactly the collapsed rows, so the invites recompute from it under the same guard. Verified by seeding a database at the old version with the two rows that used to abort the migration, and booting: it reaches version 87, repairs the collapsed row, and leaves the NULL and legacy rows untouched. Replaying the statements changes zero rows.	2026-10-07 10:23:30 -04:00
Juan Castro	ece8ad5f1d	fix: register the Apple-domain backfill with the sqlite runner ... The sqlite migration list is hand-maintained, so the file alone did nothing — a fresh boot still reported 87 files and stopped at version 86. MySQL and Postgres read their directories, so only this one needed the entry. Caught by booting a local instance rather than by the suite: the client tests pin `CURRENT_SCHEMA_VERSION`, which fails once the entry exists but says nothing while the file is simply unregistered. Verified against a seeded database at the old version. `John.Smith@iCloud.com` went from `johnsmith@icloud.com` to `john.smith@icloud.com`, `a.b+tag@me.com` from `ab@me.com` to `a.b@me.com` — subaddressing still folded — and a gmail row was left alone.	2026-10-07 09:45:26 -04:00
Juan Castro	cf817b0f0a	fix: treat dots as significant on Apple's mail domains ... `cleanEmail()` applied gmail's `dots_dont_matter` rule to icloud.com, me.com and mac.com. Apple allocates the exact string: `j.smith@icloud.com` and `jsmith@icloud.com` are two mailboxes with two owners. Collapsing them made one canonical address stand for both. What that canonical address feeds: - `findEmailOwner` falls back to it, so an OIDC sign-in whose verified claim is the dotted address resolves to the account holding the undotted one, and `linkProviderToUser` has nothing left to object to -- the provider verified a real mailbox and the account's address is confirmed. - The unique index on COALESCE(clean_email, lower(email)) reads it, so the rightful owner of a dotted Apple address is refused signup because the undotted account already "owns" it. - Pending share invites are stored and claimed on it. Subaddressing is untouched: Apple does fold `+tag`, so that rule stays. Gmail keeps both rules -- there the fold is correct. A migration recomputes `clean_email` for those domains on all three engines, since rows written before this still hold the collapsed value and the index reads the stored one. Verified the SQL against the function rather than by inspection: same five inputs through `cleanEmail()` and through sqlite give byte-identical output. It can only split rows apart, never merge them, so it cannot collide with the unique index. Not changed: `findUserByEmail` still matches on the canonical address. The cross-match it documents is correct where the provider really does fold -- `foo.bar+tag@gmail.com` is the same inbox as `foobar@gmail.com`. Requiring an exact match would break that for no benefit here; the fix is for the data to say what the provider does.	2026-10-07 09:17:03 -04:00
Juan Castro	a67cdac380	fix: bound the inputs on the grant and token routes ... Three permission routes sized their work by the request body rather than by the route, so one call could cost far more than its rate-limit slot implies. - `extra` and `meta` were only type-checked. They ride every row a grant writes (up to 16) and are re-serialised into the per-(user, app) cache on each miss, so they are now capped at 4 KiB. - `/auth/create-access-token` took a `permissions` array of any length: one permission check and one sequential INSERT each. It now uses the same 16-per-request cap the grant routes already had, and runs every entry through the validator those routes use. - Withdrawing an app's cross-app data grants looks them up by permission text, and no index on `user_to_app_permissions` led with `permission`. Indexed per engine, mirroring what mysql_mig_22 / postgres_mig_11 / sqlite 0067 did for `user_to_user_permissions`. `grant-dev-app` validated none of its input — not even the type of `extra` — so it now runs the same validator as its user-app sibling. Two paths could also carry a permission wider than the `varchar(255)` column it lands in. `grantUserAppPermission` already rejected that after rewriting; `grantDevAppPermission` now does the same, and `createAccessToken` checks it before the session row a failing INSERT would otherwise orphan. Caps are published in rate-limits-and-quotas.md. Every in-tree caller sends one permission and a small `extra`, so none of them change behaviour.	2026-10-01 16:51:39 -04:00
Juan Castro	480d9d032b	Merge remote-tracking branch 'origin/main' into juancastro/put-1988-team-force-2fa ... # Conflicts: # src/backend/clients/database/SqliteDatabaseClient.test.ts # src/backend/clients/database/SqliteDatabaseClient.ts # src/backend/clients/database/migrations/mysql/mysql_mig_42.sql # src/backend/clients/database/migrations/postgres/postgres_mig_31.sql	2026-09-28 10:24:39 -04:00
Daniel Salazar	dcb61bc4fd	fix: PUT-1824 + PUT-1884 + PUT-1952 + PUT-1827 (#3951)	2026-09-25 17:47:03 -07:00
Nariman JelvehandDaniel Salazar	3d3678a555	feat: let the Contact Us form carry screenshots and recordings (#3563) ... * feat: let the Contact Us form carry screenshots and recordings The form doubles as our bug-report channel, and the bugs worth reporting are often the ones that need a picture: a visual glitch, or something that takes five steps to reach. Up to 5 images or videos may now ride along with the message, delivered as attachments on the support email. Nothing the client says about a file is believed. The endpoint takes bare base64 and re-derives all three of the things that matter from the decoded bytes: - Type, sniffed from magic numbers and matched against an allow-list of PNG/JPEG/GIF/WebP and MP4/QuickTime/WebM. A declared MIME is never read, so it cannot smuggle anything past the list. SVG is excluded on purpose (it carries script, and support tooling renders what it is sent), as are the non-video brands that share MP4's `ftyp` box -- HEIC, M4A, JPEG 2000. - The file name, reduced to a display label with the extension taken from the sniffed type. PNG bytes named `payload.html` arrive as `payload.png`; a name can never carry the CR/LF that would break out of a Content-Disposition header, nor the bidi overrides that make `report<RLO>gnp.exe` render as `report.exe.mp4`. - Size: 10 MB per file, 15 MB per submission, counted on decoded bytes. Encoded length is capped before decoding, so an oversized payload costs a length check rather than a 10 MB allocation. The total stays well under the 25 MB most providers enforce, since the outgoing mail base64s these again. Base64 is decoded strictly, reusing the round-tripping decoder that already guards app icons -- `Buffer.from(s, 'base64')` silently drops characters it does not recognise, and the round-trip is what rejects bytes smuggled after the payload. That decoder and the image sniffer move from appIcon.ts to a new mediaSniff.ts, which gains the video counterpart. One request is now worth megabytes of parsing and outbound mail, so the existing per-user rate limit gains a per-IP backstop (which the per-user counter cannot see through freshly minted accounts), a concurrency cap, and a Content-Length gate that refuses an impossible body before anything decodes it. Payloads are not stored. They ride the email; the new `feedback.attachments` column records names, types and sizes only, so an abusive submission stays attributable once the mail has been dealt with. Also fixes the form posting an empty message when Send was pressed with nothing typed, and surfaces submit failures instead of leaving the button disabled with no explanation. * fix: stream Contact Us attachments as multipart instead of base64 JSON Base64 in a JSON body went through the global JSON parser before the route ran: raw buffer, rawBody copy, decoded string, parsed strings, then decoded Buffers plus a re-encode for the strict check. A max-size submission peaked around 110-135 MB of heap for 15 MB of files, and the route's Content-Length check ran after all of it. Attachments now arrive as multipart/form-data, which the global parser skips. Auth, rate limit, concurrency and the Content-Length 413 all run before the body is read, and busboy enforces the count, per-file and total caps while streaming, so only the decoded file bytes are held (~16 MB peak). On a broken limit the reader stops and the response closes the connection. JSON stays supported for message-only posts; a JSON `attachments` field is rejected. * fix: deliver Contact Us 413s instead of resetting the upload Closing the socket on a mid-stream limit sent the 413 and then reset the connection with the client still uploading, so the client could see a reset instead of the error. After a limit trips, the rest of the body is now read and discarded, bounded by the body budget; past the budget the request is destroyed. Adds an HTTP-level test through the full middleware stack (FormData upload, unauthenticated refusal, 413 on declared length before the body is sent, 413 on a chunked upload over the per-file cap), trims comments to the AGENTS.md length rule, and drops a box-drawing test divider. --------- Co-authored-by: Daniel Salazar <daniel.salazar@puter.com>	2026-09-25 14:19:35 -07:00
Juan Castro	db1a95260a	feat: let a team require 2FA of the accounts it provisioned (PUT-1988) ... `group.require_2fa`, toggled by the owner through `PUT /teams/:uid` and recorded in the audit log like `directory_enabled` — it changes what every seat must hold to sign in, so it is not something a team should be able to alter silently. Enforced by deriving, not by stamping a flag on the user: a stamp left behind by a member leaving, or by the rule being turned off, would lock someone out of their own account with no team left to clear it. The seat row already joins `group` and is already read per request, so it carries the flag and the gate costs nothing extra. Toggling busts those rows, or turning the rule off would leave members locked out until the TTL lapsed. The gate rides the same `allowUnconfirmed` opt-out as the other pending- requirement gates, so the routes a gated member must still reach stay reachable — `configure-2fa` among them, which would otherwise be walled off behind the very requirement it clears. Scoped to `org_owned = 1`: the owner is not a seat of their own team, and a joined member is an ordinary self-paying account whose personal login is not a team's to dictate. The owner is held to the rule at the point it matters — they cannot turn it on without 2FA of their own.	2026-09-25 10:35:24 -04:00
Daniel Salazar	132cae9f6a	Bug bash fixes (PUT-1876, 1877, 1880, 1881, 1948, 1949, 1951) and KV event fan-out by plan (#3943) ... * fix(kv): keep disableSharing entries out of other apps' events (PUT-1876) Private kv keys and values are no longer delivered to cross-app subscriptions, including forwarded and durable deliveries. * fix(fs): refuse usernames whose home path still holds another account's rows (PUT-1877) Username claims (signup, save_account, change-username, OIDC, team provisioning) now also check for leftover rows under /<name>/, and the root listing only returns the actor's own home. * fix(puter-js): reconnect the events channel after a server-side disconnect (PUT-1881) A server hang-up is retried with a bounded backoff; persistent handlers survive it and get an optional onError when the connection can't be restored. * fix(backend): let the graceful shutdown finish before telemetry exits (PUT-1948) The telemetry preload no longer exits on signals; the entry point drains, tears down top-down, flushes telemetry last, then exits. DB pools stay open until teardown. * fix(fs): reserve quota for signed uploads in progress (PUT-1880) startWrite/startBatchWrite hold each upload's declared size in a per-owner cache reservation until it completes, aborts or expires, so a burst of starts can't all read the same committed usage. Caps in-flight uploads at 10,000 per account. * fix(fs): refuse to complete a signed upload whose object never arrived (PUT-1951) Completion now 400s when the object store reports the key missing (transient errors stay lenient), abort no longer deletes an object a live entry still uses, and a ghost entry is only removed after a second miss at its recorded location. * fix(metering): move usage to v2 keys with a small totals item, per-model detail shards and a 3-month TTL (PUT-1949) The billed item holds only totals; per-model detail lives in 100 hash-sharded items read in one batch and cached for 60s, and more than 5,000 distinct usage types in a month fold into other. Global and per-app aggregates store totals only, appTotals comes from a prefix listing, a refused path only skips the key that refused it, exact reads near the allowance are throttled per key, and every metering record expires after 3 months. September usage restarts at deploy; the monthly-charge claim stays on the v1 key through September so recurring charges don't fire twice. * test(events): let the worker backoff test tolerate a loaded run * feat(events): scale KV event fan-out with the key owner's plan A KV change delivers to up to 512 matching subscriptions (2,048 filter evaluations) for a paid key owner and 128 (512) otherwise, inline values are dropped once more than 128 match, and paid accounts may hold up to 512 share handles per app. The owner's plan is only looked up when a change has more candidates than any cap. * test(metering): seed the 150-app listing test directly so it fits under coverage	2026-09-24 16:00:52 -07:00
Neal Shah	07a11240ea	PUT-1868 (#3940)	2026-09-24 00:30:47 -04:00
Daniel Salazar	0be3bc55c2	feat(metering): AI cost multiplier hook for AI drivers (#3898) ... Emits ai.cost.multiplier.<driver>.<provider>:<model> before recording AI usage, so what a model costs to charge is policy an extension owns rather than a number in core. Nothing listening records the provider cost. MeteringService.withAiCostMultiplier(driver) returns a view of the service whose recording paths scale costOverride by the hook's answer; every AI driver hands that view to its providers, so all of them are covered without touching provider code.	2026-09-17 15:18:38 -07:00
Daniel Salazar	51ce868d1e	fix: shared kv handles also allow value opt in (#3895)	2026-09-17 10:34:35 -07:00
Daniel Salazar	bb97660ad7	feat: kv events value opt in (#3894)	2026-09-17 00:17:12 -07:00
Juan Castro	5c8f02384c	test: schema version is 80 since 0084 ... #3874 added 0084_share-anyone-with-link.sql, taking the sqlite chain's target user_version to 80, but CURRENT_SCHEMA_VERSION stayed at 79 — and the test workflow only runs on PRs, so main broke silently and every open PR's 'test (base)' job now fails on it.	2026-09-16 12:48:07 -04:00
Daniel Salazar	59f73528cb	feat: share with anyone with the link (PUT-1580) (#3874) ... * feat(email): inline cid attachments and Puter mailbox delivery for sendTransactional EmailAttachment gains cid/contentDisposition so the transactional driver can send inline images. The SDK's EmailAttachment typedef now comes from types.js, which already carried cid. Docs describe delivery to <username>@puter.email recipients and the not_found code. * feat: share with anyone with the link (PUT-1580); gate sharing on a verified phone or card	2026-09-15 21:06:12 -07:00
Juan Castro	4e45fdc172	feat: a team directory apps can read, once the team opens it ... Members can already enumerate each other: `/teams/:uid/members` needs a user actor and nothing more. The only thing this adds is admitting an app actor to the same names, so an app can offer colleagues without the member driving it. That is the whole risk, so it is off until the team owner turns it on. `group.directory_enabled` defaults to 0, and a team that has not opted in answers 404 rather than 403 -- whether a team has this on is not something an app should be able to probe for either. Three things bound what an app sees. The membership tested is always the person's, never the app's, so an app installed by a member of one team can never read another's. The page carries username and uuid and nothing else -- no email, activation state, usage or role. And suspended accounts and ones that never took up their credential are left out, since offering someone who cannot sign in is noise and their existence is not this list's to disclose. Activation is the forced-change flag clearing, not the password existing: a provisioned seat holds its temporary password from birth, so testing `password IS NOT NULL` would have leaked exactly the accounts meant to be excluded. A test covers that distinction. Turning the directory on or off writes an audit row, because it changes who can read the member list and that is not something a team should be able to alter silently. Setting it to the value it already has records nothing. The toggle lives in TabTeams, and turning it on asks for confirmation while turning it off does not -- one grants access, the other only takes it away. Closes PUT-1736.	2026-09-09 14:51:05 -04:00
Juan Castro	f5b2365e83	feat: enforce the forced password change on team seats ... `user.requires_password_change` shipped with the team columns but nothing enforced it and nothing ever cleared it, so a provisioned seat kept its administrator-issued password indefinitely and `reissueCredential`'s "already activated" 409 was unreachable. Adds the fourth clause to `assertVerifiedAccount`, the only place a verification gate may live -- WebDAV builds its own actor and calls that function directly, so a second implementation would bypass it the way the phone and card gates once were bypassed. A gate that refuses everything also refuses the endpoint that clears it, so `/user-protected/change-password` opts out with `allowUnconfirmed`. That widens the route: an account pending email, phone or card verification can now change its password, which it could not before. The caller is authenticated and proves the current password, so this is benign, but it is a behaviour change to a shared route. Also here, because the gate is worthless without them: - change-password and the recovery-token path clear the flag, and record an `activate` entry when the account is a seat. - Reset takes a live account back with a fresh credential, capped at 20 per day and audited as `reset_member_password` with no credential in the row. Re-issue is audited the same way; it stays closed once a seat has chosen its own password. - An issued credential expires after 24h (new `temp_password_expires_at` column, three dialects) and login refuses it after that, so an unused reset dies instead of becoming a standing credential. - 2FA is untouched by a reset, so a reset alone is not takeover.	2026-09-09 11:51:44 -04:00
Juan Fernando Castro	00afea1285	feat: a seat is created, never adopted (#3720) ... `addMember` refuses to turn an account that already has a password into a team seat. No service path did this — `provisionAccount` always creates — but the store permitted it, and the design rules out existing accounts joining a team. Provisioning passes the guard because it admits the account before setting its temporary password. There is no bypass parameter. Both HTTP suites now provision a real seat and authenticate as it, using the same token-minting the harness uses for `POST /login`. That surfaced something worth knowing: an unactivated seat cannot call the API at all. Provisioning leaves `requires_email_confirmation` set and `requireVerified` rejects it, so the suites activate the seat first — which is the state a member is actually in when making requests. `listMembers` and `getMembership` also return `u.uuid`, which the billing events need in order to name the account without a second lookup.	2026-09-08 16:45:16 -04:00
Daniel Salazar	b784b51cf3	fix: harden the events stack for flag-on (#3752) ... * fix: harden events dispatch, single delivery and KV share handles Dispatch: a filtered subscription used the anchor path stored at subscribe time, so renaming or moving the anchor folder silently ended its deliveries; dispatch now resolves the anchor's live path from the event's own ancestor chain. A move out of a watched folder now reaches that folder's subscribers, with `from` only for rows that watched the source side. Gap markers are authorized like deliveries and coalesced per subscription and subject instead of fanning per lost event. Session subscriptions: the per-socket cap decides on the write, not before it; an orphaned watched-set token heals on refresh; durable rows keep their watch window when a session subscribe touches the same keys. `self` is false when the acting user is unknown. Single delivery: a subscription in backoff or suspended with a backlog pinned the sweeper's head and starved everyone behind it — the sweep now defers it. Only a settled handler run bills a delivery. A socket-only account row no longer wedges after two attempts nobody received. The lease is twice the handler timeout; remote candidates have their own attempt counter; the region depth reconcile runs once a minute region-wide with a bounded scan. KV share handles: a grantee no longer sees the owner's namespace and absolute prefix on the subscribe answer or listing, nor in the delivery token; revoking a wider handle retires the handles it covers; minting the same handle twice returns the existing one, after the delegation check; a row whose event cannot be re-based onto its handle is dropped rather than delivered raw. * fix: presence survives replication, long sessions and region churn One presence item per (user, app) with per-region map fields lost a region whenever two regions joined inside the replication window, and nothing ever put it back. Presence is now one item per (user, app, region): each region writes only its own, a leave or repair retires it conditionally on its own write stamp, and a read is a prefix query. Items carry a 48 h ttl refreshed by a claim-gated write off the existing socket renew path, at most once per 12 h, so a tab that stays connected keeps its region in the row. A region that answered "no socket" or completed a leave releases a shared pin, so a reconnect on another node rejoins and a flapping client cannot force a replicated write per cycle. Cached rows expire after a minute; unaddressable region names are filtered and pruned; relayed acks settle under a bounded concurrency; the forward queue is bounded in bytes as well as items. * feat: indexes for the event_subscriptions hot queries Handler publish, remove and listing, and the hourly expiry and suspension sweeps, all scanned `event_subscriptions`. Adds (app_uid, handler_name), (expires_at) and (suspended_at, id), guarded on every engine. Existing migrations: the postgres widens are now guarded so a boot does not take an exclusive lock for a no-op, the kv_share_handles grantee FK gets an index, the sqlite notification rebuild is transactional and idempotent. * fix: notification writes go through the registry The driver's `create` bypassed the type registry, producing uncatalogued rows with no size bound; it now requires a registered type, caps the payload, and answers 400 rather than 500 for a bad one. `mark_acknowledged` emits the ack other tabs listen for, and only when a row was actually changed. * fix: the handler scanner, unsubscribe, and the in-tab handler environment The free-variable scanner skipped arrows inside a declaration's initializer, so `const ids = event.items.map(x => x.id)` was refused, and treated a name after a comma in a nested initializer as bound, so a real free variable slipped through to fail on first delivery. `unsubscribe()` now drops the durable routing entry so the events socket can close. A broadcast handler running in the tab gets `user` and `fetch` like the worker gives it. `single` without a handler name is refused before the round trip. * docs: events limits, error codes and the background-workers section Retention is deployment-configured rather than a fixed 14 days, and the template no longer ships it armed. Documents `events_terminal`, the two per-event gap reasons, the subject length and listing caps, the `from` field on moves, and the handle-relative anchor. The sessions manager hides the background-workers section when the server has none to show. * feat: a background handler acts as the app does for its user A handler's `user` was a five-minute access token scoped to the subscription's `list` grant, which could stat the changed file but not read it, and could not reach the app's KV or AppData — so an app told that a file was written could do nothing with it. It now runs with the same authority the app has for that user in a tab: an app-under-user worker session, one row per (user, app) named `events:handlers`, visible and revocable in the sessions list. The `events:background` consent is what authorizes running it unattended, and is re-checked before every mint. The wider token exposed two things: puter.js opens a filesystem socket the moment it has a token, which would have parked the isolate in the app's own delivery room and steered deliveries at it; the events client now opts out of sockets (and the per-open bookkeeping) before construction, and is memoized per token in the isolate. And four filesystem operations assumed a socket exists; they no longer do.	2026-09-04 17:32:57 -07:00
Daniel Salazar	ac5446f877	feat: cross-user KV share grants and handles (PUT-1686) (#3690)	2026-09-03 19:43:27 -07:00
Juan Castro	64db3fa050	feat: add audit_team_membership and share.holder_group_id ... Insert-only record of what a workspace administrator did to an account, shaped like \`audit_user_to_group_permissions\` after 0019: nullable FK beside a NOT NULL \`_keep\` column. The FKs are ON DELETE SET NULL, never CASCADE, so hard-deleting an account cannot erase the record of the resets performed on it. Two indexes rather than one. The member's own view is the only place a reset becomes visible to the account it was performed on, so (user_id_keep, id) is a read path, not an optimisation. \`share.holder_group_id\` mirrors \`holder_user_id\` from 0067. The existing unique index does not constrain team shares at all -- it leads with \`holder_user_id\`, which is NULL on every team share, and NULLs are distinct -- so the group-scoped unique index is what prevents duplicates. Drops the \`role\` column from the specified DDL: it contradicted the settled single-administrator model.	2026-09-03 15:07:53 -04:00
Juan Castro	dc7b456afa	fix: run the membership dedup once instead of on every boot ... Review follow-ups on the dedup migration. mysql and postgres track no per-file applied state and re-execute every migration on each boot, so the unguarded DELETE self-joined the whole table at every process start, forever. Both now sit behind the same index-existence check that guards the ALTER, which also stops a rolling deploy deleting on one instance while another adds the index. The dedup test was a false green. `targetVersion: 67` never applies 0071 -- the loop breaks on `threshold + 1 >= targetVersion` but stamps the target anyway -- so the fixture asserted the current schema version on a database missing a migration. It now replays the real 0072 file against a fully migrated database, and a second test pins the off-by-one so nobody builds a fixture on it again.	2026-09-03 14:33:46 -04:00
Juan Castro	c1c1588939	fix: deduplicate jct_user_group and make membership writes unique ... \`jct_user_group\` had no unique constraint on (user_id, group_id) and \`GroupStore.addUsers\` had no conflict clause, so re-adding a member inserted a second row. \`readUserGroupPerms\` joins the junction table on group_id alone, so each duplicate returned another copy of every group permission the user holds. Deduplicate keeping the lowest id, add the unique pair index, and make \`addUsers\` ignore conflicts via the existing \`insertIgnoreInto\` helpers -- without that last part the index turns a re-add into a raised error, which five call sites would log as a failed signup step. mysql cannot delete from a table it reads in a subquery (error 1093), so it uses a self-join with the same lowest-id-wins semantics.	2026-09-03 14:32:30 -04:00
Juan Castro	eae25459f4	fix: make handle lookups case-insensitive and pin the reset flag ... Review follow-ups on the team columns. NOCASE moves onto the `handle` column itself, not just the index. Index-only NOCASE makes uniqueness case-insensitive while leaving `WHERE handle = ?` case-sensitive, so the same lookup would match on mysql (utf8mb4_unicode_ci) and miss on sqlite. Postgres still needs lower(handle) at the call site. `requires_password_change` becomes NOT NULL DEFAULT 0 on all three dialects, matching the three sibling `requires_*` flags. Left nullable, any query written as `= 0` would silently exclude every pre-existing user.	2026-09-03 12:10:48 -04:00
Juan Castro	2a5c089dbe	feat: add team columns to group and jct_user_group ... Schema foundations for teams. Ships dark — nothing reads these columns yet. group: kind, name, handle, plan_id, deleted_at jct_user_group: org_owned user: requires_password_change `kind = 'team'` marks a workspace; the seeded system groups keep it NULL so a team query can never surface them. `org_owned` distinguishes accounts the workspace created from the master account, deciding who pays rather than who may read. `requires_password_change` is a fourth `requires_*` flag for assertVerifiedAccount, needed by the phase 5 reset flow but shipped here to avoid a second three-dialect migration. `handle` uniqueness is case-insensitive, and the dialects disagree by default: mysql gets it from utf8mb4_unicode_ci, sqlite needs COLLATE NOCASE (as 0055 does for usernames), and postgres indexes lower(handle). Without this the same migration would accept `Design-Team` next to `design-team` on sqlite and postgres while mysql rejected it. Postgres handle lookups must therefore compare lower(handle) to use the index. idx_group_owner is sqlite-only: mysql and postgres already index that column. idx_jct_user_group_group is composite, unlike the existing single-column keys.	2026-09-03 12:10:40 -04:00
Daniel Salazar	04c00385b9	feat: event handlers, context, and suspension machine (PUT-1680) (#3683)	2026-09-03 01:39:06 -07:00
Daniel Salazar	9626ab9c71	feat: KV change events and KV subjects (PUT-1678) (#3682)	2026-09-03 01:39:06 -07:00
Daniel Salazar	0dfbceb047	feat: durable event subscriptions store, cache, and routes (PUT-1673) (#3679) ... * feat: durable event subscriptions store, cache, and routes (PUT-1673) * fix: durable subscription hardening (PUT-1673) - Expired rows stop delivering at dispatch time and no longer count toward the per-account cap, instead of waiting for the sweep. - The expiry sweep runs hourly with a jittered first pass shortly after boot; a 24 h interval never fired on a fleet that redeploys more often than that. - Only a durable generation bump marks peer regions cold. A session subscribe/unsubscribe in one region used to force a primary read in every other region on its next dispatch. - `subject`/`anchor_path` widen to varchar(4096) to match `fsentries.path`, and subjects longer than that are refused with `invalid_subject` rather than failing the insert on MySQL/Postgres. - The dispatch and durable integration suites wait for the specific delivery they expect and assert only within their own folder; the old any-delivery `settle()` let a late event from a previous test satisfy or pollute the next one under CI load.	2026-09-02 16:00:13 -07:00
Neal Shah	262f1dc5c5	allow concept of "home regions" (#3699) ... * allow concept of "home regions" * remove extraneous config value not applicable to repo	2026-09-02 14:11:55 -04:00
Daniel Salazar	82076ea767	feat: notification retention sweep (PUT-1668) (#3672)	2026-09-01 22:04:32 -07:00
Daniel Salazar	148b930182	feat: notification scope columns and backfill (PUT-1659) (#3670)	2026-09-01 22:04:31 -07:00
Daniel Salazar	66a975f659	feat: global outbound share listing (PUT-1664) (#3694) ... * feat: global outbound share listing (PUT-1664) * fix: address review on outbound share listing - Check share-row liveness per (holder, entry, issuer) so a grant withdrawn outside unshare doesn't stay listed while another issuer still reaches the same holder; batch the permission reads across the whole page instead of per holder. - Retire a revoked issuer's unclaimed invites in the revoke cascade, and hide invites whose issuer lost their authority at read time. - Unify the pending/active app-attribution key on `issuedByApp` and dual-read the legacy `issuerAppUid` spelling. - Add the missing share issuer index (sqlite, postgres) and correct the listOutbound plan comment. - Refuse cursors that decode but name no id instead of silently restarting from page one. - Consolidate the five hand-built ResolvedShare literals and the two listing endpoints' parse/shape code. - Ship the SDK surface: puter.fs.listSharedByMe() with docs, types, suite coverage, and the rate-limit page entry.	2026-09-01 13:52:37 -07:00
3 people	f0cd251626	🛠️ PUT-1521: Cleanup puter js permissions api + backend routes (#3607) ... * refactor(puter-js): collapse puter.perms request* to resource + access Fifteen request methods differed only by a folder name or an access level, so every new resource meant another method. Replace them with requestFolder, requestApps, requestSubdomains and requestAppRootDir, each taking the access level as an argument. The old names stay as @deprecated aliases: puter.js ships unpinned from js.puter.com/v2, so removing them would break live apps. They remain in the generated declarations because stripInternal has no effect on declarations emitted from JavaScript, and hand-omitting them would break TypeScript callers the runtime still serves. Also drops the user-to-user and user-to-group grant wrappers (groups.js and the grantUser/grantGroup half of grants.js) plus the req_ shim, none of which were documented or called. The app, origin and dev-app grants stay: the dashboard uses puter.perms.revokeApp() to clear grants on app uninstall. * docs(perms): document the collapsed puter.perms surface Replace the twelve one-method-per-task pages with requestFolder, requestApps and requestSubdomains, and rewrite the Perms overview around the seven public methods. The deprecated aliases keep working but are no longer documented. Boy Scout: drops the long-dead commented-out grantUser/revokeOrigin sidebar block for pages that were never published. * refactor(perms): drop unused user-to-user and user-to-group permission routes Filesystem access is shared through /share, which records the grant so the owner can see and revoke it. The older direct-grant paths were left behind with no caller anywhere - not the GUI, not a doc, not an app: grant-user-user (already a 501 stub), revoke-user-user, grant/revoke-user-group, and the five /group/* CRUD routes. Removing them orphans PermissionService.grant/revokeUserGroupPermission and its group-members cache bump, the three PermissionStore group writers, and six GroupStore methods, so those go too. What stays, and why: - grant/revokeUserUserPermission - ACLService and ShareService power fs.share through them. - The group permission read path (#scanUserGroup, readUserGroupPerms) - a migration seeds the admin group unrestricted driver access, so it is load-bearing. - GroupStore getByUid/addUsers/removeUsers - signup, save_account, OIDC and the self-hosted default user assign group membership. No schema change: user_to_user_permissions, user_to_group_permissions and their audit tables are untouched. Group rows now come only from migrations, so tests that need one seed it with SQL the way a migration does. * refactor(perms): reduce GroupStore to membership writes With the /group/* routes gone, `getByUid` had no production caller left — the routes were the only thing that read a group back. Removing it takes the row decoder and the GroupRow type with it, since they exist only to shape its result. What remains is `addUsers`/`removeUsers`: signup, save_account, OIDC and the self-hosted admin bootstrap all assign group membership. Permissions attached to a group are read through PermissionStore, which joins the junction table itself and never needed the store. Tests that wanted a group id now select it, which is all `getByUid` was doing for them. * chore(perms): drop the three by-hand groups no code reads freeai, experimental and dangerous exist in prod but in no migration — they were added by hand when hardcoded permissions were keyed by group name. That map is now a flat per-user floor (`default_user_permissions`), so a group nothing looks up grants nothing. Guarded rather than unconditional, because both tables the delete can reach cascade: dropping a group that still carries permissions or members would silently revoke them from every member. Only a group with neither goes. One that survives has dependents and needs a deliberate decision — query user_to_group_permissions by group_id to see what it holds. system, admin, user and temp are untouched: config names two of them and code names the others. Matches on `extra.name`, not `metadata.name` — `metadata` carries the display title and colour, and `critical: true` is set on all of these including freeai, so it does not discriminate. * feat(puter-js): collapse puter.perms onto request(resource, details) + check() One method per task meant a new method, doc page and sidebar entry for every resource. `request` now takes the resource and a payload whose accepted fields depend on it, and `check` answers the same question without prompting. request('folder', { name: 'Documents', access: 'write' }) -> path request('apps', { access: 'read' }) -> boolean request('email') -> address check('folder', { name: 'Documents', access: 'write' }) -> boolean Returns stay per-resource: a folder gives its path, email the address, the rest a boolean, and anything denied is falsy so one `if` covers both. An array asks for several at once. Everything already held is settled first, so the prompt covers only what is missing and does not appear when the whole set is held - the user answers once for the lot. `check` answers per entry, in order, so a caller can tell which parts are missing rather than only that some are. Each resource declares four things in one registry entry: how to ask for it alone, whether it is held, the strings a batch pools into a prompt, and the value once held. The strings themselves are defined once in lib/permissionStrings.js, so a request and its check cannot name them differently. `check` is built on /auth/check-permissions, already live and already used by UI.js, and it throws rather than answering false when the check cannot run: a caller that cannot tell "denied" from "never ran" would prompt someone who had already granted it. Backward compatibility: all 22 older methods stay callable and typed, marked @deprecated with the call that replaces them. A lone string still routes to the raw-permission path - no resource name contains a `:` and every permission string does, so the two forms cannot collide. The grant/revoke app methods are untouched; the consent dialog and the dashboard's uninstall path use them. Also drops three copies of the access-level assertion onto one shared validator, and gives `appRootDir` a non-prompting server probe, since `app-root-dir:` only resolves while a grant is being written and a permission check on it always answers false. * docs(perms): document request() and check() as the perms surface Five per-method pages became one `request()` page carrying the resource table, the batch form and the raw-string escape hatch, plus a `check()` page. The overview is rewritten around the two methods. requestAppData's page is re-homed as /Perms/appData rather than deleted - its scope table, private-entry guidance and lifetime notes are not signature documentation and have nowhere else to live. Inbound links from KV/set.md and Objects/app.md follow it. Playground examples move to the new call form. They are not wired into examples.js, but an example demonstrating a deprecated method is worse than one nobody loads. * fix(perms): keep /auth/revoke-user-user as a deprecated route Dropping this route with the rest of the unused user-to-user plumbing went too far. The grant side is retired and stays retired - puter.fs.share() is the only way in - but access those grants left behind has to remain withdrawable, and a caller reaching the endpoint over HTTP directly had no replacement. Revoking can only ever narrow what someone can reach, so keeping it carries no risk. revokeUserUserPermission never left the permission service; it is load-bearing for puter.fs.share(). This only re-wires the handler to it, with the gates it always had. Nothing in this repo calls the route, which makes it exactly what a later cleanup reads as dead, so a test pins the registration and its gate alongside the restored 400 and grant/revoke round-trip cases. The 501 stub at grant-user-user and the never-called /group/* routes stay deleted, as does puter.perms.revokeUser - puter.fs.unshare() replaces it and falls back to live grants when no share row exists. * fix(perms): let a write grant satisfy a read check on apps and subdomains `apps-of-user:<uuid>:write` covers managing the user's apps, which includes reading them, but nothing said so to the permission system. Prefix implication only widens the other way — an `apps-of-user:<uuid>` grant covers both modes — so a scan for `:read` missed a `:write` grant, and `puter.perms.check('apps')` reported an app holding write as holding nothing. A batched request would then prompt again for access already granted. Adds the read-from-write exploder for both namespaces, mirroring `fs-access-levels`. The widening runs one way only, and does not cross into another user's namespace; both are covered by tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(perms): answer the app-root-dir check without provisioning it `/auth/request-app-root-dir` conflates two questions: may the caller claim its root directory, and where is it. The second provisions `AppData/<uid>` on first ask, so a caller that only wanted the first — `puter.perms.check('appRootDir')` — created a directory by asking about it. Adds `check: true`, which runs the same actor guard and stops at the answer. A caller that may not claim it still gets the 403, so the flag can't widen anything. Existing callers are unaffected: without it the route behaves exactly as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(perms): put request() and check() on one path `request` dispatched to the old per-task methods while `check` asked the permission tables, so the two answered different questions about the same access. Concretely, before this: `request('folder', { access: 'write' })`, `'apps'`, `'subdomains'`, `'appData'` and `'permission'` prompted every time, whether or not the access was held — which the docs said they wouldn't; `check('folder')` reported false for a folder the app could read through an ACL grant that no `fs:` string names, so a batch prompted for it needlessly; a batch entry for `'appRootDir'` skipped the post-grant retry the single call does, resolving `undefined` after a grant that had in fact succeeded; and an N-entry batch made N permission reads plus 2N `whoami` calls. Both now run the same pipeline — resolve the permission strings, read what is held once, prompt for the remainder, resolve each entry — with per-resource hooks for the parts only that resource can answer. So a batch costs one permission read and one `whoami`, a check reports exactly what a request would skip the prompt for, and `'folder'` uses the same stat-or-permission reading in both. Also: - A resource is looked up as an own property, so `request('constructor')` is the permission string it always was rather than a TypeError. - A permission read that fails no longer decides anything: `request` falls through to the prompt it would have raised anyway, `check` throws. Before, `check('appRootDir')` folded a failed check into "not granted", which is what the documentation says must not happen. - Drops `requestFolder`, `requestApps`, `requestSubdomains` and `requestAppRootDir`. They were added in this branch and immediately deprecated — never shipped, and `request()` no longer needs to route through them. The 22 methods that did ship keep their exact behaviour, prompting without consulting what is held, which the suite now asserts alongside the new behaviour. - Documents `'appRootDir'`, which was a supported resource in every overload and in `PermsResource` but named in none of the docs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(perms): act on review of the three preceding commits - `src/puter-js/test/perms.test.js` still called `requestApps`, `requestFolder`, `requestSubdomains` and `requestAppRootDir`, which the previous commit removed. Four cases in the interactive browser harness threw. Pointed at `request(...)` instead. - `request('appRootDir', …)` made two round trips where the shipped method makes one: a read-only probe, then the call that names the directory. A request is going to claim it either way, so the claim is now the check, and the entry it returns carries through to the result. `check` keeps the read-only mode, which is the reason that mode exists. Matters because the route sits on the FS_SIGN bucket, shared with signed-URL minting. - `requestPermission` is one of the shipped methods, and the previous commit's message was wrong to say all 22 keep their exact behaviour: it forwards to `request`, so it now settles a permission the caller already holds instead of prompting for it. The value can differ, not just the prompt count — a user who would have clicked Deny on a re-prompt used to get `false`. It is the more honest answer (the app does hold the access, and denying a re-prompt never took it away), but it is a change, and the suite assertion had been switched to an unheld permission, which hid it. Asserted both ways instead, in the unit tests and the API suite. - An entry that names no permission no longer rides a grant given for the other entries in the same call. Unreachable today — every resource either names one or reports itself held — but nothing pinned it. - Reverted three type-union reformats in `LegacyFSController.ts` that a formatter had folded into the app-root-dir commit. That file was not prettier-clean to begin with; reformatting it is somebody else's change. - Docs and types: `Perms.md`'s `appRootDir` row now matches `request.md`'s, `check.md` says that a `true` is per entry and a batch still prompts if any one entry is missing, and `types.js` no longer names `requestFolder` / `requestAppData` in prose that ships in the generated declarations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Daniel Salazar <daniel.salazar@puter.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 23:48:08 -07:00
3 people	b795b219a8	✨ PUT-1497: Share file link sharing and notifications (#3595) ... * refactor(share): move share notifications into their own service * feat(share): invite an address with no account, and email it * feat(share): surface pending invites in the SDK and share dialog * fix(share): unreachable revoke confirmation, and double-encoded labels * feat(share): budget share announcements, group them, and let people block senders Sharing had one defence against noise: a 15-minute quiet window per (sender, recipient) pair, which dropped the second share rather than folding it in. Twenty senders each under their own window could still bury someone, and there was no way to make one of them stop. Announcements are now budgeted on two axes through the existing sliding-window limiter: 1 per 15 minutes and 20 per day from one sender, and 10 per hour / 50 per day to one recipient from anyone. Over budget the share still succeeds and the recipient's notification is still brought up to date — only the interruption is dropped. Invite email to an address with no account is budgeted the same way, keyed on a hash of the canonical address. Notifications now fold across senders: a new share rewrites the notification the recipient hasn't dismissed, so "alice and bob shared 5 items with you" replaces a stack of five. The record is written even when suppressed, so the count is right whenever they next look. Blocking is a new `user_block` table with enforcement in ShareService: a blocked sender's share is refused with `recipient_not_accepting_shares`, spends no quota, and writes no row, and their unclaimed invite is dropped when the address is confirmed. Existing access is untouched — that is what revoke is for. Managed from a Blocked people card in the dashboard's Security tab. Also publishes the sharing limits, including the ones already on this branch that were never documented. * fix(share): name the item in share email, instead of 'an item' * fix(share): make the invite lifecycle canonical, authorized, and race-safe * refactor(email): drop EmailClient.isConfigured; callers read config.email * feat(share): batch share email into a per-recipient digest, durably * docs(share): document the share error codes; steady the disk migration tests * fix(share): log why a digest wasn't sent, and recover orphaned ones * feat(share): email recipients about shares by default, with a way to decline Share email was off unless a deployment opted in, which meant an account holder was told about a share in the app only. It is now on unless `share_email_notifications` is set to false. The reason it defaulted off was that nobody could decline. So this also honors `user.unsubscribed` — the account-wide opt-out the /unsubscribe page already writes and app feedback already respects, which share email ignored — and the digest carries that link. Sharing and the in-app notification are unaffected by it; only the mail stops. The link is composed in the template around an interpolated uuid rather than passed pre-built: Handlebars escapes interpolated values, so a whole URL came out as `user_uuid&#x3D;…`, which browsers decode but link scanners and older mail clients need not. * fix(share): count every shared file in the digest, not just the first * feat(share): let a recipient refuse shares from everyone Blocking answered "not from this person" but had no answer to "not from anyone", so the only way to stop a stream of unwanted shares was to name each sender after they had already reached you. Stored as a key in the user row's existing `metadata` blob rather than a column: the share path already holds the recipient's row by the time it asks, so reading it costs nothing, and a one-bit preference doesn't earn a migration per dialect. `updateMetadata` merges and refreshes the cached row, so the switch bites on the very next share. Refusing everyone reports the same code as refusing one person — which of the two it is is the recipient's business, not the sender's. Enforced at both moments the per-sender block is: when the share is issued, and when a pending invite is claimed. The per-sender list is untouched while the blanket switch is on, so turning it off restores what it hid. `GET /share/blocks` now carries `all`; `POST`/`DELETE` take `{ all: true }` beside the existing `{ username }`. Managed from the same Blocked people card in the dashboard's Security tab. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(share): keep the digest sweep off a window that still has a timer The sweep treated an entry as orphaned the moment its window closed, which is also the moment the node that armed it fires. Claiming an entry is only exclusive among flushers that can see each other's deletes, so the two could each claim a share of the same digest and both send. It now waits out a grace period first, which costs a genuinely stranded digest that much delay and nothing else. Both digest listings were capped at 200 with no word when they hit it — a truncated flush sends a digest that undercounts and reads as complete. The cap is named and logged. Also: `#emailHolder` still described share email as off by default, which it stopped being; the config doc said the batch window defaults to 60s when it is 90; and the two tests that need several calls inside one window were racing a 50ms window across four sequential round trips, so they failed under full-suite load rather than on the behaviour they cover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(share): stop double-encoding the recipient in two dialog messages `i18n()` encodes what it returns, replacements included, so encoding the recipient first showed the entities to anyone whose address or username contains one. Same pattern already fixed two lines above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(notification): widen the mysql shown/acknowledged columns Both arrived from the v1 schema as `tinyint(1)`, where they were flags. The backend rework changed the writes to a unix second; sqlite (`INTEGER`) and postgres (`bigint`) took it and mysql did not, so on mysql every `markShown` and `markAcknowledged` has failed with ER_WARN_DATA_OUT_OF_RANGE and left the column NULL. Dismissing a notification never stuck — the unacknowledged count never moved and one already delivered came back on every reconnect. No backfill: every reader tests `IS NULL` / `IS NOT NULL` only, so a legacy `1` keeps meaning "yes" once widened. Guarded on the current type, because changing a column type copies the table and this directory replays on every boot. Not reachable from the test suite — it runs against sqlite and postgres, both of which already have the right type. Verified by hand against mysql: `/notif/mark-read` and `/notif/mark-ack` now persist, and a dismissed share notification is no longer the one a later share folds into. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Daniel Salazar <daniel.salazar@puter.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-18 09:36:33 -07:00
Daniel Salazar	bd06e88185	feat: bring back referrals PUT-1463 (#3599)	2026-08-17 23:49:09 -07:00
3 people	2c852bf6b3	✨ PUT-1412 File sharing backend api (#3553) ... * refactor(permissions): drop hardcoded group permission map for a flat default * test(drivers): assert credential-gate intent instead of a 403 proxy * fix(permissions): report whether a revoke removed anything and persist the linked grant row before the flat view * feat(permissions): replicate permission invalidations across regions * feat(share): extend the share table into an index of active shares * feat(share): query and maintain active shares in ShareStore * feat(users): add a batched lookup by email * fix(cache): apply cache updates broadcast from peer regions * fix(permissions): scope a revoke to the issuer that granted it * feat(share): add ShareService with a per-day share limit A share is two writes that belong together: the permission grant, which authorizes access, and a share row, which makes it listable and ties it to an fsentry so it dies with the file. Nothing else grants fs:* to a user. Authorization reuses canManagePermission — an owner satisfies it through the is-owner implicator, a delegate through an explicit manage:fs:<uid> grant. An owner may clear any issuer's share of their node; anyone else only the ones they issued, or their own access. Self-revoke skips the manage gate but still requires `see`, so it cannot be used to probe for files. The per-day limit counts shares created rather than live rows, so revoking and re-sharing cannot recycle a slot, and changing an existing share's mode is not new reach and does not spend budget. Tunable via share_daily_limit. * feat(share): expose sharing over HTTP POST /share, POST /share/revoke, GET /share/shared-with-me, GET /share/shares. The controller was registered but entirely commented out. Recipients × items fan out concurrently — every pair is a distinct (holder, entry) key, so none of them contend — bounded by runWithConcurrencyLimitSettled, which returns results index-aligned with the input for the per-pair outcome list. Responses carry usernames only, never internal ids, and the 404-not-403 rule is preserved so a failed call cannot confirm a file the caller could not otherwise see. Notifications are fired off the response path; a share must not fail over its own notification. Per-request caps on recipients and items bound one call's fan-out; the daily limit bounds the total. * feat(share): keep recipients consistent when a shared item changes * fix(fs): stop listing issuer homes at the filesystem root * fix(acl): serialize concurrent mode changes on one node and pin app containment on shared paths * fix(fs): expire signed URLs over entries the signer doesn't own signFile defaults to a ~317k-year TTL and verifySignature checks only uid, expires and signature — never the ACL. A recipient who ever signed a shared file therefore held a permanent, revocation-proof URL to its bytes: revoking the share did nothing to it. signEntry now takes the acting user and drops to NON_OWNER_SIGNATURE_TTL_SECONDS (1 hour) when the signer is not the entry's owner. Owners keep the permanent default, so no existing client changes behavior. The signature-authenticated directory listing bounds its children unconditionally: that route has no session actor, and a signature proves possession rather than ownership, so a recipient holding a short-lived directory signature could otherwise mint permanent URLs for every child. A bounded window is not revocation — the durable fix is a per-entry signature epoch folded into the HMAC and bumped on any permission change. * refactor(permissions): drop the unused permission-issuer lookup listUserPermissionIssuers and its store method listUserPermissionIssuerIds existed to synthesize the filesystem root from the home directories of everyone who had granted the caller a permission. That listing is gone — it advertised folders readdir then refused to open — and the share index answers "who shared with me" directly, so nothing wants them back. One removed test only asserted that the call returned an array; the other covered readLinkedUserUserPerms round-tripping and is kept, rewritten without the issuer lookup. * fix(share): return the created share, not just an acknowledgement * feat(puter.js): add file sharing to puter.fs share(), unshare(), listShared() and getShares() on puter.fs, following the existing FS operation shape: positional and options-object forms through defineOperation, JSDoc overloads as the published signature, relative paths resolved against the app's root directory. A bare recipient string is read as an email when it contains @ and as a username otherwise. Sharing an item with someone who already has it replaces their access rather than stacking a second grant, so raising read to write is one more call. Adds a sharing suite to the API runner, which passes unchanged on node, browser and workerd. Documents all four methods with runnable examples, and corrects the FS overview callout that told readers one user cannot read another's files — true before this, not after. * feat(gui): add a Shared folder for items others shared with you A sidebar entry listing everything other users have shared with you, backed by puter.fs.listShared(). The path is the sentinel `puter://shared` rather than /<user>/Shared: this is a query, not a directory, and a path-shaped value could collide with a folder someone actually creates. refresh_item_container and update_window_path both branch on it to skip the stat there is no fsentry for, and the listing swaps readdir for listShared. Entries render at their real paths under their owners' directories — the item container already preferred an explicit fsentry.path over joining onto the container, so nothing else had to change. Each carries who shared it and at what level, which the context menu reads next. * feat(gui): share items from the context menu A sharing dialog shaped like its neighbours — options object, HTML-string template, jQuery wiring, delegating to UIWindow() — with a recipient field, a read/edit/share dropdown, and the current access list with revoke buttons. Reached from a new "Share…" context menu entry, which is hidden on items shared *with* you: re-sharing needs manage, so the dialog would only surface an error. Those items get "Remove from Shared" in place of Delete. Delete moves an item to *your* trash, which for someone else's file means moving their data out of their tree — FSService refuses it, and the user saw a bare 403. Removing your own access is what the action was reaching for, so that is what it now does. * fix(share): withdraw what a removed recipient re-shared * feat(share): report access inherited from a parent folder * fix(gui): load the puter.js bundle the server configured * refactor(gui): extract the action icon set into a helper * feat(gui): surface Shared in the file browser * feat(gui): manage access from the share dialog * test(share): cover access inherited from a parent folder * fix(share): keep downstream access from surviving a delegate who leaves * fix(gui): name the real owner in the share dialog * fix(gui): page through every shared item instead of the first 50 * feat(share): return item metadata with a share * fix(share): invalidate a holder's cache when the entry is deleted * fix(gui): treat items inside a shared folder as someone else's * feat(permissions): let manage inherit down the filesystem tree Access already reached descendants through the ancestor chain while authority did not, so someone trusted to manage a shared folder could re-share the folder but nothing inside it, and could not see who had access to a file within it. A manage-inherits-from-ancestor implicator resolves it in the permission layer, beside is-owner, so every caller agrees rather than just ShareService. It consults only the immediate parent — resolving that re-enters one level up, making a chain of depth d cost d checks rather than d². That makes two cascade gaps reachable, both fixed here. A revoke now walks the subtree, since a grant on a descendant can rest on authority held at the folder. And it stops at a delegate whose authority survives another issuer, because what they granted was never theirs to lose. Also pins that manage is not transitive: granting it needs manage:manage:fs:<uid>, which only the owner holds, so delegation is one level deep by construction. * fix(gui): offer sharing inside a folder you manage The menus encoded "manage does not inherit" and would now hide an action that works. The Shared listing records each root's mode; the menus resolve a child's by longest matching ancestor, loading on demand so a deep link or restored window works too. * fix(share): make the daily share limit hold under concurrency * test(share): cover concurrency, measure cost, and name cases for what they verify * fix(gui): import the ownership helpers the item menu calls The single-item context menu handler calls is_owned_by_me and shared_mode_for, but the imports were only ever added to generate_file_context_menu.js — so every right-click on an item threw a ReferenceError before the menu could build, and the non-owner Delete gating never ran. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(share): authorize before resolving the recipient share() looked up the recipient in parallel with the entry, before the manage check — and the two failures carried different error codes. Any verified user with a real entry uid could probe arbitrary emails and usernames for account existence, at no quota cost. Resolve the entry, authorize, and only then resolve the recipient: an unauthorized caller now sees the identical safe 404 whether or not the recipient exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(permissions): broadcast permission row-cache invalidations to peer regions Every publishCacheKeys call for the u2u, u2a, and access-token row caches omitted broadcast, so a revoke only cleared the mutating region's Redis. A peer region applied the replicated generation bump, re-scanned, read the deleted row from its own still-warm 5-minute row cache, and re-warmed the flat view from it — revoked access outlived the revoke by the row-cache TTL instead of the intended 60-second bound. CacheReplicationService already consumes these events; the emits were just never sent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(fs): refuse to rename an entry owned by another user remove and move both refuse to act on an entry the caller does not own, even when the ACL allows the write — rename had no such guard, so a write-mode share recipient could rename the owner's file, or the shared folder itself, rewriting the owner's whole subtree's paths. rename now takes the acting user and applies the same policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(permissions): decide a flat delete from the primary, not a lagging replica revokeUserUserPermission deletes the SQL grant, then only drops the flat KV entry once no issuer still grants the permission. That remaining-check read through the row cache the delete had just invalidated, straight to a replica — under any lag the deleted row reappeared, the flat delete was skipped, and the stale rows were re-cached for another five minutes. Grant-path flat entries carry no TTL, so the holder kept working access with zero SQL rows behind it, invisible to every listing. The check now reads the primary and re-warms the cache with what it actually saw. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(permissions): keep a failed remote flat-invalidation from crashing the process The outer.permission.flatInvalidated applier was fire-and-forget with no catch, and it awaits a KV delete — one transient KV error while applying a peer region's revoke became an unhandled rejection, which is process-fatal under default Node. Its sibling appliers were already guarded; this one now logs and moves on, leaving the entry to the next invalidation or its TTL, same as a lost event. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(share): revoke every requested item and recipient, not just the first revokeShare destructured only the first recipient and first item while the parsers accept arrays up to the request caps — unshare({items: [a, b, c]}) returned success having revoked only a, leaving access the caller believes is gone. Revoke now fans out over every (recipient, item) pair exactly like POST /share, reports per-pair outcomes, and sums the revoked count; the response stays backward compatible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(share): only a confirmed email designates a recipient Recipient resolution by email accepted unconfirmed accounts, so pre-registering someone else's address (unconfirmed) was enough to receive shares meant for them once no confirmed account held it. An email now only resolves to an account that has confirmed it; username shares are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(share): accept tilde-rooted paths like the FS routes do The SDK resolves relative paths to ~/..., but the share routes never expanded the tilde — a ~-prefixed string was read as a uid and every relative-path call 404'd. Item parsing now treats ~ as path-shaped and expands it to the actor's home with the same helper the legacy FS routes use, on share, revoke, and the shares listing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(share): walk a directory revoke by parent linkage, not path prefix listByFsentrySubtree matched descendants with fsentry_id = ? OR path LIKE ?, which has two problems: fsentries.path is lazily backfilled and NULL on old rows, so those descendants' shares silently survived a directory revoke, and the OR'd predicates forced a scan of every active share. A recursive CTE over parent_id — the same shape the lineage resolver already uses — covers every descendant and runs on idx_parentId_name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(gui): give each item its own share dialog single_instance keyed the dialog on the app id alone, so opening Share… on a second file focused the first file's dialog — typing a recipient there granted access to the wrong file, with only the title hinting at it. The dialog is now instanced per path: same item refocuses, different item opens fresh. Also stops pre-encoding the title, which UIWindow encodes again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(permissions): let manage answer a write check A manage grant let its holder re-share a folder but not work in it: the ACL mode family stops at write, and the fs exploder had no rule for the narrowest mode, so `manage:fs:<uid>` never satisfied `fs:<uid>:write`. Fold manage into the candidate list for every non-manage mode, in both the access-token branch and the scan branch, and give `write` an (empty) exploder rule so the manage arm is emitted for it too. * fix(fs): authorize restructuring by write on the parent * fix(fs): let a share recipient work inside a shared folder rename, remove and move refused outright when the entry belonged to someone else, so a recipient with write could neither delete nor rename anything inside a folder shared with them. The GUI compounded it by hiding Delete for any item it did not own. Authorize the three by ACL write on the entry's parent. For an owner that is the same answer; for a recipient it grants the inside of a shared folder and withholds the folder itself, whose parent is the owner's private tree. Deleting sends the item to its owner's trash rather than the deleter's, so it leaves the recipient's view without leaving the owner's account and without changing hands. A move may not otherwise carry someone else's entry out of their tree. * fix(fs): give a new entry to the owner of the folder it lands in A file a share recipient added to a shared folder was recorded as theirs while living in the owner's tree, so a subtree could hold rows belonging to several people — and the storage it consumed was checked against the writer while being counted against the owner. Take the owner from the parent row at every insert, charge the allowance to that owner, and hand a moved entry over to the tree it moves into. An entry now always belongs to whoever owns the directory holding it. * feat(fs): address shared entries as ~/share/<uid> A recipient could read the owner's whole path off any shared entry — where they keep the file and what sits beside it, neither of which the share is about. Give shares their own namespace. `~/share/<entry-uid>/rel/path` resolves to the real path on the way in, and outgoing paths are rewritten to it on the way out. Entries the actor owns pass through untouched, so no existing client contract moves. * revert(fs): mask only the directory bar, not the addressing fe535d598 made `~/share/<uid>` the actual address for every shared entry. That reached far past the intent: item names became uuids, the Shared views rendered uuids instead of filenames, and navigation addressed entries through a namespace nothing else understood. Put real paths back everywhere — responses, the share listing, and request handling — and do the masking where it was wanted, in the window's directory bar. A recipient sees `Shared › Contents › sub` while every crumb keeps the real path it navigates to. The share listing now carries the entry's name, content type, owner and a signed thumbnail. A share row has no fsentry behind it for a client to stat, and the stored thumbnail is an `s3://bucket/key` URI that no client can render and none should see. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: path obfuscation, webdav + small ui stuff * test(share): assert the masked share path by its exact shape The substring check tripped on the scratch files' own names, which start with `sharing-`; the exact-equality assertion on `/<owner>/<uid>/<name>` already proves nothing above the share leaks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(share): rename policy, shared-view guards, webdav share parent, quota/bucket invariants - Rename: a directly-shared FILE renames with write on it; a shared folder root stays fixed (its name is the owner's tree structure). GUI can_rename mirrors the backend, guards all editor entry points. - Up from a share root goes to the Shared view on both surfaces; the Shared view gets a single crumb, a disabled Up button, and refuses drops, New/Paste, uploads and ctrl+V everywhere (it is a query, not a directory). - WebDAV: PROPFIND on /owner/uuid answers as a virtual collection holding the share root (ACL-gated; 404 for strangers). - Storage allowance override no longer crosses user boundaries: a recipient's plan cannot raise the owner's cap. - Overwrites stay in the bucket the entry already lives in instead of repointing to the handling server's bucket and stranding the old object. - manage mode documented as implying write (matches enforcement); share dialog label now "Can edit & share". Documented that fs socket events are owner-only. - Sidebar: saved orders gain the Shared entry once the user has shares. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(share): close review findings, and retire /auth/grant-user-user Security - Recipient writes no longer echo the owner's real path back. The pending-write event and the upload-progress meta both go to the *acting* user, so a write into a shared folder handed out the layout above the share root. - `maskerFor` failed open: the first caller fixed the actor, and one that ran before the request knew who was acting pinned `undefined` — after which every path published unmasked. It now adopts the first real actor and rebuilds for a different one. - `/auth/grant-user-user` returns 501. It wrote user-to-user grants straight to the permission tables with no share row, so nothing could list or cascade a revoke over them — and the new `manage` mode meant a delegate could reach it for the owner's files. Filesystem access goes through `/share`; nothing else is meant to pass between two users. Undocumented (its docs page was never written) and no callers. Revoking is untouched, so grants made before this can still be withdrawn. - `#assertCanManage` asks about the permission actually being granted rather than a fixed `read`, so sharing at `manage` is refused at the share layer instead of by `grantUserUserPermission` two levels down. - `listSharedWithMe` checks the grants, not just the index: withdrawing access any other way left the row publishing name, size and a signed thumbnail URL. - Moving an item into another tree now retires its shares. Grants are keyed on uuid, so they followed it and left the new owner with recipients they never agreed to. Correctness - `acl.check` gets the real path again, not the masked one. ACL matches on the path string, and a mask hides the `AppData/<appUid>` shape it needs. - "Leave this share" works for a grant that predates the index; the fallback scoped the delete to the caller as issuer, which can never match. - `move`/`copy` reject a name with a slash or a `.`/`..` segment, as `rename` already did. It matters more here: a move into the owner's Trash skips the destination write check. - Descendant walks scope by path, not by owner, so rows predating the one-owner-per-subtree invariant aren't orphaned by their parent's deletion. Performance - Index `user_to_user_permissions(permission)`. Retiring a node's grants is a prefix match, but the primary key is (issuer, holder, permission), so it was a full scan. - Retirement is coalesced and chunked. `remove()` emits one event per descendant, so deleting a directory fired one unindexed lookup per file, all at once and unawaited. - `listReaching` is a plain `fsentry_id IN (...)` on `idx_share_fsentry`. It runs behind every file write, and the join-plus-OR it replaced was unindexable. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Daniel Salazar <daniel.salazar@puter.com>	2026-08-17 09:08:23 -04:00
Daniel Salazar	22f5bf5429	fix: duplicate emails (#3556)	2026-08-12 22:00:48 -07:00