224 Commits

Include renames

Author	SHA1	Message	Date
Daniel Salazar	590c87c00a	refactor(stores): one row cache and keyset page helper, batched cache round trips, dead store methods removed (#4191) ... * refactor(stores): one read-through RowCache for UserStore and AppStore UserStore and AppStore each carried the same multi-key row cache: key builder, cache read, tombstone-guarded backfill, broadcast refresh, invalidate and tombstone-on-delete. They now share RowCache on PuterStore, which also single-flights concurrent misses for one key and batches getByIds/getByUids: one pipelined tombstone check for the loaded rows and one pipelined backfill instead of per-row awaits. Cache keys and TTLs are unchanged. * perf(apps): invalidate cached app lists by generation, not by tracker scan Every app write SMEMBERS'd a global set of every cached list key, parsed each key back into its filters and deleted the matches. List pages are now keyed by their owner's generation token (or a shared one for unscoped lists), and a write replaces the tokens it affects: O(1) per write, and a page loaded before a write can no longer be cached after it. * refactor(stores): one keyset page helper for store listings Thirteen store listings each cut a LIMIT n+1 read to a page and sealed the last row's position, and each opened an id cursor its own way. They now use keysetPage() and openIdCursor() from util/pagination. Cursor payloads and sealing are unchanged; a cursor without a usable id still fails with 400 in ShareStore and still restarts the listing elsewhere (strict option). * refactor(stores): delete store methods nothing calls AppStore.existsByIndexUrl, OIDCStore.unlinkByUserId, TeamStore.listByOwner/listActiveSeatUuids, SubdomainStore.getByDomain/ listByDomain, FSEntryStore.ensureDirectoriesForUser, ShareStore.listByIssuer/listByRecipientEmail/deleteByRecipientEmail/ getDailyShareCount/listByHolderAndFsentries and PermissionStore.invalidateScanCache/deleteUserUserPermsByPermissionPrefix had no production callers. ShareStore.listByFsentries was a copy of listReaching; its callers use the latter. SubdomainStore SMEMBERS'd a per-user prefix-list tracker on every write and broadcast its deletion, but nothing ever wrote that set or cached a prefix listing. The reads are gone. * refactor(stores): shared escapeLike and bumpGeneration helpers escapeLike replaces the LIKE escaping copied into PermissionStore (twice), ShareStore and FSEntryStore, and now also guards SubdomainStore's prefix filters, which passed caller prefixes into LIKE unescaped. bumpGeneration sends INCR and EXPIRE in one pipeline instead of two awaited commands in PermissionStore, EventSubscriptionStore and PresenceStore. * refactor(stores): read affected rows off WriteResult Every database client returns affectedRows and anyRowsAffected, so the `affectedRows ?? changes ?? 0` fallbacks in ShareStore, NotificationStore, UserStore and UserBlockStore read those directly. * perf(stores): batch flat-permission KV deletes and durable cache drops PermissionStore deleted each flat KV entry with its own kv.del; one kv.batchDel now covers the set. DurableSubscriptionStore's suspend and reap dropped each row from the cache separately; EventSubscriptionStore gains dropDurables, which drops them per owner in one pass, and dropDurable calls it. suspend keeps one conditional UPDATE per row, which is how concurrent settles learn which rows each of them suspended. * refactor(fs): fold FSEntryStore's duplicate getters, sort maps and cache keys - getEntryByUuid and getEntryById share one by-column getter. - listChildren and listChildrenPage read one sort table; the offset listing now coalesces NULL modified/size like the cursor listing, so both order NULLs the same way on every engine. - Entry cache keys come from one builder instead of hand-built strings. - getEntriesByIds, getEntriesByPaths and the per-user path reader fetch their cache hits in one pipelined round trip instead of a GET per key, through readCacheValues, which RowCache.readMany now uses too. * fix(apps): cache the row a lost createFromOrigin race reads from the primary Keeps AppStore's private #writeCache as a RowCache write, so code that still calls it (#4166's read-back) loads in either merge order.	2026-10-11 22:07:53 -07:00
Daniel Salazar	45f6e9d8ee	refactor(drivers-wiring): AppService views and launch gates, batched app lookups, shared driver policy and guards (#4186) ... * perf(workers): skip hot-reload lookups for files no worker is bound to The fs.write.file / fs.remove.node / fs.move.node listeners listed every workers.puter.* row of the file's owner on every event, for every user. The event's FSEntry already carries the worker rows bound to it, so a file with none returns before any I/O; a bound file reads just its own rows, batched, from the primary. * perf(apps): read the uid alias only after a direct lookup misses Every AppDriver uid lookup fired an uncached system-KV GetItem for the merge alias in parallel with the direct read, even when the direct read hit. The alias is now read only on a miss. * refactor(apps): one app view with the launch gates, built by AppService AppDriver's serializer and its private-app / hosted-backing gates move to AppService.views, built on a pure toAppView that is the one place index_url is withheld. AppDriver, the subdomains driver's embedded apps and the GUI shell view all use it. Visibility of protected apps (filterReadable) moves with it. Subdomain embeds (associated_app, app_owner) used to return index_url with no gate; they now withhold it from a viewer who may not open a private app or whose hosted backing is gone, and carry privateAccess, godmode and feedback_enabled like a direct app read. * refactor(apps): one toAppSummary; batch the recommended and taskbar lookups SuggestedApps, RecommendedApps, /get-launch-apps and the taskbar each built their own launch summary; they now share toAppSummary (the extra feedback_enabled / external flags appear on the lists that lacked them). getRecommendedApps awaited getByName once per recommended app (29) and getTaskbarItems did the same per taskbar entry on every whoami; both now make one batched call per key through the new AppStore.getByNames, which keeps getByName's case-insensitive and old-name matching. The taskbar's raw UPDATE of the user row goes through UserStore.update, and /get-launch-apps checks hosted backings in one batch and reaches the stores without casts. * refactor(apps): app routes read through AppService; /delete-site calls the subdomains driver AppController used the global driversContainers through a getter whose boot-order comment no longer held; it uses the injected drivers and AppService. GET /apps/:name looked each name up, read it again through the driver and resolved private access a second time, emitting app.privateAccess.resolveLaunch twice per private app; it now batches the names and resolves once. POST /query/app did a lookup, a driver read and a raw app_filetype_association SELECT per selector (up to 200, serially); it now makes one batched lookup per key, one grant check and one filetype read. The driver's read shape is a pure toAppReadView both use. HostingController's /delete-site re-implemented the subdomains driver's delete (protected check, delete, subdomain.delete event); it keeps its owner-scoped 404 and calls the driver. Hosting and Desktop controllers drop constructors that discarded the drivers argument, and trivial getters. * refactor(drivers): one per-method policy validator and resolver; drop @Driver Four policy blocks (rateLimit, concurrent, requireSubscription, requireReputation) share the { default, methods } shape but each had its own validator and resolver. They become validatePerMethod(value, label, policy) and resolvePerMethod(cfg, method); error messages are unchanged. The @Driver class decorator and the prototype keys it wrote had no users outside its own tests; drivers declare their fields, and resolveDriverMeta reads only those. DriverController takes the driver name from the meta it already resolved instead of probing the prototype. resolveDriverMethodRateLimit/Concurrent stay as deprecated aliases for the email extension. * refactor(drivers): shared actor guards and crud-q concurrency policy The apps, subdomains, workers and notifications drivers each carried a private #requireActor, two carried the same #requireVerified, and apps and subdomains the same owner/app_owner write predicate. These become requireContextActor, requireContextUserActor, assertActorEmailVerified and actorOwnsRow in core/actor.ts; each driver keeps its own error for the cases that differed (system actors still read apps; notifications still answer 403 for an actor without a user). The identical crud-q concurrency block is one CRUD_CONCURRENT constant. The metering, installedApps and appTelemetry extensions re-checked an actor their routes already require; those checks and the tests for them go. * refactor(extensions): store methods instead of raw SQL in OSS extensions installedApps and appTelemetry queried user_to_app_permissions directly and metering read apps by name with its own SELECT. PermissionStore gains listAppsGrantedByUser, listAppAuthenticatedUsers (with the email-share check folded in) and countAppUsers; metering uses AppStore.getByName. * refactor(server): one layer builder and one lifecycle hook runner PuterServer built each of its five layers with its own copy of the same loop (built-ins, then extension additions, into the layer and its export container), and ran each lifecycle hook with five near-identical loops per hook. Both are now one helper each: #instantiateLayer and #runHook(order, hook). Construction order, the args each layer receives, and hook order (start and prepare bottom-up, shutdown top-down) are unchanged. * fix(workers): tear down a deleted source's worker after its FK nulls root_dir_id	2026-10-11 22:04:23 -07:00
Daniel Salazar	576e893a5b	refactor(services-core): read-only socket reauth, one token-grant read per check, shared grant/revoke and hosted-asset paths (#4187) ... * perf(socket): re-check live sockets without touching their sessions The 5-minute sweep reused the request-path authenticate(), which slid every connected session's expiry and wrote user activity each tick, and could fill SessionStore's touch-throttle map. authenticate() now takes touch: false for re-checks; the sweep uses it, checks credentials through runWithConcurrencyLimitSettled, and a call while a sweep runs joins it instead of starting another. * perf(acl): read a scoped token's grants once per check ACLService.check did a Redis GET + JSON.parse of the token's grant list for every ancestor x mode, and the permission scan did the same per option. Both now read the list once (PermissionStore.listAccessTokenPerms, which replaces the per-permission hasAccessTokenPerm) and test it in memory. * refactor(metering): incrementUsage goes through batchIncrementUsages incrementUsage was a copy of the batch path that let an infinite amount through and skipped NaN, where the batch path counts either as one unit. It now calls batchIncrementUsages with a one-entry batch, so both apply the batch clamping (documented on batchIncrementUsages). The overuse alarm loses the single-path fields nothing sets any more. * perf(metering): memoize an actor's spent addons on the increment path Every increment and credit refresh read the actor's addons record from the KV store. A record with no purchased credit left can only change through a grant, and every grant announces credits-changed, so it is now kept per actor beside the credit memo and dropped with it. Records with credit left are still read fresh, since increments on other nodes draw them down without announcing it. A read that straddles a drop isn't kept. * refactor(services): route raw table writes through their stores - access_token_permissions: AuthService inserted grants one row at a time through a cast to the store's db client, read the authorizer, and deleted the manifest with raw SQL, then asked PermissionStore to drop its cache. PermissionStore now owns insertAccessTokenPerms (chunked multi-row insert), deleteAccessTokenPerms and getAccessTokenAuthorizerId, and invalidates inside the writes. - TeamService's 2FA reset clears the OTP columns through UserStore.update. - AppIconService's data-URL icon swap is AppStore.replaceDataUrlIcon. * fix(user): correct cascadeDelete's FK note; seat cleanup is best effort The header said most user_id foreign keys are ON DELETE SET NULL; the migrations cascade nearly all of them and keep SET NULL for audit and analytics history. forgetSeat ran unguarded after the row was already gone, so a cache failure there failed the whole delete; it is now logged and stepped over like the other teardown steps. * refactor(acl): one grant/revoke path per holder kind ACLService's user and group stat/set pairs were copies differing only in which PermissionService methods they called. They now share #stat and #set over a holder adapter ({ lockKey, query, grant, revoke }); lock keys and errors are unchanged. PermissionService's eleven grant/revoke methods repeated the same guard blocks. Each keeps its own order of checks, now through shared helpers (#requireUserId, #refuseAppActor, #assertCanManage, #requireUser, #requireApp, #assertFitsColumn, #audit), so every refusal keeps its status, legacyCode and wording. Audit writes stay fire-and-forget. * refactor(auth): one hosted-asset token API keyed by kind The private-asset and public hosted-actor cookies had separate create/ verify methods, four cookie-name getters and two identical options getters, differing only in the kind claim and what the host field was called. They are now createHostedAssetToken(kind, claims), verifyHostedAssetToken(kind, token, expected), getHostedAssetCookieOptions(), and the exported HOSTED_ASSET_COOKIES names. Token claims, cookie names and every error message are unchanged; puterSite and privateAppGate call the new methods. * refactor(auth): one revoked/expired check for every actor builder The session, app-under-user and access-token builders (and the access token's parent-session check) each spelled out the same revoked/expired tests. They now share #endedSession, which returns the same reauth results; only a session token passes its auth_id. Adds coverage for an access token whose parent session expires, is revoked, or is gone.	2026-10-11 21:55:20 -07:00
Daniel Salazar	ecef154630	refactor(services): page fs subtree walks, gate share fan-out, one fs ACL helper (#4190) ... * refactor(acl): read ancestor chains from FSEntryStore and share one fs ACL helper getAncestorChain (split a path, batch-read the entries) moves to FSEntryStore, with getAncestors for callers that want the rows. Services read it from the store instead of calling FSService sideways; FSService.getAncestorChain stays for drivers and extensions. ACLService gains fsDescriptor (memoized chain), assertFsAccess, assertFsCreate and refusal, plus the pure aclRefusal mapping. The eight hand-rolled descriptor and safe-error copies in FSService, ShareService, events/authorization, FSController, legacyFsHelpers, LegacyFSController, AuthController and WebDAVController use them. Each route keeps its status, legacyCode and message: fs routes answer access_denied on a 403, the legacy routes still hide refusals from apps as 404, share and events keep forbidden and their own 404 text. * perf(fs): page recursive deletes and batch their event dispatch A recursive delete read every descendant into memory, and an empty-check for a non-recursive one did the same. It now walks the subtree a page at a time in descending path order (children before the directory holding them, so the parent_id cascade never takes a row before it is handled), deleting objects and rows and publishing each page as it goes. A non-recursive delete probes with one LIMIT 1 query. Removals publish once per entry, after the row is gone (the same change as the open fix for the double emit), with one surviving-ancestor chain read at most once for the whole subtree. Events-API dispatch for a batch runs as one background task at bounded concurrency instead of a fire-and-forget promise per entry. FSEntryStore gains hasDescendants and listDescendantsInOrder (keyset paged by path, then id). * perf(fs): copy directories a level at a time A directory copy walked every descendant in memory and copied them one by one: an insert per directory and, per file, an S3 CopyObject, a batchCreateEntries of one and its read-back. It now pages the source in ascending path order (parents before children) and takes each page a depth at a time: rows for directories, links and empty files at bounded concurrency, then the depth's S3 copies at bounded concurrency and one batchCreateEntries for all of them. A single-file copy goes through the same path. A failed S3 copy still fails the call (a ghost file still answers 404), but only after the objects that did copy have their rows, so nothing is left in the bucket unreferenced. Copied entries publish per depth in one dispatch. * refactor(share): build masked share paths in one place sharePathMask exports maskedRoot(owner, uuid, name) and maskUnder(root, realPath). ShareService's maskedPathVia goes, its maskedSelfPath and shareDeepLink's maskedSharePath build on them, and the masker, maskUnderAnchor and learnShareRoots use maskedRoot. Output is byte-for-byte the same; tests pin it. * fix(share): send old_path on a recipient's item.moved The desktop's item.moved handler reads old_path; ShareService sent the former masked path only as from_path. Both are sent now. * perf(share): skip the audience lookup for owners with nothing shared Every fs write, create, rename, move and delete looked for share recipients: an ancestor walk plus two share queries, for an owner who in nearly every case has shared nothing. #sharesReaching now reads the chain through FSEntryStore.getAncestors (no third copy of the walk) and asks a per-owner "has anything shared" flag before the share queries. The flag is the per-user cache events already keeps for subscriptions (SubscriptionCache): one ShareStore.ownerHasShares read per owner per TTL, coalesced across a burst. Creating a share (user, team, claimed invite) or moving a subtree to a new owner bumps it locally and over outer.pubsub.share.ownerShared; the TTL covers a process that misses the bump. A failed lookup reads as shared, so the full lookup decides. * perf(share): batch the per-row reads behind share listings Share listings looked things up one row at a time: - teams by id: TeamStore.getByIdsIncludingDeleted (the single-id read calls it) for listSharedByMe, listSharesOf and #revokeGroupSharesBy - team shares per node and ancestor: ShareStore.listGroupOnFsentries, one query for the chain, in the same node-then-ancestor order - #liveGroupGrants: PermissionStore.queryIssuerGroupPermsByPrefixes reads every issuer/group pair under every prefix in one query - #revokeGroupSharesBy: the five canManagePermission checks per row run together instead of one after another - ShareNotificationService #unblocked: users via getByIds and blocks via UserBlockStore.blockersAmong, two queries instead of two per member - digest flush: entries are taken at bounded concurrency rather than one at a time * perf(events): ask background consent once per holder and app on a revoke settle Withdrawing events:background re-checked consent row by row, resolving the holder's actor and asking the permission service for every one of their background rows. Rows of one holder and app stand on one consent, so it is asked once and shared. * refactor(events): drop the registry's unused push fields notify, defaultDelivery, PushMessage, NotifyProjection and pushProjection were read only by registry.test. Every subject was unpushable and broadcast by default; the subscription's delivery class stays in DeliveryClass, which the stores use. * refactor(share): retire trashed shares through onEntryOwnerChanged onEntryTrashed was a copy of onEntryOwnerChanged: the same subtree share sweep, grant retire and row delete. * fix(fs): page subtree walks by depth, not path MySQL sorts a PAD SPACE string on its first max_sort_length (1024) bytes, so under utf8mb4_unicode_ci paths past about 512 characters tie and fall back to id. A file moved under a newer directory then sorted after it: the delete walk reached the directory first and the parent_id cascade took the file without its object delete or event, and the keyset skipped or repeated rows. Pages now order by segment count, then id, which no collation can reorder. * fix(share): cache an owner as shared when the bump arrives A bump only dropped the cached answer, so the next lookup could read a replica that hadn't applied the new share yet and cache "nothing shared" for another TTL. The bump now records "shared" outright. * fix(fs): probe a directory for children by parent_id, from the primary * fix(fs): keep the copied-size check in the batched directory copy A copied file records the size CopyObject actually duplicated; growth past the source row is charged against the allowance, and a copy that no longer fits is deleted and refused, as a single-file copy already does.	2026-10-11 21:54:13 -07:00
Daniel Salazar	b5eefaa9cc	fix(fs): include supplementary-character names in subtree path queries on MySQL (PUT-2135) (#4198) ... * fix(fs): include supplementary-character names in subtree path queries on MySQL (PUT-2135) MySQL ends the idx_fsentries_path range for `path LIKE 'p/%'` at 'p/' + U+FFFF, and utf8mb4_unicode_ci sorts supplementary characters above it, so a child whose name starts with one (most emoji) was skipped with its whole subtree by listing, delete, copy, rename/move rewrites, counts and sizes. On MySQL, FSEntryStore now looks up those children by parent_id and ORs a LIKE pattern for each onto the subtree match, recursing into directories. Every extra pattern starts with the prefix, so only the scanned ranges widen; rewrites past 500 patterns run in several statements. The prefix rewrite also passed a UTF-16 length to SUBSTR, which counts characters, so a rename under a path holding an emoji dropped characters from every descendant path on all engines. It now counts code points. * test(fs): seed the supplementary-name subtree with createNonFileEntry	2026-10-11 21:21:18 -07:00
Daniel Salazar	6a938bd8e8	fix(db): detect duplicate keys with isUniqueViolation everywhere (#4175) ... FSService, OIDCStore, UserStore and TeamService each carried their own duplicate-key check. FSService only matched mysql and the bare SQLITE_CONSTRAINT code, so the mkdir/move race recovery never ran for a SQLite or Postgres unique violation. OIDCStore matched any message containing "UNIQUE" or the bare SQLITE_CONSTRAINT code, UserStore any SQLITE_CONSTRAINT_* code, and TeamService any message mentioning unique or duplicate. All four now use util/dbError's isUniqueViolation; UserStore still checks the index name on top.	2026-10-11 18:24:13 -07:00
Daniel Salazar	847a4fc031	fix(fs): route thumbnail writes through FSEntryStore (#4174) ... * fix(fs): route thumbnail writes through FSEntryStore Add FSEntryStore.updateEntryThumbnailByUuid, which sets a thumbnail with no owner check (callers authorize) and refreshes the entry cache, with an optional expected-value guard. The thumbnails extension's inline data: URL migration, its copy duplication, and the legacy /update-fsentry-thumbnail endpoint now write through it instead of raw UPDATEs, so cached entries stop serving the old thumbnail. The migration validates the image like the upload path, awaits the write, and only repoints a row that still holds the data URL; a read that loses that race deletes its upload. * fix(fs): report a thumbnail update by whether the row changed updateEntryThumbnailByUuid returned the refreshed entry, which is null when the cache misses and the replica hasn't seen the row yet. The read-time migration took that null for a lost race and deleted the object the row had just been pointed at. It now returns whether the UPDATE changed a row; the cache refresh still runs after it.	2026-10-11 18:23:58 -07:00
Daniel Salazar	7ad963bacf	fix(cache): broadcast fs entry and team invalidations to peer regions (#4169) ... * fix(cache): broadcast fs entry and team invalidations to peer regions FSEntryStore's invalidations called publishCacheKeys without broadcast, so a delete, move, rename or overwrite only cleared the writer's cluster; other regions served the old row until the 60 s TTL. They now broadcast. Batch paths (deleteEntries per chunk, the batch overwrite) send one event for the whole batch. Read-path backfills stay local. TeamStore's busts were a local redis.del, so a removed member kept passing getMembership in other regions until the TTL. They now go through publishCacheKeys with broadcast, and softDelete busts every key in one call. Remove FSEntryCacheInvalidationEventHandler. It re-derived the same invalidations from outer.gui.item.* with a replica SELECT per holder, and its fs.remove.node branch never ran (FSEntry has no get()). The store's own broadcast covers what it did. * fix(fs): cap how many entry invalidations a node broadcasts per window FSEntryStore now broadcasts at most 10,000 entries' cache invalidations per broadcast flush window (outbound_flush_ms, 2 s by default). Past that, keys are dropped locally only and peers fall back to the 60 s entry-cache TTL; the store logs once per window when this happens. Each flush goes to a peer as one POST that is re-sent whole on failure, so a large recursive delete could otherwise build a batch the peer rejects on every retry. The budget is per store instance, not per call, so a delete issued page by page is bounded too, and single-entry writes count against the same budget.	2026-10-11 17:53:26 -07:00
Daniel Salazar	b67acdbf6f	fix(stores): return written rows without reading them from a lagging replica (#4166) ... * fix(stores): read rows back from the primary after writing them ShareStore (create, upsertPending, upsertActive, upsertActiveGroup, applyPending, upsertAnyone), AppStore (create, createFromOrigin, update, claimOwnership), NotificationStore.create, SubdomainStore.update and NotificationDriver.create returned the row they had just written by reading it back through `db.read`, which goes to the replica. Under replica lag that came back null or stale. Those read-backs now go to the primary, through a `primary` option on the getters. AppStore caches the primary copy on create, and claimOwnership no longer reads through getById, which cached the replica's pre-claim row on a miss. * fix(stores): skip the notify read-back; read OIDC link conflicts from the primary NotificationStore gains `insert`, which writes and returns the uid; `create` is now `insert` plus the primary read-back. NotificationService .notify ignores the row, so it uses `insert` and no longer pays a primary round trip per recipient. NotificationDriver.create still reads the row back from the primary. OIDCStore.link reads the conflicting row from the primary after a unique violation, so a replica that has not seen another account's link can no longer make the call report success. * perf(stores): return written rows instead of reading them back ShareStore.create, NotificationStore.create, AppStore.create and AppStore.createFromOrigin build the row they return from the values they inserted and the insert id, instead of reading it back from the primary. The DB-defaulted creation timestamp is now written explicitly from unix seconds (`timestampFromUnixSql`), so the returned value matches what a read returns (`timestampValue`); postgres INSERTs add RETURNING id. The AppStore rows fill every column the INSERT leaves to the schema and are cached as before. ShareStore.upsertPending's re-invite branch selects the full pending row in its dedupe read and returns it with the new mode and data; it falls back to the primary only when the UPDATE matched nothing. The remaining primary read-backs stay and say why: the share upserts and the new-invite insert (an upsert's surviving uid/created_at and the entry-owner subquery are only known to the primary), applyPending and claimOwnership (only an id or uid in hand), AppStore.update and SubdomainStore.update (the base row is a replica copy that would be cached), NotificationDriver.create (notify returns only the uid) and the OIDC link conflict.	2026-10-11 17:51:38 -07:00
Daniel Salazar	491729e060	fix: backend test flakes and a socket shutdown hang (#4170) ... - SocketService's prepare-shutdown hook awaited io.close(), which also closes the http server and waits for every HTTP connection to end. A connection mid-request when close() runs stays open for the keep-alive timeout, so shutdown stalled before PuterServer severs connections. The hook no longer waits on the http close. - Test servers disable Ollama unless a test configures it. Its boot probe logged after files had ended, which failed shared-worker shards with EnvironmentTeardownError. - vitest ignores that teardown error for a console call still in flight. - setupPuterTestEnv boots again on a fresh port after EADDRINUSE, and a failed setupTestServer start shuts the server down. - UploadReservationStore expiry tests pin the clock instead of racing a 50 ms TTL.	2026-10-10 19:34:32 -04:00
Daniel Salazar	d4589caefe	fix: session store revokes (#4168)	2026-10-10 07:55:43 -07:00
Neal Shah	5fb025c109	Misc subdomain and worker fixes (#4157) ... * PUT 1902 * LocalWorker fetch policy fixes for 127.0.0.1	2026-10-09 19:59:52 -04:00
Juan Fernando Castro	50bcdc9c6c	fix: let a revocation be seen by the reads and the nodes that outlive it (#4151) ... * fix: let a revocation be seen by the reads and the nodes that outlive it Four places where a revoke lands on the primary and the next reader can still be told the old answer. The u2a cache key was dropped but not re-filled, so the next miss read a replica that may still carry the row and cached it for the TTL -- and a revoke's own settling pass is usually that miss. It re-fills from the primary, as the u2u side already does. A revoked session dropped its keys the same way. It now leaves the revoked row in the cache, read from the primary, so there is nothing for a lagging replica to re-introduce. A link share is its own authority rather than an index over a permission, so a replica still carrying a withdrawn row answers with it. For a short window after one is withdrawn, those reads go to the primary. Socket eviction rode a local event, so it only reached the node that did the revoking. It is announced on the fan-out channel too, which is where the node actually holding the connection hears it. * refactor: carry the revocation on outer.* instead of outer.pubsub.* Eviction runs through `io.in(room).disconnectSockets()`, which goes via the socket.io Redis adapter a cluster shares, so one node handling the event disconnects that user on every node in it. Fanning out to siblings had each of them repeat the same cluster-wide disconnect. The adapter's Redis is per-cluster, so peer clusters still need the hop -- `outer.*` reaches them and skips the sibling fan-out. The local `auth.sessions.revoked` goes with it: `outer.*` fires on the emitting node too, and nothing else subscribed.	2026-10-09 16:36:26 -04:00
Juan Fernando Castro	ff35fbd66e	Share attribution and cascade: finish PUT-1856 and PUT-1857 (#4119) ... Replayed onto main after #4118 squash-merged. Same tree as the reviewed head (8d8bd871f); only the redundant merges from the old base are gone.	2026-10-09 16:23:18 -04:00
Juan Fernando Castro	1ff5d93244	fix: a signed overwrite lands on the live key only once its completion is accepted (#4136) ... * fix: a signed overwrite lands on the live key only once its completion is accepted A presigned overwrite URL was signed against the live object key, so the browser's PUT replaced the owner's bytes directly. That PUT never passes through Puter, which leaves the access check on `completeWrite` with nothing left to protect: by the time it runs, what /read serves has already changed. The upload now goes to a key of its own. `completeWrite` is already the authorized step, so that is where the staged object is copied onto the live key, server-side, and the staged one dropped. The entry keeps its uuid, so everything written against it still holds. Aborting deletes the staged object, which the old shared key made unsafe to do. `copyObjectOfAnySize` carries the copy past the single-request ceiling by copying part by part, so a large file is no different. Two behaviours follow from the key no longer being shared: completing an overwrite that uploaded nothing is refused rather than reported as an overwrite of unchanged bytes, and abort always drops the session's object. Only the two presigned paths changed; `prepareBatchWrites` uploads server-side within one authorized request and still writes in place. * fix: the review's findings on the staged overwrite The batch signed path still signed against the live key, so the route the SDK's signed upload actually takes was unchanged. It stages now, which also makes the abort simplification safe: the guard that had protected a shared key was removed while that path still shared one, so aborting a batch overwrite would have deleted the owner's object. The completion is allowed against a path but lands on the uuid captured at start, and a move between the two puts the bytes somewhere the caller was never authorized for. The promote refuses unless the entry is still where the check looked. The batch promote failing left every session pending with its lease held; it now fails them as the rest of the function does. The pending GUI tile and the thumbnail key were keyed on the staged uuid, which no entry ever carries, so neither correlated with what the completion emitted. Responses carry `entryUid` for that. * fix: do not treat a pre-staging session's key as this upload's own A session signed by the previous release points straight at the live key, since `objectKey` was `existingEntry?.uuid ?? uuidv4()`. Completing one is already handled -- the promote returns early when the two keys match -- but abort deleted `objectKey` unconditionally, which during a rolling deploy would take the owner's file rather than the upload's litter. Abort now deletes only a key the upload owns. The multipart branch was always safe: discarding parts never touches an object already in place.	2026-10-09 16:22:36 -04:00
Juan Fernando Castro	86cc3490f7	fix: settle which plan an org seat resolves to, and confirm a seat's address (#4139) ... * fix: settle which default plan an org seat resolves to, and what counts as paid Default subscription resolvers ran in registration order, which is boot order: whichever layer registered first answered first. A seat is provisioned by a team and its address is never confirmed, so the seat resolver now states a priority and wins over anything a deployment infers from an address. The seat cap asked whether the owner's plan was outside the free set, which a default plan can be without anyone having bought it. Metering now reports whether an explicit subscription answered, and the cap rests on that. * fix: the review's findings on plan resolution `fromDefault` was read off which resolver answered, not off which policy applied. An explicit id naming a policy nobody registered falls through to the default one, and that read as bought. It follows the applied policy now. A deployment with unlimited metering has no explicit resolver, so every owner on it became unpaid and its teams dropped from the paid seat cap to the free one. Unlimited answers for what a paid plan would, so it counts as one. Also: the new constants had been placed between `FREE_SUBSCRIPTION_IDS` and its docblock, the register docstring still described first-wins, and prettier had reflowed an untouched licence header. * feat: a seat confirms the address its team typed in An address on a seat is the team's claim about someone else's inbox, and nothing asked the inbox. It is now demanded and sent a code, so the address counts for nothing until the seat hands it back. Resolvers that read a domain are the reason this matters. No address is still fine and demands nothing: the team is the trust anchor for the account, not for a mailbox it does not hold. `/confirm-email` allows an unconfirmed actor, so a seat can sign in with its temporary credential and clear the demand; nothing locks it out. * fix: clear the confirmation demand with the address it names `clearUnconfirmedEmail` releases a seat's address so the inbox's owner can sign up with it. The demand to confirm that address has to go at the same write: the gate reads `requires_email_confirmation && !email_confirmed`, and once the address is gone there is nothing left to send a code to. * fix: keep a seat's address correctable, and only ask what can be answered Review of the confirmation demand turned up two ways a seat could be left holding one it had no way to meet: - With no mail transport configured, the code is never sent and a resend cannot reach the seat either, so the demand stood with nothing able to lift it. The seat is now gated only when a code can actually travel. - An address the owner mistyped gated the seat on an inbox nobody could read, and the gate itself closes `/user-protected/change-email`, the route the seat would have fixed it by. `updateSeatEmail` lets the owner who typed the address retarget it, and refuses once it is confirmed, when it belongs to whoever proved it. Also resolves the owner's plan sequentially, so the second read takes the cache the first wrote and the pair cannot come from two resolutions, and restores the licence header a jsdoc reflow had rewrapped. * fix: let the confirm-email window open before the profile has loaded Booted from a stored token, the window renders before `window.user` is populated, and reading `.email` off null threw inside the window itself -- so the one screen that could have asked for the code never drew, leaving a blank page. Reproducible without a team: sign up, then load the GUI again before confirming. The address is now encoded rather than interpolated raw, and both forms of the line are i18n keys; it was hardcoded English before.	2026-10-09 16:17:51 -04:00
Juan Fernando Castro	7de8360a6e	Sharing cleanup: bounded outbound listings, and what see does not reach (#4118) ... * fix: keep a file name to one line in a subject, and the inbound total to the account **A name reaches the digest subject as the user typed it.** Subjects compile with `noEscape` on purpose -- a header is not HTML, and escaping would show entities to the reader -- and the comment there says free-form values collapse whitespace upstream. A file name does not: `#assertUsableName` refuses only an empty name, a slash, `.` and `..`, so a name may carry newlines. It is collapsed to one line and bounded before it goes in. **`includeTotal` answered an app with the account's figure.** The page is filtered by what the caller can reach -- an app sees only what its user handed it -- while `countByHolder` counts every inbound share. The docs already call `total` an approximation for the filtering; they did not cover an app reading the size of everything else. It is answered only for a caller acting as the account now, and the page documents that. The team-share fixture built its actor as `{ user }` with no `effectiveApp`, which reads as an app rather than a session. A session acts as the account, so it says so. * fix: stop `see` reaching inside a folder, and bound mode changes **A `see`-only folder recipient was pushed every child's name, size and type.** The fan-out picked holders by reach -- any live grant on a reaching root -- and `see` is a grant. But `see` says a node is there; `list` is what says what is inside it, and the payload carries exactly what `list` would: name, path, `is_dir`, type, size, modified. Descendants now go only to a holder whose mode reaches inside. A change to the shared node itself still goes to everyone who holds it, which is what `see` covers. **Mode changes skipped the daily quota entirely.** Not spending share budget is right -- changing a mode is not new reach, and the existing test says so -- but unbounded it lets one account write a pair as fast as it can ask. They have their own budget now, ten times the share limit by default and configurable, so the exemption stays an exemption from *that* budget rather than from any. * fix: let a half-finished downgrade be finished, and stop Trash answering **A mode downgrade that failed between its two halves could not be repaired.** `setUserUser` grants the new mode and then revokes the superseded ones. If the revokes did not all land, the higher grant stood -- and the retry returned early, because the mode it was asked for was now present. "Nothing to do" now means this mode is the only one standing, so asking again is what finishes it. The same two halves, and the same early return, were in the group variant. **Trashing someone else's item answered for what is in their Trash.** Moving into the owner's Trash is deliberately authorized without write on it -- that is what lets a recipient delete from a shared folder -- but a name collision there came back as `An entry already exists at /owner/Trash/<name>`. The mover picks the name, so that is a question they can ask about a folder they cannot list. It dedupes instead, which is also what the GUI wants: trashing should not fail over a name. * perf: record the entry's owner on the share row The outbound listings answer "shares on things I own that someone else issued" by joining `share` to `fsentries` and filtering on the owner there. Neither side of that join bounds the work: driven from `fsentries` it walks every file the user owns, driven from `share` it walks every share until it has a page. For a user who owns a lot and delegates nothing, that is the whole table for an empty result. `#outboundAppsSql` is worse than the ticket says -- it has no LIMIT at all, so every call aggregates the join across both tables. The owner now rides on the row, indexed with `id`, so all three read it directly. An entry cannot change owner -- a move out of the owner's tree is refused -- so it is written once, with the row. Set in SQL from `fsentry_id` rather than threaded through callers: four insert sites, each of which already has the id, and none of which can now forget. The risk in denormalising is a write path that does: a row with no owner recorded is a share that silently stops appearing in its owner's listing. The new test covers exactly that -- it fails when one insert is made to write NULL. Verified against a real boot: a pre-existing share row, on a database that genuinely predates the column, comes out with the owner its entry has. * fix: the review's findings on this branch **The Trash collision still had both halves open.** `overwrite` was checked before the trash case, so a caller setting it -- it is wire-supplied -- could still delete whatever it collided with in the owner's Trash, and the 409 survived for anyone who did not. The trash case is decided first now. Dedupe alone did not close it either: a name that changes only when there is a collision answers the same question the refusal did, and the mover sees that name even though the path is masked. Trashing someone else's item renames it every time, so the result carries no signal. `original_name` already records the real name, which is what a restore reads. Two tests pinned the old name; they say what it is now. **The owner column had no fallback.** Reads went only through it, so a row written by anything that predates it -- a pod mid-roll -- would never appear in its owner's listing again. The three reads now also cover `IS NULL` rows through the old join, indexed on the null and empty once none remain. **The grant became conditional on a replica read.** `if (!already)` skipped it on a stale row while the revokes below stayed unconditional, which could leave the holder with nothing where the old early return left their access alone. It grants unconditionally again; only the early return changed. **The re-issue budget covered one path of four.** Team, link and invite re-issues reserved nothing, so three ways round it were left open. They share one `#reserveRemodeQuota` now. Two docblocks had been pushed off the functions they describe, and two comments still described the join that is gone. * chore: one-line the comments added here * test: measure the prefix walk by what it keys on, not by the clock `checks many long paths without re-reading every prefix of each` gave the walk 1500ms of wall clock. It measured nothing useful: the shape it guards against — keying each step on the accumulated prefix text rather than on the parent id — costs 29ms against the current 8ms on that input, so both ends sit far under the bound. Under coverage the same test ran 1511ms and failed, which is the only thing the number could ever report on. `PathPrefixIds` now counts the characters it keys on, and the walk is asserted on that: proportional to the input, and twice the depth costs twice as much rather than four times. Keying on the accumulated text fails both. The original case keeps its real assertion, that a request this size is refused. * fix: keep the recorded entry owner current, and the trash guard on the retry A move into someone else's tree re-owns the whole subtree, which the migration comment claimed could not happen. The shares on it are re-pointed at the new owner, or they drop out of the listing the column keys and stay in the old owner's. The unique-violation retry re-entered the collision resolver without the trash flag, which is the one call site the guard was not threaded through.	2026-10-09 16:14:23 -04:00
Daniel Salazar	8a6929847d	Auth limits, reborn-app tokens, seat addresses, drop-area encode (#4148) ... * auth: per-account 2FA code limit, single-use otp-login tokens, recovery mail limits - /login/otp and /login/recovery-code share a per-account bucket charged on wrong codes; an otp-login token completes one sign-in. - /send-pass-recovery-email adds an IP bucket, a per-account cap and captcha, and reuses the outstanding recovery token instead of rotating it. - Reauth tokens carry the rejection reason; /signup re-attaches a temp account only from an expired session. * auth: app-issued access tokens older than their app stop authenticating An access token whose row or parent session predates the app now holding its uid is refused, the same check app and worker sessions already get. * teams: signup can take an address a seat never confirmed Password signup, save-account and a provider-verified OIDC sign-up take an address a team seat holds unconfirmed; the seat's copy is cleared at the write. A seat address that was confirmed still blocks. * dev-center: encode dropped directory child names	2026-10-09 00:12:59 -07:00
Daniel Salazar	360f6b1079	fix(events): bill reached deliveries, bound dispatch reads, seal cursors (PUT-1900) (#4146) ... * fix(events): bill reached deliveries only, bound dispatch reads, seal cursors - A broadcast socket copy is billed only when a connection for its holder exists on this node, anywhere in this region (connection count) or, for a durable row, in a region presence names. Without peers it bills as before. - Compiled match filters are held in a bounded LRU with a TTL, and dropped when another process reports a subscription ended. - GET /events/fetch reads the continuation from `cursor`, falling back to `after`. - Cursors from /events/fetch, /events/subscriptions and /events/kv-handles are encrypted with sealCursor/openCursor; plain cursors still read. - A durable subscribe counts again after its insert and removes its own row when over the cap; the generation bump for that removal is published. - Subscribe checks access before reading the anchor's owner, and an anchor gone in between answers with the subject the caller sent. - Dispatch reads at most the rows per token routing can use, scanning large token hashes; a removal still settles every row on the removed anchor. - puter.events.unsubscribe() stops routing after the server ends the subscription or reports it gone; fetch() accepts `cursor`. * fix(events): forward kv values only to regions with rows that ask - A region counts its session rows asking for kv values per token and announces the token's value variant (`v#<token>`) over the existing `watch` item on the first such row, withdrawing it with the last. Add, remove, reap, reanchor and refresh all keep it, reading `includeValue` off the row as it is dropped. - A kv write reads the value variants alongside the watch index and attaches the value only to forwards bound for a region that announced one. * fix(pagination): seal id-keyed cursors on every list endpoint Apps, subdomains, workers, team (members, directory, teams, audit, own audit), shares (inbound, outbound), the user-to-user permission audit and readdir/descendant listings now hand out cursors sealed with sealCursor and read them with openCursor, so a cursor no longer carries a row id. Plain cursors issued before still read. * test(events): pass socket ids to noteConnect in the billing tests	2026-10-09 00:08:13 -07:00
Daniel Salazar	dcfb1263de	fix(events): single-writer presence items, live-socket counts and region-wide leave grace (PUT-1759) (#4142) ... * fix(events): keep each region's presence item written only by that region - A region that receives a forward for a pair it holds no socket for retires its own presence item through the pin-ordered leave path, gated by a region-shared claim. The sender no longer retires another region's item, and the forward reply no longer carries noSocket. - Presence reads skip items for regions outside the peer list instead of retiring them. - removeConnection decrements and deletes the count in one script. - A failed join releases the join pin, and any connect that finds the pin free writes the join. - A single delivery treats a socket on any node of the region as local, using the region-wide connection count, and picks the next remote candidate by the regions already tried instead of by index. - Correct the connection-count TTL comment. * fix(events): count presence by live socket and honor the leave window region-wide - A region's presence for a pair is a sorted set of socket ids scored by last renewal under a new key, replacing the integer count. A socket counts while renewed within the concurrency-slot window; add, renew and remove each prune lapsed members and report the live count in one script, so a node that dies stops holding the pair. - The last socket going marks the pair as leaving region-wide for the grace window, in the same script; a connect clears it. A forward that lands on another node inside the window does not retire the item. - Add the events.presence.write counter, labelled by op (join, leave, refresh, retire).	2026-10-08 23:55:55 -07:00
Juan Fernando Castro	d1d55a6bc3	SDK auth plumbing: cache scope, dedupe keys and the token listener (#4112) ... * fix: pin the token sender, scope the grant flag, and stop two stragglers Five of the SDK auth-plumbing items. **The global `puter.token` handler pinned origin but not source.** Its three siblings all pin both, and each says why: the GUI origin is shared by every frame on that domain, so origin alone identifies a domain, not a sender. `signIn` pins its popup and binds `msg_id`, the consent dialog pins its popup, and the reauth wait pins the embedder. This one pinned the embedder only when `env === 'app'`. Since `authenticateWithPuter` now hands off to `signIn`, which adopts the token itself, the embedder is the only sender this handler is left serving, so it pins in every environment. **`is_grant_user_app_permission` was a request-wide flag held across an await.** It tells the `app-root-dir` rewriter that a row is being written, and while it is set that rewriter resolves to a real `fs:` permission instead of `PERMISSION_FOR_NOTHING_IN_PARTICULAR`. Anything evaluating an `app-root-dir:` permission beside the grant -- the grant route runs one under `Promise.all` with fs work -- read the same flag. It now lives in a scope derived for that call, so what the write needs to say about itself is not also said to its neighbours. `runInDerivedContext` is new: `runWithContext` starts an empty store, which would drop the `actor` the rewriter reads. **`fs.stat` and `fs.readdir` deduped on parameters alone**, so two identities in one realm could share a result. They now key on origin and token as `os/user.js` does, which documents the reason. **`signIn`'s relay poll never stopped.** Closing the popup settles the promise; the loop kept asking `/login/wait` every second for the life of the page. **`triggerReauth`'s five-minute timeout was never cleared**, so a successful reauth left a timer to fire into a settled promise. The concurrency test parks the rewrite and reads the flag while it is held -- my first attempt read before the flag was even set and passed either way. * fix: scope the fs cache per identity, and retarget two paths that never ran The remaining SDK auth-plumbing items. **`puter._cache` outlived the identity that filled it.** `_clearAuthToken` dropped the token and the stored session but left the cache and `whoami` as they were, and its keys were raw paths -- `item:/alice/Documents` and a `~`-relative key alike, identical whoever asked. A GUI user switch without a reload could be served the previous user's stat or readdir. Keys now carry the origin and token they were read as, built in one place so the writers, the readers, the invalidations and the warm-up probes cannot drift apart, and signing out drops the cache and `whoami` with the token. **The driver permission prompt could not fire.** Both paths waited for a 200 carrying `{success:false, error:{code:'permission_denied'}}`. The driver route answers `success: true` or throws, so a refusal arrives as a 403 -- the shape the retry now reads. Its two tests synthesised the old envelope, so they passed against a response the API does not send; they use the real one now, and the prompt goes from never called to called once. This is a behaviour change: a driver call refused for permissions will now prompt, which is what the path was written to do. Happy to delete it instead if the prompt is not wanted. The half-written copy of the same idea in `utils.js` is gone -- it prompted for one hard-coded interface and its retry was a `// todo`. **The verification-gate single-flight was keyed on nothing**, so a phone-gated request in flight beside an email-gated one took the email answer, replayed, failed again and surfaced raw without ever asking. Keyed per gate: same gate still shares one dialog, a different gate gets its own. * test: ask the cache the same way the code does The API suites built `item:<path>` keys by hand, so they stopped finding anything once the real keys carried the identity they were read as. They go through `fsCacheKey` now, which is the point: a test that spells the key itself passes until the moment the spelling changes, and then fails for a reason that has nothing to do with the behaviour it is guarding. These run against a live server, so a plain `vitest run` does not reach them -- they were green locally and red in CI. * fix: the review's findings on this branch Eight, and three of them were regressions this branch introduced. **Pinning the token sender to `globalThis.parent` in every environment broke the case it was meant to protect.** On a top-level page `parent` is the page itself, and the token arrives from a popup, so the handler stopped accepting anything. The GUI posts `puter.token` to `window.opener` for the file pickers too, not only for sign-in, and this handler is their only consumer -- a site calling `showOpenFilePicker()` would have signed the user in, dropped the token and run the following read with none. It now pins the embedder when framed and a window this SDK opened when not; `UI.js` already tracked its picker popups for the same reason, so that set is shared rather than duplicated. **The GUI still spelled a cache key by hand.** `updateSubdomainsForItems` built `item:<path>`, which no longer matches what the SDK writes, so its `get` could not hit and its `set` wrote an orphan nothing reads and nothing evicts -- a published website's badge quietly stopped appearing on a cached render. It goes through `fsCacheKey` now, and that is the last hand-spelled key in the tree. **Retargeting the driver permission prompt to 403 was wrong and is reverted to the deletion the ticket asked for.** The route refuses with `forbidden`, not `permission_denied`, so the retarget still missed it; meanwhile a driver that raises `permission_denied` for its own reasons would have opened a prompt that had nothing to do with the refusal. And the prompt asks for `driver:<iface>:<method>` while the route checks `service:<driver>:ii:<iface>`, so granting it changes nothing. The path cannot work as written. The rest were pre-existing or latent: `cacheWhoami_` wrote its answer without checking the token it was issued under was still current, so a reauth landing mid-flight could repopulate `whoami` with the previous identity. `_dropIdentityCaches` also left `whoamiCache_`, which other call sites await. `setAuthToken` now drops the identity caches when the token actually changes: keys carry the token and the store is persisted with no TTL, so a rotation that did not go through `_clearAuthToken` orphaned a whole key space with nothing to reclaim it. The verification-gate entry was registered after the body could already have deleted it -- a synchronous throw stranded a resolved "not verified" in the map for the life of the page. It registers first and clears on a microtask. `signIn`'s poll is bounded as well as settle-checked: on a cross-origin isolated page opened from a gesture, nothing watches the popup, so closing it never settled the promise and the loop ran on exactly as before. `runInDerivedContext` says in its docblock that callee writes do not survive; only the one flag needs the scope, and a future caller should not expect to hand a value back through it. * fix: the review's findings on the SDK auth plumbing **The relay bound stopped asking without saying so.** The deadline left the `signIn` promise pending and its listener attached. On an isolated page opened from a gesture the relay is the only thing that can settle it -- nothing watches the popup there -- so a user who took longer than the deadline was left with a promise that never resolved, and `authenticateWithPuter` queues every later implicit-auth call behind it. It rejects now. **Pinning the token source left `signIn`'s own popup untracked.** Only the picker popups registered, so on a top-level page the global handler bailed on a sign-in it had opened itself: `puter.onAuth` never fired, and `puterAuthState.authGranted` was never set. `openAuthPopup` is the one place auth popups are opened, so they register there. **A closed popup is ours too.** Rejecting one that had already closed loses a message from a popup that posts and then closes itself, which is why `UI.js` tolerates the same case. Kept, with the set bounded instead. **The gate key carried the factors**, so one route naming them and another not opened two dialogs for the same gate. It keys on the gate, as `ctx.done` does. **`fsCacheKey` left `~` unresolved**, so a writer and an invalidation spelling the same place differently missed each other -- the drift the one key builder was meant to end. The dead `opts.permission` went with the path that used to read it. Four workerd-runner failures are left: `ai > chat stream adapter supports cancellation`, `kv > a GUI boot key written by another client reads fresh` and two cache-warmer ones. All four fail the same way on `origin/main`, and the node and browser runners pass. * chore: one-line the comments added here * chore: reflow the comments the removed retry cause left ragged	2026-10-08 15:04:49 -04:00
Daniel Salazar	d395a3aa94	fix(kv): page through flush, cap per-call batch sizes, validate decr and del (#4106) ... - flush follows LastEvaluatedKey until the namespace is empty and rejects when a batch delete fails; cached reads for every key a page returned are still invalidated and the flushed event still fires. - decr validates the amount map before negating it, so it accepts exactly what incr accepts. - del checks the key like every other store method (oversized key is a 400). - list offset emulation treats a missing continuation key as end of data instead of restarting from the first key. - Array get maps results through a Map instead of a linear find per key. - The driver gates includeTotal on credit like value listings. - New driver caps in drivers/kv/limits.ts: 1,000 keys per array get, 1,000 items per batchPut (both rejected past the cap), list limit lowered to 1,000. Published in rate-limits-and-quotas.md and the set/list pages.	2026-10-07 12:00:50 -07:00
Juan Castro	ad1be70d06	Merge branch 'juancastro/flat-permission-view-scope' into juancastro/permissions-access-control-cleanup	2026-10-06 19:44:37 -04:00
Juan Castro	6277f106c4	Merge branch 'main' into juancastro/flat-permission-view-scope	2026-10-06 19:44:19 -04:00
Juan Castro	bd5aa901ab	perf: batch the per-page reads in a team delete ... Measured first, against a real server, counting DB round trips: a 40-seat delete was 284 queries, 7.1 per seat, scaling linearly. Four of those seven answered for the whole page and had no reason to be asked per account. 120x SELECT * FROM `user` WHERE `id` = ? 41x INSERT INTO `audit_team_membership` 40x SELECT COALESCE(SUM(size), 0) FROM fsentries 40x UPDATE `user` SET `suspended` = ? 40x SELECT `uuid` FROM `sessions` WHERE `user_id` = ? The page now reads who is already suspended, what each holds, and the rows the notices need in one query each, and writes the audit as one statement. What is genuinely per account -- the suspension, its billing event, the mail -- stays per account under the same concurrency bound. Same 40-seat delete is now 170 queries, 4.3 per seat. Three new store methods rather than raw SQL at the call site: `getSuspendedByIds` reads the primary, deliberately not `getByIds`, which is cache-first. A stale `suspended` is exactly what re-charges a seat on a retry, which is the thing this path was just fixed to stop doing. `getHeldBytesForUsers` is `getHeldBytes` with a GROUP BY. `appendAuditMany` is `appendAudit` as one multi-row insert, still written before any suspension, so a failed append cannot leave an unlogged one. What is left is concentrated in `#suspend`: the UPDATE, the forced re-read the user store does afterwards to refresh its cache, and the session drop. Those are shared store paths and batching them means owning their cache invalidation, so they are not in scope here.	2026-10-06 18:55:53 -04:00
Juan Castro	25f132b28d	fix: address code review on the teams cleanup ... Six findings, all real. `total` described rows the page withheld. The count ran over every live team the user belongs to, while the app-actor filter ran afterwards on the items only, so `GET /teams?includeTotal=true` from an app returned `items: []` with a non-zero `total` -- the exact thing the filter exists to prevent, and newly reachable because `includeTotal` was inert before this branch. The directory-open restriction moves into the query, so items, cursor and total all describe the same set. The GUI told owners the opposite of what the server now does: "we email the username and temporary password to it." An owner who believed it would dismiss the one copy of the password and strand the seat. Rewritten. `runWithConcurrencyLimit` rejects on the first error but cannot cancel the workers beside it, so a failed seat let up to seven others finish suspending and emailing "your team has been closed" for a team that was still live. The page now settles before the rejection is rethrown, which leaves a state the retry can resume from instead of an arbitrary one. My earlier claim that failure semantics were unchanged was wrong. `#disableSeatForDelete` returned `true` on both paths, so the boolean read as if it distinguished them. It returns nothing; the counter uses the settled results. `listTeamsForUser` had picked up a second stacked docblock -- the same defect this branch removes from `#cached` one file over. `reissueCredential` sent the provisioning mail, which after the credential change was a byte-identical duplicate saying nothing about the password it had just invalidated, while the GUI promises the member hears about it. It has its own template now. Regression tests for the count leak and the reissue notice; both confirmed to fail without their fix.	2026-10-06 16:22:48 -04:00
Juan Castro	665314535e	docs: say what TeamStore actually caches ... The cache docblock claimed `isMember` and `getByUid` were left uncached "for that reason". Both go through it -- `getByUid` directly, `isMember` through `getMembership` -- so the comment told a reader the opposite of the code, on the two reads that decide authorization. What makes it safe is the busting, not an exemption: every mutation here drops the keys it touches, and the TTL is the backstop for the one path that does not come through this store, the ON DELETE CASCADE on the membership row. No stale window is constructible today; the comment now says why. Also drops a one-line docblock stranded above it, describing a method that is no longer in the file.	2026-10-06 15:55:40 -04:00
Juan Castro	f3b05ca822	fix: page GET /teams like every other team listing ... `listTeamsForUser` returned every row and the controller never read `req.query`, so `limit`, `cursor` and `includeTotal` were accepted and ignored on the one team route that did not paginate. The docs promise a capped page there, and every sibling listing already honours it. Keyset-paginated on `id` per doc/pagination.md, with the envelope the other routes return. The app-actor directory filter still runs over the page, so an app sees the same teams it saw before, just a page at a time.	2026-10-06 15:55:22 -04:00
Juan Castro	ac5fa20388	fix: keep the other account out of the OIDC link conflict ... Linking a provider identity already bound elsewhere answered 409 with the raw provider subject and the other account's internal id in the message. The caller needs to know there is a conflict, not whose it is; both stay in the log for whoever is debugging it. Status and legacy code are unchanged — OIDCService branches on the 409.	2026-10-06 11:14:19 -04:00
Juan Castro	ac740984a3	fix: keep the flat permission view to the row it stands for ... Two ways the flat KV view could say more than the SQL behind it. On a miss the warm flattened the whole traversal and wrote every leaf under the holder. Those leaves belong to the issuer: a holder granted one mode of a resource by someone who holds the resource itself had the issuer's wider permission written under their own key, and it answers later checks. The warm now writes the holder's own row and nothing else, which is what the grant path has always written. Deleting an account cascaded the grants it issued out of SQL, but the flat entries are keyed on the holder, so nothing removed them and they are trusted without an issuer check. The access outlived the account, and the owner could no longer see it to withdraw it. The refs are read before the row goes and cleared after it, through the same batched delete a revoke uses, so peer regions hear about it once. Both fail without the fix: the warm test sees the issuer's wider permission written under the holder, and the delete test reads the grant back from an account that no longer exists.	2026-10-05 17:51:31 -04:00
Daniel Salazar	ba081e2332	feat(ddb): log the table and key of throttled DynamoDB requests (#4078) ... A client middleware inside the SDK retry loop logs every throttled attempt (including ones that succeed on retry) with its operation, table, key, client region and DynamoDB's throttling reason, plus batch items returned unprocessed. Each target logs at most once a minute with a count of the throttles folded into it. Puts log only the key attributes, using key schemas registered by the KV store at startup.	2026-10-05 02:01:38 -07:00
Juan Fernando CastroandDaniel Salazar	a46eb15d5e	Let the account's own API token reach the routes a privileged app needs (#4032) ... * feat: let the account's own API token reach the routes it should A privileged app is launched with the GUI session token today. Running one on a full-access personal access token instead costs much less when it leaks, but several routes those apps depend on refuse every access token, so the swap would break them. Admitted, each a read or a write the token's own HTTP reach already covers: - the realtime socket handshake, which refused access tokens outright; FS live updates and notifications stop without it - GET /get-dev-profile and POST /profile, both puter.js surface - GET /auth/list-permissions — a read of what was granted, not a grant - GET /share/shared-by-me/apps and GET /share/audit, over grants /share/shared-by-me already lists for the same credential - POST /rao, which puter.js calls on every setAuthToken Gated on `isAccountContext`, not the `full_access` claim alone, so a token with an app anywhere in its chain is still refused and a socket for one never joins the user room. Left session-only: mint, session, 2FA, grant, team and billing, plus two the list named. /share/blocks is a personal safety control whose only caller is the desktop's Blocked Senders window. /app-feedback is reachable only from our own GUI pages, and being unsubmittable programmatically on a user's behalf is the property it was built with. The new HTTP suite drives a real server with a minted token and fails on all six admissions without the change; the block routes' existing metadata check already pins their refusal. * test: sort the two token shapes at the events handshake * fix: carry the account-context invariant into the HTTP gate `requireUserActorGate` admitted on the raw `full_access` claim while the socket and /rao use `isAccountContext`, so one shape — a token carrying the claim with an app in its chain, or an actor built without `makeActor` — would have been refused in two places and admitted in the third. Unreachable today, since the mint and the auth-time read both drop the claim when `app_uid` is present, but it is the invariant this change states and it should hold at every enforcement point. The payout address goes back behind a plain session: `/get-dev-profile` is open to the account's own token now, and the rest of what it answers is a developer's own name and incentive-program status. Also drops a cross-reference in EventsService to the socket posture this change relaxed, and rewords the block-routes comment, which read as if access tokens were admitted there rather than refused. * fix: drop a revoked token's sockets, and read the dev-profile columns Two gaps the review found, both reachable only because this branch lets the account's own token hold a socket and call /get-dev-profile. `#revokeAccessTokenTail` soft-revoked the session row and told no one, so a revoked token kept its connection — and the account's `outer.gui.*` fan with it — until the five-minute reauth sweep. Sockets now join a per-token room and a new `auth.access-token.revoked` event drops exactly that room, so the account's other tabs stay up. Session revoke keeps its account-wide eviction. `/get-dev-profile` read `first_name`, `last_name`, `paypal` and the two incentive flags off the user row, which carries none of them: the columns have always been `dev_`-prefixed, so the endpoint answered nulls to everyone and `puter.apps.getDeveloperProfile()` has never returned anything. Reads the real columns, keeping the unprefixed name as a fallback for a deployment carrying both. The payout address stays behind a plain session, so opening the route to the account's token hands over a name and incentive status and nothing else. Both fail without the fix: the socket test waits five seconds on a connection that should already be gone, and the profile test reads back a seeded row. * fix: close /open_item to tokens, and stop the node runner leaking sockets `/open_item` carried `allowFullAccessToken`, but it grants an app a permission on the user's behalf and mints the app token to go with it — a grant and a mint, which stay session-only. Pre-existing, and missed by the first sweep here, which only asked whether routes lacking the flag should gain it and never re-read the ones that already had it. Refused at the top of the handler too, so a credential that gets past the gate cannot leave an ACL row behind. The node runner loads each SDK into a vm context and never closed it, so every test leaked a socket, and every socket holds a per-(user, origin) connection slot. Admitting the account's own token to the handshake pushed that past the cap, and the whole events block failed with `events_connection_failed` while the browser and workerd runners — which do not leak — passed. Closing the FS socket and the events channel after each test gives the slots back. * fix: announce only the revokes that can have a socket to drop Every access-token revoke broadcast an eviction cluster-wide, including the read-URL tokens `revokeReadUrl` retires constantly. A scoped token is refused at the handshake, so none of those broadcasts could ever reach a connection. The announcement now goes out only for a token the handshake would have admitted: full access, no app in the chain. `revokeOwnAccessToken` refuses full-access tokens outright, so that whole path is silent; a revoke by raw uuid says nothing about the token and still announces, which costs a no-op broadcast rather than leaving a revoked socket up. Fails without it: the test revokes a scoped token and a full-access one through the same entry point and counts the announcements. * test: pin the mint route against the account's own token `/auth/create-access-token` carries only `requireAuth`, so it never showed up in the sweep over `requireUserActor` routes — the refusal lives in `AuthService.createAccessToken`, which turns away any access-token actor outright. That holds today, and a leaked full-access token minting itself a sibling that survives revoking the original is exactly what the token swap is meant to rule out, so it is worth a test rather than a reading. * fix: let the recommended-apps test follow the list it tests Two commits on main rewrote `RECOMMENDED_APP_NAMES` without touching the test, which seeds `editor` and `camera` and asserts the result contains them. Neither is on the list any more, so the resolved set came back empty and `test (base)` has failed on every PR since. The test now seeds the first two names off the list itself and asserts on those, so the next edit to it cannot strand the test: prepending two unknown apps keeps all four cases green. * feat: let an extension waive a plan gate for one route or driver method A plan gate about to refuse a caller now emits `subscription.gate.<surface>` through `emitAndWait` first: `route.<method>.<path>` for a route, `driver.<iface>.<method>` for a driver method, `share.anyone` for link shares. The payload carries the actor, the request and the requirement, and a listener that sets `allow` waives the plan for that call. Callers the plan already covers never reach the hook, and a listener that throws leaves the refusal in place. * feat: mint godmode apps their own full-access token `/auth/get-user-app-token` answers a desktop launch of a godmode app (the `godmode` column) with a full-access token instead of an app token. Origin lookups, the sign-in popups for pages outside the desktop, keep getting ordinary app tokens, and only a web session may mint one. The token carries the app in `godmode_app_uid`, read into `accessToken.godmodeApp` for attribution only: `effectiveApp` stays null, so it keeps the account's reach and passes the same gates as a personal access token, including the vendor-compatible AI routes. It is refused once the app is gone or no longer godmode. Its row is an `access_token` session parented to the web session that asked, with `app_uid` set, so signing out, revoking that session or revoking all sessions takes it along, and the sessions list shows it under the app. Each mint re-signs onto the same row with a 12h `exp` and moves the row's expiry out, so the desktop keeps an open app alive while a copy taken earlier still expires on schedule. A full-access token may now mint scoped tokens (read URLs), parented to its own row; it still may not mint full access. Revoking a full-access token cascades to the tokens it minted. An access token whose session row has a parent now authenticates only while that parent is live, so a read URL a godmode app minted lapses with the app's token, and through it with the desktop session. * feat: launch godmode apps on their own token and renew it The desktop asks for a godmode app's token at launch and blocks the launch if it can't get one, as it does for other apps; a godmode app never runs on the desktop's session token. While the window is open the desktop re-mints the token once less than 6h of its 12h remain, checking on a timer and when the tab becomes visible, and answers the app's `reauth_required` with a fresh one, posted to the app's origin. puter.js keeps a godmode token in sessionStorage like a session token, never in localStorage. In app mode a 401 on a godmode token, background requests included, asks the desktop for a new token and replays, and a `puter.token` message is only taken from the embedding frame. The sessions list shows a godmode token row with its app's title and icon. * fix: keep a godmode token the same across renewals Apps hand their launch token to code that never sees a replacement (the terminal's Drive mount, a CLI's API key), so a token re-signed every renewal broke those copies once the first one expired. The JWT now carries no `exp` and a renewal moves only its row's expiry, so every copy keeps working while the desktop keeps the app alive. What still bounds it: - only the desktop's web session can extend it; using the token never does, and signing out or revoking that session revokes it; - it lapses 12h after the desktop last asked, and a lapsed row is never revived: the next launch gets a new token; - however often it is renewed, a token is replaced after 7 days, and the old one is revoked on the spot along with the tokens it minted and its sockets. The desktop no longer re-posts a renewal that kept the same token, so the app's sockets aren't rebuilt every few hours; an app that reports its token stopped working is always answered. * fix: reserve the feedback and no-reply mailbox names again Signup checks names against `util/reservedUsernames`, but `fbl`, `noreply` and `no-reply` were added to a second copy of the list in AuthController that nothing reads any more, so they became claimable. Move them into the live list and drop the copy. --------- Co-authored-by: Daniel Salazar <daniel.salazar@puter.com>	2026-10-05 01:10:29 -07:00
Daniel Salazar	e7c89b0639	fix(apps): keep ids longer than an app uid out of the app-stats query (#4070) ... ClickHouse's app_opens.app_uid is FixedString(40) and fails the whole query on a longer value. Marketplace callers pass item ids into getAppsStats, so one long listing-request id turned the seller's listing-request list into a 500. Such ids can't have opens; they now resolve to zero without being queried.	2026-10-04 17:03:19 -07:00
Daniel Salazar	5e34258da5	fix: drop database_id from the subdomain reclaim; MySQL has no such column (#4067) ... reclaimByUuid (#4064) cleared `database_id`, which exists only in the SQLite and Postgres schemas. On MySQL the UPDATE fails with "Unknown column", so ensureSystemSite throws: the puter-profiles row stays registered to the admin, the site keeps 404ing, and every profile save returns 500 as it retries the bootstrap. puter-app-icons takes the same path if its row isn't already protected. System rows never carry a database binding, so there is nothing to clear.	2026-10-04 02:45:03 -07:00
Daniel Salazar	c0f4fe93b1	fix(auth): stop a deleted app's sessions authenticating as the next app with its uid (PUT-2049) (#4057)	2026-10-04 00:24:55 -07:00
Daniel Salazar	d57f638eda	fix: keep system sites writable and their names out of reach (#4064) ... Follow-up to #4056. Icons and profiles are now written as the system user, so on a storage-limited install they counted against its allowance (100 MB by default) and every write past that returned 413, including a user's profile save. Both writes now pass the unmetered allowance, as writes the system makes on users' behalf already may. ensureSystemSite reclaims a mismatched subdomain row in place instead of delete+create, so the name is never free for a user to register, the row keeps its uuid, and `protected` is never cleared; the previous holder's bindings are dropped. If the row vanished since the read it falls through to create. puter-app-icons is now protected like puter-profiles, both names are reserved from user registration, and they live in util/systemSite so the subdomain driver doesn't import the services. The icon service's shutdown wait is capped at 5 s so a stalled run can't hold up the metering flush.	2026-10-04 00:23:01 -07:00
Daniel Salazar	26abd4e4a5	fix(fs): hide the issuer's home from scoped-token root listings (PUT-2053) (#4047)	2026-10-04 00:22:43 -07:00
Daniel Salazar	37c1678897	fix(fs): keep a vacated username off-limits to other accounts for an hour (#4039) ... * fix(fs): keep a vacated username off-limits to other accounts for an hour Renaming a home rewrites descendant paths in the database, but their cached entries keep the old path until they expire. ACL grants a home by path prefix, so whoever takes the old name next must not be able to before those entries are gone. renameUserHome now records the vacated name, and the claim check every username-claim site already runs treats it as taken for anyone but the account that left it. * test: seed recommended apps that are still in the default list The default list no longer includes editor or camera, so the ordering test found neither.	2026-10-02 14:53:34 -07:00
Daniel Salazar	a4f1e665df	fix: return 400 when a streamed upload is shorter than its declared size (#4023) ... A streamed write sends the caller's declared size to the object store as the body length. A body that ends short of it was rejected as IncompleteBody and surfaced as a 500.	2026-10-02 10:05:13 -07:00
Daniel Salazar	e6bda950f0	perf: batch per-app origin, canonical and hosted-backing lookups in app listings (PUT-1892) (#4013)	2026-10-01 18:30:06 -07:00
Daniel Salazar	4f31388dad	fix: cap app lookups and fields, dedupe icon processing, rate-limit app landing pages (PUT-1891) (#4012)	2026-10-01 17:54:25 -07:00
Juan Castro	a67cdac380	fix: bound the inputs on the grant and token routes ... Three permission routes sized their work by the request body rather than by the route, so one call could cost far more than its rate-limit slot implies. - `extra` and `meta` were only type-checked. They ride every row a grant writes (up to 16) and are re-serialised into the per-(user, app) cache on each miss, so they are now capped at 4 KiB. - `/auth/create-access-token` took a `permissions` array of any length: one permission check and one sequential INSERT each. It now uses the same 16-per-request cap the grant routes already had, and runs every entry through the validator those routes use. - Withdrawing an app's cross-app data grants looks them up by permission text, and no index on `user_to_app_permissions` led with `permission`. Indexed per engine, mirroring what mysql_mig_22 / postgres_mig_11 / sqlite 0067 did for `user_to_user_permissions`. `grant-dev-app` validated none of its input — not even the type of `extra` — so it now runs the same validator as its user-app sibling. Two paths could also carry a permission wider than the `varchar(255)` column it lands in. `grantUserAppPermission` already rejected that after rewriting; `grantDevAppPermission` now does the same, and `createAccessToken` checks it before the session row a failing INSERT would otherwise orphan. Caps are published in rate-limits-and-quotas.md. Every in-tree caller sends one permission and a small `extra`, so none of them change behaviour.	2026-10-01 16:51:39 -04:00
Daniel Salazar	ed7b8f828a	fix: kv path errors and ttl hardening, events subscription end and race fixes (#4004)	2026-10-01 09:50:34 -07:00
Juan Castro	d1b11c793c	Merge branch 'main' into juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a ... One conflict, in workerInvoker.integration.test.ts: #3997 fixed the same gap-marker flake this branch did, with a different barrier. Took theirs — waiting on the `ev:qf` counter, which `#handlerFailed` writes strictly after `discard`, so the marker is in and the lease cleared by the time it reads 1. Their ordering also asserts the counter before flipping `answer` back to 200, which closes the redelivery-clears-failures hole this branch's version had removed the line to avoid. Converted the two `vi.waitFor` calls #3997 added to `waitUntil`, the helper this branch introduced for the rest of the file. Neither is wrong today, since the new block fakes no timers, but a `jump()` added near them later would hit exactly the drained-backoff trap the helper exists for.	2026-09-30 17:49:36 -04:00
Juan Castro	d6b7c8ff52	docs: drop the replica-lag testing runbook from the open-source tree ... It describes our local two-server MySQL replication setup, which is internal tooling rather than anything a self-hoster or contributor needs. It now lives in the heyputer repo alongside the other internal docs. The suite it documented stays: the header comment now states the env-var gate directly instead of pointing at a file this repo no longer carries.	2026-09-30 17:42:04 -04:00
Daniel Salazar	4448103ef0	fix: misc events and kv issues found during recipes (#3997) ... * fix: misc events and kv issues found during recipes * test: cover handler depth on narrowed actors and created notifications * test: fix stale legacy batch directory upload test and seat user-agent flake * test: wait for the refusal to be recorded before claiming its gap marker	2026-09-30 14:40:47 -07:00
Juan Castro	2e0bf26cac	Merge branch 'main' into juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a	2026-09-30 16:31:42 -04:00
Juan Castro	23c8f73bcb	test: verify the replica-lag fix against real MySQL replication ... The unit tests inject the driver error because sqlite has neither a read replica nor foreign-key enforcement, so neither half of this bug can occur locally — they prove the handling, not that the situation arises. This suite runs two MySQL servers in real replication and freezes the follower with STOP REPLICA, so the row really is gone on the primary and really is still served by the replica, and MySQL raises the foreign-key errors itself. Against the pre-fix code all four cases fail, including the unhandled "Cannot add or update a child row" the reported 500s were. Opt-in behind PUTER_TEST_REPLICA_LAG, matching the env-gated provider integration tests and their skipUnlessEnv helper, so CI and anyone without the containers skip it. It builds and drops its own database rather than touching local dev data.	2026-09-30 10:02:36 -04:00
Daniel Salazar	b9efac2363	fix: misc fixes from alert logs (#3995)	2026-09-30 00:51:27 -07:00
Juan Castro	abd27c0192	fix: replicate tombstones, and attribute foreign-key failures correctly ... Review of the first pass turned up two holes that made it ineffective. CacheReplicationService always deleted the keys a peer broadcast and never adopted a payload, so tombstones stopped at the region boundary: peers dropped the marker, kept caching deleted rows off their own replicas, and the multi-region case the fix was for stayed broken. Tombstones are now the one payload a peer adopts — they carry no row, only the fact of a delete. user_to_app_permissions keys both the app and the user, and the drivers don't all name the constraint, so a deleted *account* was being read as a deleted app: a live app got tombstoned, the request 404'd for the wrong reason, and the dead account was never retired, so the client retried and re-tombstoned indefinitely. The insert site now asks the primary which parent actually went away and answers 404 or 401 to match. Also: re-check the tombstone after a cache write lands, since a row cached under a live tombstone is never looked at again; extend the guard to #refreshCache, which update() could otherwise use to undo a delete; stop markDeleted tombstoning an address the deleted row never owned, which was blocking the account that did own it; let the primary rather than the tombstone decide in batched lookups; tombstone the id key when no cached row names the others; and correct the leaf-store comment in stores/index.ts now that SessionStore depends on UserStore.	2026-09-29 13:12:31 -04:00