13 Commits

Include renames

Author	SHA1	Message	Date
Daniel Salazar	9ff224e22f	perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259) ... Pipeline - CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s. - `req.ua` is parsed on first read instead of on every request. - JSON `rawBody` reuses the parsed buffer instead of copying it. - `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call. - SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under load is unchanged. Auth and permissions - Token-to-actor auth starts the user, session and app lookups together and checks them in the same order: fewer sequential Redis round trips per authed request. - get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select after its own insert and doesn't wait on the cache write for replica rows. - getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer hands out a session that was never written. - whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged. - OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight per URL, and the Apple key is parsed once. - gui-token and sync-cookie use `req.actor.user` instead of re-reading the user. - listSessions batches app lookups. App icon probes run together, and the original-icon probe skips the lineage fallback. - Default-granted permissions are answered from config before the scan cache, and exploded options are deduped: one less Redis read per /drivers/call. - ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node first, then the ancestors concurrently on a miss. Filesystem - Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about 1.3 s to about 40 ms per 100 new files. - MySQL single-file inserts return the row built from the written values, falling back to the primary read: one less primary round trip per new file. - Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row fallback. - storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries `userUuid`: quota checks and /df no longer serialize the listener reads. - readdir cache backfill is one pipeline instead of 3xN SETEX. - Profile reads skip the lineage fallback; the own profile is read alongside the user lookup. - Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently. - Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream upstream error. - mkdir looks up target and parent together. - Upload completion reuses the session from the access check (status is still re-read before completing), and multipart signing reads it once. - Moving a shared file to Trash skips two lookups that always returned the entry itself. Drivers and events - Subscription resolvers run concurrently and are folded in registration order. - SubdomainDriver and WorkerDriver overlap independent lookups. - The events "anyone subscribed?" check is one Redis pipeline when the region is warm. - Handler publish overlaps the count with the size check. - Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays cross-region invalidations to the others. Sites, share and AI - Hosted sites resolve the owner first, then the app and root together. Site config loads only on 404s. The hosted-app query selects only the columns it uses and moves into AppStore. - shareFlags asks about the plan only when an item is shared by link only. - Share listings run the count, teams and plan reads alongside their page query. - Claude attachment cleanup no longer blocks the response. - New util `awaitLater` for promises started early and awaited later.	2026-10-17 12:03:35 -07:00
Daniel Salazar	d8f9aff3dd	fix: follow-ups to #4161–#4203 (rate limits, broadcast, AI routing and billing, fs subtree walks) (#4205) ... * fix(rate-limit): count the newest KV rate rows first The KV backend's rate check, peek and concurrent acquire list the key's rows with a limit + 1 cap that the store applies before dropping expired rows, so unswept rows from past windows (oldest keys) could fill the fetch and the live count read as zero. They now list newest first. * fix(broadcast): split webhook flushes under a byte budget, per-peer backlogs Flushes go out in bodies of at most `broadcast.outboundMaxBodyBytes` (default 4 MiB) instead of one POST, and each peer keeps its own backlog of what it missed, capped by count and four bodies' worth of bytes, oldest dropped first, so one failing peer can no longer grow every peer's body past their 50 MB limit. * fix(ai-chat): classify dropped upstream connections as provider failures A connection reset or refusal (SDK APIConnectionError, Mistral's ConnectionError, an undici reset on the cause) carries no status, so a chain of them fell through to 500 internal_error. Attempts now record connectionFailed; a chain of only resets and 5xx answers 400 upstream_provider_unavailable, and a reset counts as an upstream signal in a mixed chain. * fix(ai-chat): keep SDK retries on Anthropic count_tokens The Claude client is built with SDK retries off so the driver can fall back, but count_tokens has no fallback route. It now passes sdkRetryOptions, like the OpenAI moderation call, so a transient 429 or reset is retried. * fix(ai-chat): mark a route unhealthy for 60 s after a transient failure With SDK retries off, one 5xx, 429, timeout or dropped connection kept a route marked for 10 minutes, sending that model's traffic to resellers. Those failures now mark the route for 60 s; bad credentials, an exhausted account and other route-level failures keep 10 minutes. * fix(metering): don't memoise spent addons inside the force-exact window The spent-addons memo could cache a pre-grant record read from a lagging replica right after a credits-changed announcement, so overage skipped the new credit and requireCredits kept answering 402 for up to 15 s. A spent record is now only cached once the actor's force-exact window has ended. * fix(share): send the digest entries a flush claimed when another take fails Digest entries are taken concurrently, and a single rejected take threw away every entry the other takes had already removed from KV. The flush now logs the failed takes and sends what it claimed; an entry whose take failed stays queued for a later flush. * fix(fs): announce an overwriting move's replaced entry only when it took its name /move and v2 /fs/move looked up the entry at the destination name before moving and always announced its removal. A move into someone else's Trash renames instead of replacing, so the owner's desktop hid a live entry and legacy /move reported it as `overwritten`. Both routes now pass the target on only when the moved entry kept the name it was looked up by. * fix(ai): name the vendor in upstream error fields from media providers upstreamFetch now takes the provider id plus a display label, and UpstreamResponseError carries the id. translateProviderError copies a string `provider` off the error into `fields`, and the BytePlus video and Cloudflare image errors set one, so upstream_auth_failed and upstream_credits_exhausted alarms say which vendor failed again. * fix(teams): delete a provisioned seat whose home folders could not be created provisionUser logs a failed default-folder creation and still returns the account, so provisionAccount went on to add the membership, issue a password, mail the seat and emit team.account.created for an account with no home. It now cascade-deletes the new user and returns 500 internal_error before any of that. * fix(ai-chat): load model catalogs without holding up server start onServerStart awaited every gateway catalog (up to its 10s fetch timeout) while HTTP was already listening, so drivers started after it had no providers yet and chat requests found an empty model map. The first build now runs in the background; complete(), models() and token counting wait on it instead of failing. * fix(ai-chat): serve direct models while gateway catalogs load at boot The first model-map build now publishes a fresh map as each provider's catalog lands, so static catalogs are routable at once. complete() and token counting wait for the full first build only when the model, or the provider asked for, isn't in the map yet. Refreshes still swap once. * fix(ai-chat): split gateway-billed cost across input and output A completion billed at the cost OpenRouter, Infron or Neuralwatt reports was quoted to `ai.prompt.cost-calculated` as all input and no output. The ledger still records one `billedUsage` line; the reported per-key costs now spread that bill over the usage keys at the catalog's relative rates, in whole µ¢ that sum to it. Neuralwatt energy units take no share. * fix(ai-image): hold Replicate credits for the whole prediction The output and input-surcharge holds took the default 10-minute TTL, the same as the prediction window alone, so a slow prediction could outlive them once creation and input measurement were added. They now last 15 minutes, like the video job holds. * feat(ai-chat): report what a completion was charged as usage.cost Chat usage (the non-stream result and the stream's final usage line) now carries `cost`: the µ¢ the ledger records for the request after the cost factor, from the request's recorded charges and the gate's factor lookup. `usd_cents` is unchanged. A hooked recording now copies the usage object first, so keys the driver adds later never land in the ledger. * fix(ai): charge streamed speech once its first byte arrives Per-character TTS and the voice changer recorded their charge when the upstream's response headers arrived, so a body that failed before any audio was billed in full. A shared relay now charges on the first byte, or when the caller hangs up, and charges nothing for an upstream that fails or ends before sending audio. * fix(ai): release credit holds only after the request's usage lands A hold was released as soon as its operation returned, while the usage it stood for was still being written (behind the cost-factor lookup), so a concurrent request briefly saw neither. AI metering now tracks the request's usage writes and its holds (chat included) release once those settle, without delaying the response; a streamed charge chains its own. A failed write still releases the hold, and the hold's TTL backstops it. * test(ai-chat): wait for a stream's credit hold to come back after its usage lands * perf(fs): page subtree delete and copy by parent link, not by depth over the path range Each page of a recursive delete or copy sorted the whole remaining subtree by a computed depth, so a walk cost grew with the square of its size. It now pages the children of up to 100 sibling directories at a time by parent_uid, depth first, keyset on (parent_uid, id), so a page reads only its own rows; rows with no stored path are reached too. SQLite gets the parent_uid index the others have. * fix(fs): find children by parent_uid in the empty check and supplementary-name lookup Rows the old backend wrote carry parent_uid and no parent_id, so a non-recursive delete of a directory holding only such rows passed the empty check and orphaned them, and on MySQL the supplementary-character child lookup missed them, leaving their subtrees out of listings, counts and renames. Both now key on parent_uid; the empty check still probes parent_id too, which the delete cascade follows. * docs(fs): drop the single-POST note from the entry broadcast cap	2026-10-12 01:10:54 -07:00
Daniel Salazar	ed7e0b9cc4	fix: sharing, events and worker-credential authorization hardening (#3855) ... - PUT-1800: gate `createWorkerSessionToken` on actor type, so an app or an access token can no longer mint an app-less, root-shaped worker session; `WorkerDriver` binds on `effectiveApp` instead of `app`. - PUT-1799: add `isAccountContext` and read it where "no app" was being read as "the account" — handler publish, events-worker listing, kv handle mint/revoke/list. A scoped API token is no longer an account session. - PUT-1802: re-authorize a durable row before its backlog drains, settling it permanently when the grant is gone. Covers an ancestor-level unshare, which the revoke settle deliberately leaves to the delivery re-check. - PUT-1803: mask the owner's absolute path out of deliveries and subscription anchors on a foreign node, the way every FS surface already does. - PUT-1804: let a revoke reach rows already suspended for a resumable reason, re-stamping them so a resume cannot hand over the held backlog. - PUT-1805: apply the subscribe path's audience gate to `/events/fetch` before the query, so a cursor can no longer count and name invisible notifications. - PUT-1807: refuse `mode: 'manage'` from any actor holding an app — inside its own AppData the ACL short-circuit would otherwise supply the reach. - PUT-1808: take the sending peer from the verified signature header rather than the request body. - PUT-1810: re-base a kv share-handle row's stored match filter on the handle, so the owner's absolute key prefix stays hidden. - PUT-1814: escape LIKE wildcards and anchor the issuer-prefix queries on a segment; anchor `manage:` stripping; reject a backslash in a share prefix; assert a resolved actor in `subscribeDurable`. - PUT-1815: bound the char/varchar columns behind `event_subscriptions` and `kv_share_handles` at the store layer.	2026-09-11 21:18:49 -07:00
Daniel Salazar	eb8f497e9f	feat: presence and cross-region event forwarding (PUT-1679) (#3686) ... * feat: presence and cross-region event forwarding (PUT-1679) * fix: fan cache bumps to sibling nodes and stop the forward shed cascading (PUT-1679) `outer.events.generationBumped` and `outer.events.presenceBumped` rode `outer.*`, which the broadcast service only webhooks to peer regions; only `outer.pubsub.*` also fans over Redis to a region's other nodes. Both caches are per-process maps with no expiry, so a bump landing on one node left its siblings stale until that user's next transition. Renamed onto `outer.pubsub.events.*`; the listeners already accept the `from_outside` copy the Redis re-emit carries. `PeerForwardQueue.push` called `onOverflow` synchronously and the handler pushed markers straight back, each of which re-tripped the bound and shed the next item: one item over a 5000 bound recursed ~2200 deep, threw a RangeError, and turned ~2200 queued deliveries into gap markers. It also re-summed `bytes` over the whole queue per drop. The handler now returns its markers and the queue appends them past the bound check, sheds deliveries before markers, keeps one pending marker per (peer, subscription), and subtracts bytes per dropped item.	2026-09-03 15:39:21 -07:00
Juan Fernando Castro	5e1645d40e	🔧 PUT-1524: Sanity check file sharing api (#3603) ... * fix: sharing answers "who can reach this" from the grants, not the index Two ways a share listing could name access that was no longer there. A recipient's listing paired the flat permission read with the list of permissions it asked for by array index. That read drops misses and dedupes its keys, so the two are not positional: one entry's live grant vouched for another entry that had none, and listShared() kept publishing a withdrawn item's name, size and signed thumbnail URL to someone who could no longer open it. Read the permission off the value instead. getShares() had the same gap from the owner's side, with no liveness check at all — a grant withdrawn through /auth/revoke-user-user or an ACL mode change left the index row behind, and the owner was told someone could reach a file they could not. Checked against the grants now, one batched read per distinct holder. Pending invites are not subject to it: they have no grant yet, which is the point of them. Also covers the access-token actor, which reaches the same reach bound as an app through a different arm of the ACL check. No behavior change there — it was correct and untested. * test: pin that a revoke crosses regions A revoke reaches a peer region as a replicated SQL delete plus three invalidation events, one per cache the region owns: the u2u row cache, the flat view and the scan generation. The delete alone changes nothing there, and each cache has a different consequence if its event is lost — 20s for the generation, 5 minutes for the row cache, and forever for the flat view, whose grant-path entries carry no expiry. Nothing covered the whole path end to end. The store tests prove each event is emitted and applied; this proves the result, which is that the recipient stops being able to read. It asserts the read still succeeds after the SQL delete alone, so the test also records why replication is not sufficient on its own. * test: pin that a leaked uuid buys no access A masked share path hides which folder an item sits in; it was never the thing deciding who may open it. Nothing checked that at the route level, so the guarantee rested on unit tests of the resolver alone. Reads one shared file through its masked path, then tries the sibling four ways: the shared uuid with the sibling's name, the sibling's own uuid, a `..` back out of the root, and the owner's real path. Worth knowing about this one: it's mutation-checked. Removing the head !== root.name guard in sharePathMask.ts fails it with reachable: /testuser/55dd54c0…/share-http-1df8933e.txt. I verified that specifically because two tests I wrote earlier in this chunk passed with their guards broken — both were vacuous, and I deleted them rather than commit false assurance. * docs: say what listShared's total actually counts `total` counts the shares recorded for you; items are filtered after the page is read, so a withdrawn grant leaves the count higher than anything paging will yield. The page description already explained the short-page behaviour, but the field read as an exact count and the example printed it as one. The test pins the gap it describes: two shares, one withdrawn outside the index, one item listed and a total of two. * fix: keep an undelivered broadcast event instead of dropping it The outbound queue was cleared before the send, so a peer that timed out took its events with it. Most were survivable — a lost cache invalidation heals when the entry's TTL lapses. A revoke's flat-perm invalidation is not: grant-path entries carry no expiry, so a peer went on serving a withdrawn grant until something else wrote that key. Failed sends now go back on the queue, which the existing flush timer retries. Anything queued since wins over the retry, and the queue is bounded at 10,000 with the oldest dropped first, so a peer that stays down cannot grow it without limit. Each retry is signed at send time, so it is not rejected against the replay window. * fix: stop plus-addressing from deciding who a share reaches	2026-08-18 15:51:23 -07:00
Daniel Salazar	116d6e6663	tests: big test push for better coverage (#3490)	2026-08-01 14:29:59 -07:00
velzie	27def94d8b	feat: support cross-origin-isolated login (#3338)	2026-07-13 17:36:56 -04:00
Nariman Jelveh	2c0a1b9bbb	Revert "feat: add sign-in flow for crossOriginIsolated contexts (#3253)" (#3335) ... This reverts commit 6fff1fd75f.	2026-06-30 23:32:27 -07:00
velzie	6fff1fd75f	feat: add sign-in flow for crossOriginIsolated contexts (#3253) ... * poc * route login key through broadcastservice * cleanup * security validation and prevent additional webhook sends * block double-broadcast from redis	2026-06-30 20:17:53 +01:00
Daniel Salazar	b188942436	feat (PUT-1016 & PUT-1020) (#3164) ... * feat (PUT-1016 & PUT-1020) temp account preservation on forced relogin hosted asset cookies to v2 token too * fix: remove llm dashes and ugly comments * update agents	2026-05-26 23:35:16 -07:00
Daniel Salazar	594cbe03ef	tests: big test push with lots of missing areas (#3067) ... - Strengthen Broadcast tests - strengthen oidc tests - add middleware tests - add extensions test - add puter hosting tests - add Chat Completion Driver tests	2026-05-10 15:06:48 -07:00
Nariman Jelveh	267f464232	Add AGPL license headers to source files (#2877)	2026-05-01 13:50:42 -07:00
6 people	d4d78ac7db	rework: change backend and backend extensions to use simpler code structure and patterns (#2815) ... * fix: dynamodb health checks and client recreation (#2789) * wip: no nanoServices groundwork * feat: data clients in new shape * wip: auth and perms in new system * more wip * middlewaters mainly done * wip: fsv2 in new layout * old fs v2 migration * driver system * driver and old fs fixes * ai drivers wip * stream support * metering in ai chat driver * wip: new auth * rate limit and auth routes * captcha and anti csrf * fix: types * auth store * app logic * wip most other dricvers * fs * mostly kill all legacy stuff * fs finish * fix: redis usage * ai controller * driver cleanup * socket io in v2 * broadcast and crudq stuff * subdomains * notifcations and shares * fix bad syntaxes * auth wip Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * extensions * extension setup * more routes * sql migrations and default services * home router * tier 7 * everything else * everything else * remaining missing bits * server health * logs * cleanup * deps * cleanup 2 * more cleanup 2 * boot * fix launch * config fix * move file * fix: tsconfig things * fix: extension loading * launching * fix: drivers * fix: others * fix: icons * fix: file uploads * fs fixes * fix: fs api * fix: dev-center * config * add back telemetry * lint stuff * husky hooks * fix: fs oss * fix: config migration * config migration * migrate scripts + replicate * runner * fix: merge defafult config * fix: default region * fix: api domain * fix paths in readfile * fix fs entry default s3 * NS: Remove Referral && Entri Service * dep cleanups * fix: static assets * fix: kv and perms * fix: driver registrations * fix: home mapping * fix: rao * adding back 500 alarm * fix: build paths * fix: fs and kv shapes * fix: kv shape * more kv coercing and ai chat matching format as prior * fix: private app gates * private app caches * fix: whole bunch of legacy shape issues * update template jsonc * fix caching partial oidc and fs signed paths * more oidc fixes * fix: wip * fix: private apps * admin route fixes * fix: last few things hopefully * claude uploads * fix security for app only routes * fix kv system namespace * stuff * fix: app and kv and suggested apps * fix:open item * fix: FS operations * fix: default app icons * add back token-read and WSL support * metering fixes * fix: fsEntry * perm scanners and implicators * proper download endpoint * fix: download * fix anti csrft on v2 * fix file extensions, app icons * fold in v1 fixes from origin/main into v2 equivalents Re-applies the v1 fixes that landed on origin/main into their v2 counterparts since the v1 files were deleted on DS/wip during the v2 migration. v1 commits referenced below. - SQLBatcher: flush immediately when queue hits maxBatchSize instead of racing the timer (v1 12f48238). - RedisClient: drop maxRetriesPerRequest from 2 to 1 to shrink failure window (v1 b6776ab4). - ChatCompletionDriver: default minimumCredits to 1 when unset/zero so zero-cost precheck doesn't auto-pass (v1 36bd6073). - OpenAiImageProvider: add gpt-image-2 support — open-ended size rules, token-based cost estimator, arbitrary-size normalizer, isGpt prefix broadened to gpt-image- (v1 f14f1bf4). models.ts auto-merged via rename detection. - AppStore: bump row cache TTL from 5m to 24h (v1 6b3196ed). Not ported: v1 app-object Redis cache (bdfa12b5/b886dde3) — v2's #toClient recomputes filetype_associations/created_from_origin per read; adding a second cache layer is a larger change for a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * remoe anti-csrf from auth routes that had not used them * more icon fixes * fix worker functionality * fix: app and subdomain es Co-authored-by: Copilot <copilot@github.com> * fix PUT-761 * fix: PUT-748 * fix: rename fsService * Add security back to WorkerDriver * Migrate worker from fsEntry to fs. Fix cache issue * remove ability to create symlinks * strict webdav acl * require auth for wisp * chore: service renames * Add metering back to puter peer api * fix: PUT-760 PUT-749 * fix: PUT-746 * fix: peer cost Co-authored-by: Copilot <copilot@github.com> * fix: 771 * change order of peer controller * fix: create appdata folder for app on get auth token * fix: align delete site and list sites * delete: putility * fix subdomains * Add support for tilde in subdomains, fix subdomain update * cleanup PeerController.ts and fix billing oversight (#2844) * fix: PUT-786 * fix: bugs * fix: issues with multiple subdomain queries, or permission checks * fix: harden response shapes to not contain uneeded fields * fix: move state to redis * fix: missing kv methods + better sec Co-authored-by: Copilot <copilot@github.com> * fix: subdomainStore limit * fix: missing path resolution Co-authored-by: Copilot <copilot@github.com> * fs fixes * fix: undef error * fix fs + cleanup * fix: npm audit fixes * heal path entries where missing Co-authored-by: Copilot <copilot@github.com> * fix: caching Co-authored-by: Copilot <copilot@github.com> * fix: cache inconsistencies Co-authored-by: Copilot <copilot@github.com> * fix: app driver metadata Co-authored-by: Copilot <copilot@github.com> * remove extraneous comma * fix: associated app icons * fix: bad tool call * Add validation to WorkerDriver#getFilePaths * misc fs and auth issues Co-authored-by: Copilot <copilot@github.com> * fix: oidc errors Co-authored-by: Copilot <copilot@github.com> * fix: PUT-797 * fix: legacy appdata_app Co-authored-by: Copilot <copilot@github.com> * fix: add alert logs Co-authored-by: Copilot <copilot@github.com> * fix: error handling * Disable sharecontroller * fix: remove private user identifier for ai * fix: private app fixes * Add backback signup_server * fix: completionId size Co-authored-by: Copilot <copilot@github.com> * fix: revalidate path for oidc * fix: revalidate path for oidc * fix: email validation Co-authored-by: Copilot <copilot@github.com> * fix: user create query * fix: middleware extensions Co-authored-by: Copilot <copilot@github.com> * use x-forwarded-for for req ip forwarded * fix: missing last_activity ts * feat: add cache broadcast to subdomains * fix: update config typing --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: ProgrammerIn-wonderland <3838shah@gmail.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Nariman Jelveh <nj@puter.com> Co-authored-by: velzie <velzie@velzie.rip>	2026-04-30 12:13:43 -07:00