11 Commits

Include renames

Author	SHA1	Message	Date
Daniel Salazar	4851df716c	fix(kv): KV share lifecycle hardening (#4429) ... * fix(kv): harden the KV share lifecycle - Retire a deleted app's live shares before an origin app is registered again under the same uid. The retire runs before the app row exists and throws on failure, so a failed launch leaves no row and the next one retries. - Record deleted-app retires and uninstall cleanups (retire, then purge) as reserved items before they run. A lease keeps one node on a job; a failed run raises an alarm and backs off, and a run that hits its time budget or is cut short by a restart resumes on the next check. Each job is bounded to handles made before it was recorded. - Uninstalling with deleteData records the job and returns; the retire and purge run off the request. The purge checks whether the app was authorized again before deleting each page, so data written after that is kept. - Take a per-region lock when minting and re-check for a live handle under it, so concurrent shares of one region make one handle. - Retire the handles an account holds before the account is deleted, so the owners' subscriptions through them end instead of staying active. * fix(events): bound share-handle session rows by the per-connection cap The count across a holder's connections also counted connections that had dropped but weren't reaped yet (a crashed node, a deploy, a network blip the server hasn't noticed), so resubscribes after a reconnect could be refused with `events_subscription_limit`. Rows made through a share handle now count toward the 50-per-connection cap like any other, and with the open-connections cap per account that bounds what one holder can put in the owner's keyspace. The handle index stays for ending rows on retire. Rows are added to it with the row itself and taken out when the row is dropped, and members whose row is gone are cleared when a retire reads it. * fix(events): keep a retire's durable settle running if ending sessions fails Ending a retired handle's session subscriptions reads Redis. A failure there is now logged and the durable suspension that follows still runs, so those rows' backlogs are still purged. * perf(events): ask once per handle when a refresh screens rows, and share handle reads A socket refresh asks whether a share handle still stands once per handle, holder and app, rather than once per row: rows through one handle share its grant. Concurrent cache misses on one handle share a single read; a forget drops the shared read so later callers start a fresh one. * fix(permissions): chunk the exact-match subtree delete The holder-scoped subtree delete names every matched grant, so a large subtree could exceed an engine's limit on bound parameters. It now deletes 500 grants per statement. * fix(kv): give each holder a share of a namespace's handle budget One holder could spend a namespace's whole handle budget and starve the others. Each holder now gets at most 3,750 of the namespace's 15,000 units per 10 seconds, and the owner's own calls through a handle spend none. The holder's budget is checked before the grant lookups, so throttled retries cost none; the namespace's is only told to callers the grant admits. Multi-unit charges use a new counted claim that keeps two counters per key instead of a sorted-set member per unit. * fix(kv): refuse handles older than their app and claim cleanup jobs globally - A share handle created before its app row reads as gone: in the KV driver, in the handle cache, on the events subscribe path, and in the delegate implicator. An origin app registered again under a deleted app's uid can't reach its predecessor's shares, whatever created the row. The app-token launch no longer retires inline; it records a background retire and never waits on it or fails because of it. The clock-skew rule is shared with app sessions. - Cleanup job records now carry their lease, attempts and retry time, and a node claims a job by compare-and-set on its record. A job stays with the deployment that recorded or last ran it, and another one adopts it only after it has gone unclaimed well past due. - The job check pages through records from where the last look stopped, so waiting records don't hide due ones. After 10 failed runs a job stops retrying with a warning alarm, and its record expires after 30 days. - A mint that waits out the region lock looks for the share again and returns it, or goes ahead, instead of failing with too_many_requests. - The record-failure alarm dedups; an unfinished run is a log line. * fix(kv): hide shares older than their app from re-shares and listings After an app is registered again under a deleted app's uid, its predecessor's live handles stayed visible: sharing the same region again handed back the old, dead handle, and both sides' listings showed the old handles and labels. findLive and both listing queries now leave out a live handle created before its app row, the same rule handle resolution uses, so a re-share makes a new handle and the listings, totals included, skip the old ones until their retire lands. A new share also retires, grants and all, the handles its owner gave the same holder for the earlier app first, so withdrawing those later can't take the new grant with it. * fix(events): keep a loose cap on subscriptions through one share handle Without a count across connections, a holder could put up to 20,000 rows in the owner's keyspace (50 per connection, 400 connections), each one evaluated on every owner write. The cap is back at 1,000 per holder per handle, far above what connections that dropped without being reaped leave behind. The index is only read row by row once it's past the cap. * fix(kv): harden the share cleanup jobs - Job records are keyed by domain, so a deployment that shares the table but not the database or broadcast (staging beside production) never runs another's jobs. - A job whose record can't be written runs unrecorded instead of not at all, and a failed check for live handles records the job anyway so it retries. Failures that used to be swallowed are logged. - Recording a job again keeps its attempts and backoff; one that gave up gets one more attempt. - A system retire gives back the grant of a live share it doesn't cover inside a withdrawn region, and a share made under a reused uid grants again once its row stands, so a concurrent retire can't leave a live handle without its grant. * fix(kv): don't cache a live handle a retire overtook during its app lookup The live read decided whether it could cache before awaiting the app lookup, so a retire's forget that landed during that lookup was overwritten with the handle as live for up to 60 s. It's now checked after the lookup. * chore(kv): word the job key comment for any install	2026-10-31 12:12:29 -07:00
Daniel Salazar	2dfd93687f	perf(database): defer primary fallback for user lookups (#4359)	2026-10-26 21:37:06 -07:00
Daniel Salazar	9ff224e22f	perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259) ... Pipeline - CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s. - `req.ua` is parsed on first read instead of on every request. - JSON `rawBody` reuses the parsed buffer instead of copying it. - `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call. - SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under load is unchanged. Auth and permissions - Token-to-actor auth starts the user, session and app lookups together and checks them in the same order: fewer sequential Redis round trips per authed request. - get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select after its own insert and doesn't wait on the cache write for replica rows. - getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer hands out a session that was never written. - whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged. - OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight per URL, and the Apple key is parsed once. - gui-token and sync-cookie use `req.actor.user` instead of re-reading the user. - listSessions batches app lookups. App icon probes run together, and the original-icon probe skips the lineage fallback. - Default-granted permissions are answered from config before the scan cache, and exploded options are deduped: one less Redis read per /drivers/call. - ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node first, then the ancestors concurrently on a miss. Filesystem - Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about 1.3 s to about 40 ms per 100 new files. - MySQL single-file inserts return the row built from the written values, falling back to the primary read: one less primary round trip per new file. - Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row fallback. - storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries `userUuid`: quota checks and /df no longer serialize the listener reads. - readdir cache backfill is one pipeline instead of 3xN SETEX. - Profile reads skip the lineage fallback; the own profile is read alongside the user lookup. - Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently. - Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream upstream error. - mkdir looks up target and parent together. - Upload completion reuses the session from the access check (status is still re-read before completing), and multipart signing reads it once. - Moving a shared file to Trash skips two lookups that always returned the entry itself. Drivers and events - Subscription resolvers run concurrently and are folded in registration order. - SubdomainDriver and WorkerDriver overlap independent lookups. - The events "anyone subscribed?" check is one Redis pipeline when the region is warm. - Handler publish overlaps the count with the size check. - Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays cross-region invalidations to the others. Sites, share and AI - Hosted sites resolve the owner first, then the app and root together. Site config loads only on 404s. The hosted-app query selects only the columns it uses and moves into AppStore. - shareFlags asks about the plan only when an item is shared by link only. - Share listings run the count, teams and plan reads alongside their page query. - Claude attachment cleanup no longer blocks the response. - New util `awaitLater` for promises started early and awaited later.	2026-10-17 12:03:35 -07:00
Juan Fernando Castro	e32bfa839d	fix: count one app open per account per app per hour (#4152) ... * fix: count one app open per account per app per day `app_opens` is read two ways at once: `open_count` is a row count and `user_count` is distinct accounts. A repeated open moves the first without the first moving the second, and the incentive report prices opens. An open is now marked as it is recorded and skipped for the rest of the day, before both the row and the event ClickHouse ingests, so the two stores agree. The marker is set only when absent, so the first open of the day wins a race; an unreachable marker counts the open, since counting twice beats not counting. * fix: count one app open per account per app per day, at the count Dropped the row-suppression approach. `app_opens` is not only a counter: `getRecentAppOpens` orders the desktop's recent list by `MAX(_id)`, and `app.opened` is the socket event other tabs reorder on. Suppressing the repeat row froze both at the first open of the day. Every open is recorded as before. `open_count` counts each account once per day instead, which is the figure `user_count` is read against. `countDistinctUserDays` folds the pair into one number because `COUNT(DISTINCT a, b)` is MySQL-only, and `dayBucket` covers MySQL's decimal division. ClickHouse has uniqExact over the tuple. Note the figure's meaning changes at deploy: a period spanning it mixes rows with account-days, so a comparison across that point is not one. * fix: count one app open per account per app per hour Review feedback: a day is too long a window for the repeat to collapse into. The period is now an argument rather than a baked-in 86400, held in one constant on AppStore beside the other store policy numbers, so the SQL and ClickHouse sites cannot drift and a future change is one line. The fold `COUNT(DISTINCT user * N + bucket)` is only injective while the bucket index stays under N. Hourly buckets are 24x the day index and pass the old 1000000 in 2084, so N moves to 10000000. Buckets are clock hours, not a rolling window, so two opens a minute apart either side of the hour still count twice — tested, and the same property the day version had at UTC midnight. The detailed-stats fixture gave its repeat open a 60s offset, which under hourly buckets would have counted twice whenever the suite ran in the first minute of an hour. It now shares a timestamp with the open it repeats.	2026-10-14 11:54:17 -04:00
Daniel Salazar	c815dc82a1	refactor(core-util): shared bounded map, cheaper route lifecycle, DDB/DB/rate-limit/gate dedup (#4185) ... * refactor(util): add BoundedTtlMap for core caches and log throttles One size-capped, LRU-evicting map with an optional TTL and a once-per-window shouldEmit helper. Replaces the hand-rolled maps in the DynamoDB throttle and repair loggers (the repair one was unbounded), the Slack repeat throttle, the auth probe's reauth log throttle, and the card-verification status memo. * perf(http): skip route lifecycle work when nothing listens The per-route lifecycle middleware built a payload, opened an event.emitAndWait span and hooked finish/close on every request, listeners or not. Each phase now checks hasListeners first. EventClient's three copies of the key-prefix walk become one memoized #matchKeys. * perf(ddb): write batch chunks concurrently and retry the TTL sweep batchPut/batchDel sent their 25-item chunks one after another; they now run up to four at a time through runWithConcurrencyLimit, and a failed chunk stops the ones not yet started. batchGet and the batch write share one #sendWithUnprocessedRetry instead of two copies of the backoff loop, and the TTL sweep goes through #batchWrite, so the unprocessed deletes it used to drop are retried. The AWS and local clients are built by one #bindClient. * refactor(db): share migrations and tryHardRead in the base client MySQL and Postgres each carried a copy of the migration-path runner and of the replica-racing tryHardRead; both now live on AbstractDatabaseClient (applyMigrationPaths, and tryHardRead gated on hasReadReplica), and SQLite drops its own tryHardRead override. Removes MySQL reinitPrimary, reinitReplica, isRetriableError and readWithRetry and the base requireRead, which had no callers outside tests. SQLite prepared statements and the Postgres SQL rewrite are memoized per query text in a bounded map. * refactor(rate-limit): route every entry point through one charge() The nine rate/peek/acquire entry points each repeated key and limit resolution, the backend call and the fail-open catch. They now share charge(), routeKey() and guardSlot(); log lines are unchanged. Driver concurrency slots release at most once like the other slot handles, and checkRateLimit takes { failOpen: false } for limits that must hold when the backend is down. * refactor(util): add secretsEqual for constant-time secret comparison One HMAC-then-timingSafeEqual helper, safe for inputs of any length, so callers stop hand-rolling the length guard. File signature verification uses it. * refactor(fs): infer signed-file MIME types with mime-types mimeFromName looks extensions up in mime-types instead of a 35-entry inline table, keeping the two types the table reported differently (ico, wav). The purpose-labelled signing key is derived once per secret instead of on every signature, where signFile derived it twice. * refactor(http): build route gates from one gate() helper Each gate repeated a try/catch -> next(err) wrapper or a .then(next, next) chain around an assertion. gate(check) does that once: a throw or rejection goes to next(err), and a synchronous check still calls next synchronously. The credits, subscription and reputation gate files held nothing but such a wrapper; their factories move into gates.ts and the files go. * perf(hosting): build the puter-site hosting config once createPuterSiteMiddleware rebuilt the normalized hosting-domain config on every hosted-site request; it is derived from static config, like the domain lists the middleware already builds at construction. * fix(db): tryHardRead's replica leg doesn't fail over to the primary MySQL's read() retries on the primary after a load-shed or transient replica error, so routing tryHardRead through it read the primary twice on that path. The replica leg now reads the replica alone. * fix(ddb): settle in-flight batch chunks before rethrowing A failed chunk stopped unstarted chunks but rejected while others were still retrying, so writes could land after the caller had handled the error and invalidated its cache.	2026-10-11 21:53:33 -07:00
Daniel Salazar	b67acdbf6f	fix(stores): return written rows without reading them from a lagging replica (#4166) ... * fix(stores): read rows back from the primary after writing them ShareStore (create, upsertPending, upsertActive, upsertActiveGroup, applyPending, upsertAnyone), AppStore (create, createFromOrigin, update, claimOwnership), NotificationStore.create, SubdomainStore.update and NotificationDriver.create returned the row they had just written by reading it back through `db.read`, which goes to the replica. Under replica lag that came back null or stale. Those read-backs now go to the primary, through a `primary` option on the getters. AppStore caches the primary copy on create, and claimOwnership no longer reads through getById, which cached the replica's pre-claim row on a miss. * fix(stores): skip the notify read-back; read OIDC link conflicts from the primary NotificationStore gains `insert`, which writes and returns the uid; `create` is now `insert` plus the primary read-back. NotificationService .notify ignores the row, so it uses `insert` and no longer pays a primary round trip per recipient. NotificationDriver.create still reads the row back from the primary. OIDCStore.link reads the conflicting row from the primary after a unique violation, so a replica that has not seen another account's link can no longer make the call report success. * perf(stores): return written rows instead of reading them back ShareStore.create, NotificationStore.create, AppStore.create and AppStore.createFromOrigin build the row they return from the values they inserted and the insert id, instead of reading it back from the primary. The DB-defaulted creation timestamp is now written explicitly from unix seconds (`timestampFromUnixSql`), so the returned value matches what a read returns (`timestampValue`); postgres INSERTs add RETURNING id. The AppStore rows fill every column the INSERT leaves to the schema and are cached as before. ShareStore.upsertPending's re-invite branch selects the full pending row in its dedupe read and returns it with the new mode and data; it falls back to the primary only when the UPDATE matched nothing. The remaining primary read-backs stay and say why: the share upserts and the new-invite insert (an upsert's surviving uid/created_at and the entry-owner subquery are only known to the primary), applyPending and claimOwnership (only an id or uid in hand), AppStore.update and SubdomainStore.update (the base row is a replica copy that would be cached), NotificationDriver.create (notify returns only the uid) and the OIDC link conflict.	2026-10-11 17:51:38 -07:00
Daniel Salazar	116d6e6663	tests: big test push for better coverage (#3490)	2026-08-01 14:29:59 -07:00
Ron HernausandDaniel Salazar	d2fee51844	Support PostgreSQL database backend (#3167) ... * feat(database): add postgres database client Adds the PostgreSQL database client, native bootstrap migration, SQL preparation helpers, and database config/factory wiring.\n\nRefs #3165. * feat(database): make backend queries postgres-aware Updates runtime SQL call sites for database-specific booleans, identifiers, insert-ignore, upserts, JSON extraction, intervals, and Postgres insert ids.\n\nRefs #3165. * test(database): cover postgres client behavior Adds unit coverage for SQL preparation, factory selection, write-result mapping, and transaction rollback/commit ordering, plus an env-gated PostgreSQL integration flow.\n\nRefs #3165. * docs(self-hosting): document postgres database setup Adds PostgreSQL configuration examples and migration path guidance for self-hosted deployments.\n\nRefs #3165. * fix: harden postgres oidc tests * fix(postgres): normalize query results and SQL prep * fix(user): preserve normalized cache booleans * test(postgres): run integration coverage with pgmock * tests: add way to run all tests with postgres though slow Also adding note that postgres is not in active use so might not work out the box --------- Co-authored-by: Daniel Salazar <daniel.salazar@puter.com>	2026-05-28 14:12:17 -07:00
Daniel SalazarandCopilot	50679e6a01	selfhosting: p2 fixed default configs and guide + some other errors (#2903) ... * fix: ollama support * fix: more self host cleanup * fix: bad auth * fix: more session cookie stuff * allow emailclient Co-authored-by: Copilot <copilot@github.com> * fix: broken migration * fix: full-stack comment * fix: hardcoded perms * new docker build steps --------- Co-authored-by: Copilot <copilot@github.com>	2026-05-04 18:14:23 -07:00
Nariman Jelveh	267f464232	Add AGPL license headers to source files (#2877)	2026-05-01 13:50:42 -07:00
6 people	d4d78ac7db	rework: change backend and backend extensions to use simpler code structure and patterns (#2815) ... * fix: dynamodb health checks and client recreation (#2789) * wip: no nanoServices groundwork * feat: data clients in new shape * wip: auth and perms in new system * more wip * middlewaters mainly done * wip: fsv2 in new layout * old fs v2 migration * driver system * driver and old fs fixes * ai drivers wip * stream support * metering in ai chat driver * wip: new auth * rate limit and auth routes * captcha and anti csrf * fix: types * auth store * app logic * wip most other dricvers * fs * mostly kill all legacy stuff * fs finish * fix: redis usage * ai controller * driver cleanup * socket io in v2 * broadcast and crudq stuff * subdomains * notifcations and shares * fix bad syntaxes * auth wip Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * extensions * extension setup * more routes * sql migrations and default services * home router * tier 7 * everything else * everything else * remaining missing bits * server health * logs * cleanup * deps * cleanup 2 * more cleanup 2 * boot * fix launch * config fix * move file * fix: tsconfig things * fix: extension loading * launching * fix: drivers * fix: others * fix: icons * fix: file uploads * fs fixes * fix: fs api * fix: dev-center * config * add back telemetry * lint stuff * husky hooks * fix: fs oss * fix: config migration * config migration * migrate scripts + replicate * runner * fix: merge defafult config * fix: default region * fix: api domain * fix paths in readfile * fix fs entry default s3 * NS: Remove Referral && Entri Service * dep cleanups * fix: static assets * fix: kv and perms * fix: driver registrations * fix: home mapping * fix: rao * adding back 500 alarm * fix: build paths * fix: fs and kv shapes * fix: kv shape * more kv coercing and ai chat matching format as prior * fix: private app gates * private app caches * fix: whole bunch of legacy shape issues * update template jsonc * fix caching partial oidc and fs signed paths * more oidc fixes * fix: wip * fix: private apps * admin route fixes * fix: last few things hopefully * claude uploads * fix security for app only routes * fix kv system namespace * stuff * fix: app and kv and suggested apps * fix:open item * fix: FS operations * fix: default app icons * add back token-read and WSL support * metering fixes * fix: fsEntry * perm scanners and implicators * proper download endpoint * fix: download * fix anti csrft on v2 * fix file extensions, app icons * fold in v1 fixes from origin/main into v2 equivalents Re-applies the v1 fixes that landed on origin/main into their v2 counterparts since the v1 files were deleted on DS/wip during the v2 migration. v1 commits referenced below. - SQLBatcher: flush immediately when queue hits maxBatchSize instead of racing the timer (v1 12f48238). - RedisClient: drop maxRetriesPerRequest from 2 to 1 to shrink failure window (v1 b6776ab4). - ChatCompletionDriver: default minimumCredits to 1 when unset/zero so zero-cost precheck doesn't auto-pass (v1 36bd6073). - OpenAiImageProvider: add gpt-image-2 support — open-ended size rules, token-based cost estimator, arbitrary-size normalizer, isGpt prefix broadened to gpt-image- (v1 f14f1bf4). models.ts auto-merged via rename detection. - AppStore: bump row cache TTL from 5m to 24h (v1 6b3196ed). Not ported: v1 app-object Redis cache (bdfa12b5/b886dde3) — v2's #toClient recomputes filetype_associations/created_from_origin per read; adding a second cache layer is a larger change for a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * remoe anti-csrf from auth routes that had not used them * more icon fixes * fix worker functionality * fix: app and subdomain es Co-authored-by: Copilot <copilot@github.com> * fix PUT-761 * fix: PUT-748 * fix: rename fsService * Add security back to WorkerDriver * Migrate worker from fsEntry to fs. Fix cache issue * remove ability to create symlinks * strict webdav acl * require auth for wisp * chore: service renames * Add metering back to puter peer api * fix: PUT-760 PUT-749 * fix: PUT-746 * fix: peer cost Co-authored-by: Copilot <copilot@github.com> * fix: 771 * change order of peer controller * fix: create appdata folder for app on get auth token * fix: align delete site and list sites * delete: putility * fix subdomains * Add support for tilde in subdomains, fix subdomain update * cleanup PeerController.ts and fix billing oversight (#2844) * fix: PUT-786 * fix: bugs * fix: issues with multiple subdomain queries, or permission checks * fix: harden response shapes to not contain uneeded fields * fix: move state to redis * fix: missing kv methods + better sec Co-authored-by: Copilot <copilot@github.com> * fix: subdomainStore limit * fix: missing path resolution Co-authored-by: Copilot <copilot@github.com> * fs fixes * fix: undef error * fix fs + cleanup * fix: npm audit fixes * heal path entries where missing Co-authored-by: Copilot <copilot@github.com> * fix: caching Co-authored-by: Copilot <copilot@github.com> * fix: cache inconsistencies Co-authored-by: Copilot <copilot@github.com> * fix: app driver metadata Co-authored-by: Copilot <copilot@github.com> * remove extraneous comma * fix: associated app icons * fix: bad tool call * Add validation to WorkerDriver#getFilePaths * misc fs and auth issues Co-authored-by: Copilot <copilot@github.com> * fix: oidc errors Co-authored-by: Copilot <copilot@github.com> * fix: PUT-797 * fix: legacy appdata_app Co-authored-by: Copilot <copilot@github.com> * fix: add alert logs Co-authored-by: Copilot <copilot@github.com> * fix: error handling * Disable sharecontroller * fix: remove private user identifier for ai * fix: private app fixes * Add backback signup_server * fix: completionId size Co-authored-by: Copilot <copilot@github.com> * fix: revalidate path for oidc * fix: revalidate path for oidc * fix: email validation Co-authored-by: Copilot <copilot@github.com> * fix: user create query * fix: middleware extensions Co-authored-by: Copilot <copilot@github.com> * use x-forwarded-for for req ip forwarded * fix: missing last_activity ts * feat: add cache broadcast to subdomains * fix: update config typing --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: ProgrammerIn-wonderland <3838shah@gmail.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Nariman Jelveh <nj@puter.com> Co-authored-by: velzie <velzie@velzie.rip>	2026-04-30 12:13:43 -07:00