mirror of
https://github.com/HeyPuter/puter.git
synced 2026-11-04 13:03:37 +00:00
2dfd93687f09b402679cd411eec8d5900dbec564
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Daniel Salazar
|
2dfd93687f | perf(database): defer primary fallback for user lookups (#4359) | ||
|
Daniel Salazar
|
9ff224e22f |
perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259)
Backend Tests / changes (push) Canceled after 0s
Backend Tests / typecheck (push) Canceled after 0s
Backend Tests / test shard 1/4 (push) Canceled after 0s
Backend Tests / test shard 2/4 (push) Canceled after 0s
Backend Tests / test shard 3/4 (push) Canceled after 0s
Backend Tests / test shard 4/4 (push) Canceled after 0s
Backend Tests / test (main) (push) Canceled after 0s
Backend Tests / coverage (push) Canceled after 0s
Maintain Release Merge PR / update-release-pr (push) Canceled after 0s
Notify HeyPuter / notify (push) Canceled after 0s
release-please / release-please (push) Canceled after 0s
Pipeline - CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s. - `req.ua` is parsed on first read instead of on every request. - JSON `rawBody` reuses the parsed buffer instead of copying it. - `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call. - SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under load is unchanged. Auth and permissions - Token-to-actor auth starts the user, session and app lookups together and checks them in the same order: fewer sequential Redis round trips per authed request. - get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select after its own insert and doesn't wait on the cache write for replica rows. - getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer hands out a session that was never written. - whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged. - OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight per URL, and the Apple key is parsed once. - gui-token and sync-cookie use `req.actor.user` instead of re-reading the user. - listSessions batches app lookups. App icon probes run together, and the original-icon probe skips the lineage fallback. - Default-granted permissions are answered from config before the scan cache, and exploded options are deduped: one less Redis read per /drivers/call. - ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node first, then the ancestors concurrently on a miss. Filesystem - Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about 1.3 s to about 40 ms per 100 new files. - MySQL single-file inserts return the row built from the written values, falling back to the primary read: one less primary round trip per new file. - Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row fallback. - storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries `userUuid`: quota checks and /df no longer serialize the listener reads. - readdir cache backfill is one pipeline instead of 3xN SETEX. - Profile reads skip the lineage fallback; the own profile is read alongside the user lookup. - Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently. - Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream upstream error. - mkdir looks up target and parent together. - Upload completion reuses the session from the access check (status is still re-read before completing), and multipart signing reads it once. - Moving a shared file to Trash skips two lookups that always returned the entry itself. Drivers and events - Subscription resolvers run concurrently and are folded in registration order. - SubdomainDriver and WorkerDriver overlap independent lookups. - The events "anyone subscribed?" check is one Redis pipeline when the region is warm. - Handler publish overlaps the count with the size check. - Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays cross-region invalidations to the others. Sites, share and AI - Hosted sites resolve the owner first, then the app and root together. Site config loads only on 404s. The hosted-app query selects only the columns it uses and moves into AppStore. - shareFlags asks about the plan only when an item is shared by link only. - Share listings run the count, teams and plan reads alongside their page query. - Claude attachment cleanup no longer blocks the response. - New util `awaitLater` for promises started early and awaited later. |
||
|
Daniel Salazar
|
c815dc82a1 |
refactor(core-util): shared bounded map, cheaper route lifecycle, DDB/DB/rate-limit/gate dedup (#4185)
* refactor(util): add BoundedTtlMap for core caches and log throttles
One size-capped, LRU-evicting map with an optional TTL and a once-per-window
shouldEmit helper. Replaces the hand-rolled maps in the DynamoDB throttle and
repair loggers (the repair one was unbounded), the Slack repeat throttle, the
auth probe's reauth log throttle, and the card-verification status memo.
* perf(http): skip route lifecycle work when nothing listens
The per-route lifecycle middleware built a payload, opened an
event.emitAndWait span and hooked finish/close on every request, listeners
or not. Each phase now checks hasListeners first. EventClient's three copies
of the key-prefix walk become one memoized #matchKeys.
* perf(ddb): write batch chunks concurrently and retry the TTL sweep
batchPut/batchDel sent their 25-item chunks one after another; they now run
up to four at a time through runWithConcurrencyLimit, and a failed chunk
stops the ones not yet started. batchGet and the batch write share one
#sendWithUnprocessedRetry instead of two copies of the backoff loop, and the
TTL sweep goes through #batchWrite, so the unprocessed deletes it used to
drop are retried. The AWS and local clients are built by one #bindClient.
* refactor(db): share migrations and tryHardRead in the base client
MySQL and Postgres each carried a copy of the migration-path runner and of
the replica-racing tryHardRead; both now live on AbstractDatabaseClient
(applyMigrationPaths, and tryHardRead gated on hasReadReplica), and SQLite
drops its own tryHardRead override. Removes MySQL reinitPrimary,
reinitReplica, isRetriableError and readWithRetry and the base requireRead,
which had no callers outside tests. SQLite prepared statements and the
Postgres SQL rewrite are memoized per query text in a bounded map.
* refactor(rate-limit): route every entry point through one charge()
The nine rate/peek/acquire entry points each repeated key and limit
resolution, the backend call and the fail-open catch. They now share
charge(), routeKey() and guardSlot(); log lines are unchanged. Driver
concurrency slots release at most once like the other slot handles, and
checkRateLimit takes { failOpen: false } for limits that must hold when the
backend is down.
* refactor(util): add secretsEqual for constant-time secret comparison
One HMAC-then-timingSafeEqual helper, safe for inputs of any length, so
callers stop hand-rolling the length guard. File signature verification
uses it.
* refactor(fs): infer signed-file MIME types with mime-types
mimeFromName looks extensions up in mime-types instead of a 35-entry inline
table, keeping the two types the table reported differently (ico, wav). The
purpose-labelled signing key is derived once per secret instead of on every
signature, where signFile derived it twice.
* refactor(http): build route gates from one gate() helper
Each gate repeated a try/catch -> next(err) wrapper or a .then(next, next)
chain around an assertion. gate(check) does that once: a throw or rejection
goes to next(err), and a synchronous check still calls next synchronously.
The credits, subscription and reputation gate files held nothing but such a
wrapper; their factories move into gates.ts and the files go.
* perf(hosting): build the puter-site hosting config once
createPuterSiteMiddleware rebuilt the normalized hosting-domain config on
every hosted-site request; it is derived from static config, like the domain
lists the middleware already builds at construction.
* fix(db): tryHardRead's replica leg doesn't fail over to the primary
MySQL's read() retries on the primary after a load-shed or transient
replica error, so routing tryHardRead through it read the primary twice
on that path. The replica leg now reads the replica alone.
* fix(ddb): settle in-flight batch chunks before rethrowing
A failed chunk stopped unstarted chunks but rejected while others were
still retrying, so writes could land after the caller had handled the
error and invalidated its cache.
|
||
|
Daniel Salazar
|
dcb61bc4fd | fix: PUT-1824 + PUT-1884 + PUT-1952 + PUT-1827 (#3951) | ||
|
Daniel Salazar
|
132cae9f6a |
Bug bash fixes (PUT-1876, 1877, 1880, 1881, 1948, 1949, 1951) and KV event fan-out by plan (#3943)
* fix(kv): keep disableSharing entries out of other apps' events (PUT-1876) Private kv keys and values are no longer delivered to cross-app subscriptions, including forwarded and durable deliveries. * fix(fs): refuse usernames whose home path still holds another account's rows (PUT-1877) Username claims (signup, save_account, change-username, OIDC, team provisioning) now also check for leftover rows under /<name>/, and the root listing only returns the actor's own home. * fix(puter-js): reconnect the events channel after a server-side disconnect (PUT-1881) A server hang-up is retried with a bounded backoff; persistent handlers survive it and get an optional onError when the connection can't be restored. * fix(backend): let the graceful shutdown finish before telemetry exits (PUT-1948) The telemetry preload no longer exits on signals; the entry point drains, tears down top-down, flushes telemetry last, then exits. DB pools stay open until teardown. * fix(fs): reserve quota for signed uploads in progress (PUT-1880) startWrite/startBatchWrite hold each upload's declared size in a per-owner cache reservation until it completes, aborts or expires, so a burst of starts can't all read the same committed usage. Caps in-flight uploads at 10,000 per account. * fix(fs): refuse to complete a signed upload whose object never arrived (PUT-1951) Completion now 400s when the object store reports the key missing (transient errors stay lenient), abort no longer deletes an object a live entry still uses, and a ghost entry is only removed after a second miss at its recorded location. * fix(metering): move usage to v2 keys with a small totals item, per-model detail shards and a 3-month TTL (PUT-1949) The billed item holds only totals; per-model detail lives in 100 hash-sharded items read in one batch and cached for 60s, and more than 5,000 distinct usage types in a month fold into other. Global and per-app aggregates store totals only, appTotals comes from a prefix listing, a refused path only skips the key that refused it, exact reads near the allowance are throttled per key, and every metering record expires after 3 months. September usage restarts at deploy; the monthly-charge claim stays on the v1 key through September so recurring charges don't fire twice. * test(events): let the worker backoff test tolerate a loaded run * feat(events): scale KV event fan-out with the key owner's plan A KV change delivers to up to 512 matching subscriptions (2,048 filter evaluations) for a paid key owner and 128 (512) otherwise, inline values are dropped once more than 128 match, and paid accounts may hold up to 512 share handles per app. The owner's plan is only looked up when a change has more candidates than any cap. * test(metering): seed the 150-app listing test directly so it fits under coverage |
||
|
Daniel Salazar
|
0be3bc55c2 |
feat(metering): AI cost multiplier hook for AI drivers (#3898)
Emits ai.cost.multiplier.<driver>.<provider>:<model> before recording AI usage, so what a model costs to charge is policy an extension owns rather than a number in core. Nothing listening records the provider cost. MeteringService.withAiCostMultiplier(driver) returns a view of the service whose recording paths scale costOverride by the hook's answer; every AI driver hands that view to its providers, so all of them are covered without touching provider code. |
||
|
Daniel Salazar
|
116d6e6663 | tests: big test push for better coverage (#3490) | ||
|
Daniel Salazar
|
645409eddb | fix: misc hardening (#3418) | ||
|
Daniel Salazar
|
17a9485fb2 | Add spans to drivers and high frequency paths (#3376) | ||
|
Ron HernausandDaniel Salazar
|
d2fee51844 |
Support PostgreSQL database backend (#3167)
* feat(database): add postgres database client Adds the PostgreSQL database client, native bootstrap migration, SQL preparation helpers, and database config/factory wiring.\n\nRefs #3165. * feat(database): make backend queries postgres-aware Updates runtime SQL call sites for database-specific booleans, identifiers, insert-ignore, upserts, JSON extraction, intervals, and Postgres insert ids.\n\nRefs #3165. * test(database): cover postgres client behavior Adds unit coverage for SQL preparation, factory selection, write-result mapping, and transaction rollback/commit ordering, plus an env-gated PostgreSQL integration flow.\n\nRefs #3165. * docs(self-hosting): document postgres database setup Adds PostgreSQL configuration examples and migration path guidance for self-hosted deployments.\n\nRefs #3165. * fix: harden postgres oidc tests * fix(postgres): normalize query results and SQL prep * fix(user): preserve normalized cache booleans * test(postgres): run integration coverage with pgmock * tests: add way to run all tests with postgres though slow Also adding note that postgres is not in active use so might not work out the box --------- Co-authored-by: Daniel Salazar <daniel.salazar@puter.com> |
||
|
Daniel SalazarandClaude Opus 4.7
|
ac5eecb7f3 |
feat (put-1019 put-1021): v2 auth revoke endpoints + silent v1->v2 to… (#3158)
* feat (put-1019 put-1021): v2 auth revoke endpoints + silent v1->v2 token migration
PUT-1019 (AUTH-5): full revoke-endpoint coverage
- /logout: soft-revoke web session + its asset cookies via revokeCascade
(app sessions and access tokens survive)
- POST /auth/revoke-session: cascade per row kind (web/app/access_token/asset)
- POST /auth/revoke-all-sessions: revoke all web rows for user; optional
include_apps=true nuclear option; gated by userProtected (cookie-only)
- revokeAccessToken: soft-revoke matching access_token row in addition to
removing access_token_permissions
- All revokes are UPDATE revoked_at = now(); no DELETE statements remain
PUT-1021 (SDK-1): backend POST /auth/migrate-token
- v1 access_token/app -> mint matching-kind v2 token, idempotent on
(auth_id, kind, token_uid)
- v1 web/session -> 409 { code: "reauth_required" } (interactive relogin only)
- Same-origin / signed-referer hardening; rate-limited per IP and auth_id
- Gated by auth.allow_v1_tokens; emits puter_token_v2 cookie for app-in-browser
DB migrations: mysql_mig_10, sqlite 0053 (sessions.access_token_uid column)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(put-1019): reject self-revoke; enrich list-sessions response
- handleRevokeSession refuses uuid === req.actor.session.uid (use /logout
instead). The cookie that authenticated the call should never be the
target of a self-revoke — the response can't write fresh auth state
and the client ends up with an ambiguous identity. revoke-all-sessions
still has the explicit include_current opt-in for the nuclear case.
- AuthService.listSessions now joins the apps table for kind='app' rows
(returning { uid, name, title, icon } so the manage-sessions UI can
render the authorizing app without a second round trip), surfaces
kind / expires_at / label / last_ip / created_via, and filters out
asset rows (per-cookie children of web rows, revoked transitively via
cascade — surfacing them as standalone entries would be confusing).
- Sort order: current session first, then most-recently-active. UI
relies on this to anchor "you are here" at the top.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(put-1019 put-1021): review nits — origin normalization, cookie fallback, types, migration order
B1: createAccessToken.options.expiresIn widened to string | number.
The impl (#hardExpiryFromExpiresIn) and existing callers/tests use
jsonwebtoken-style strings ('1h', '30d'); narrowing to number forced
unsafe casts at every call site. Cast at the single sign() boundary
where jsonwebtoken's typed template-literal SignOptions clashes
with the wider runtime contract.
B2: Inline comment on the DELETE in access_token_permissions. The
AUTH-5 "no DELETE on revoke" rule scoped to the `sessions` table
(where the cascade graph + audit trail matter). Permissions rows
are the grant manifest for an active token — once its session is
soft-revoked they're dead-weight cache entries. A future audit
requirement would land as a `revoked_at` column on this table,
not a behavior change in this PR.
B3: handleMigrateToken now sets the puter_token_v2 cookie (with the
shared sessionCookieFlags + httpOnly) when the migration result
is kind='app'. The endpoint is already gated on Origin so the
caller is by definition in a browser; access tokens deliberately
skip the cookie since they're programmatic.
B4: #isMigrateTokenOriginAllowed normalizes both incoming origin and
config.origin / allowlist entries (trim + strip trailing slash +
lowercase) before equality. A misconfigured `config.origin =
"https://puter.com/"` would otherwise reject every same-origin
browser call.
B5: Replaced 4x `this.config.cookie_name!` non-null assertions in
AuthController with `(this.config.cookie_name ?? 'puter_token')`.
IConfig is `Partial<IConfigOptional>` so cookie_name is undefined
at runtime in some deployments / test setups; the fallback matches
the pattern in userProtected / OIDCController / puterSite.
B6: MySQLDatabaseClient sorts migrations numerically by trailing
integer instead of lexically. Existing files use unpadded names
(`mysql_mig_<N>.sql`), so plain `.sort()` ran mysql_mig_10 before
mysql_mig_2 — a future migration that depended on _2..9 running
first would break. Non-numeric filenames fall through to
localeCompare for determinism.
B7: Restored the docstring for SessionStore.getOrCreateApp's
`opts.auth_id` ("Stable per-user identity (survives re-login);
carried on every v2 JWT so manage-sessions can group by identity")
— the previous edit truncated it to a fragment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test + feat: backend test coverage for PUT-1019/1021 review fixes + worker session methods
Tests
-----
- compareMigrationFilenames (new) covering B6 numeric-sort: confirms
mysql_mig_10.sql lands after mysql_mig_9.sql; non-numeric files sort
after numbered ones; stable for already-ordered input.
- listSessions (AuthService.test.ts): excludes kind="asset" rows;
enriches with kind/expires_at/last_ip/created_via; joins kind="app"
rows with the apps table; sorts current first then by last_activity
desc.
- handleRevokeSession (AuthController.test.ts): refuses self-revoke
with 400; still allows revoking a sibling session.
- handleMigrateToken (AuthController.test.ts): rejects missing/
disallowed Origin; tolerates trailing slash and uppercase Origin
(B4 normalization); returns 409 reauth_required for v1 web tokens;
does NOT set the cookie for access-token migration; DOES set the
puter_token_v2 cookie (httpOnly + sessionCookieFlags) for
app-under-user migration.
- SessionStore tests updated to import APP_WINDOW_SECONDS /
WEB_WINDOW_SECONDS rather than hardcoded 30/90 day values — the
windows just got bumped to 1y and the assertions need to follow
the constant.
Refactor
--------
- MySQLDatabaseClient exports compareMigrationFilenames so the sort
logic is unit-testable in isolation.
Worker tokens
-------------
- AuthService.createWorkerSessionToken(user, meta?): mints a new
kind="web" row tagged meta.worker=true, expires_at =
WORKER_WINDOW_SECONDS, returns { session, token, gui_token }
with worker: true on each JWT.
- AuthService.createWorkerAppToken(actor, appUid): mints a new
kind="app" row tagged meta.worker=true, expires_at =
WORKER_WINDOW_SECONDS, returns an app-under-user JWT with
worker: true. Note the existing idx_sessions_user_app_active
unique index will collide with an existing non-worker app
session for the same (user, app) — future schema work can
carve workers out of that uniqueness.
SessionStore.js: WEB/APP_WINDOW_SECONDS now 1y;
WORKER_WINDOW_SECONDS = 99y added for the worker path.
Full backend suite: 2172 passed / 16 skipped / 0 failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Daniel SalazarandCopilot
|
50679e6a01 |
selfhosting: p2 fixed default configs and guide + some other errors (#2903)
* fix: ollama support * fix: more self host cleanup * fix: bad auth * fix: more session cookie stuff * allow emailclient Co-authored-by: Copilot <copilot@github.com> * fix: broken migration * fix: full-stack comment * fix: hardcoded perms * new docker build steps --------- Co-authored-by: Copilot <copilot@github.com> |
||
|
Daniel Salazar
|
2de8807942 |
selfhost: fullstack self host p1 (#2902)
* wip: self hosted full setup * wip: self hoast modes * more wip self host stuff * wip: full release * fix: custom apps * fix: readme * full-stack md * docs update * fix * fix: commands * remove comments * fix: docs and migs * mysql mig * fix: docker changes * fix: mysql checks * fix: mysql mig * fix: docker more * fix: redis conn issues |
||
|
Nariman Jelveh
|
267f464232 | Add AGPL license headers to source files (#2877) | ||
|
Daniel SalazarandCopilot
|
f6e1f90225 |
fix: dbConfigs and ai routes (#2868)
Docker Image CI / build-and-push-image (push) Has been cancelled
Maintain Release Merge PR / update-release-pr (push) Has been cancelled
Notify HeyPuter / notify (push) Has been cancelled
release-please / release-please (push) Has been cancelled
test / test-backend (24.x) (push) Has been cancelled
test / API tests (node env, api-test) (24.x) (push) Has been cancelled
test / puterjs (node env, vitest) (24.x) (push) Has been cancelled
* fix: db configs Co-authored-by: Copilot <copilot@github.com> * fix: ai routes Co-authored-by: Copilot <copilot@github.com> --------- Co-authored-by: Copilot <copilot@github.com> |
||
|
|
d4d78ac7db |
rework: change backend and backend extensions to use simpler code structure and patterns (#2815)
* fix: dynamodb health checks and client recreation (#2789) * wip: no nanoServices groundwork * feat: data clients in new shape * wip: auth and perms in new system * more wip * middlewaters mainly done * wip: fsv2 in new layout * old fs v2 migration * driver system * driver and old fs fixes * ai drivers wip * stream support * metering in ai chat driver * wip: new auth * rate limit and auth routes * captcha and anti csrf * fix: types * auth store * app logic * wip most other dricvers * fs * mostly kill all legacy stuff * fs finish * fix: redis usage * ai controller * driver cleanup * socket io in v2 * broadcast and crudq stuff * subdomains * notifcations and shares * fix bad syntaxes * auth wip Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * extensions * extension setup * more routes * sql migrations and default services * home router * tier 7 * everything else * everything else * remaining missing bits * server health * logs * cleanup * deps * cleanup 2 * more cleanup 2 * boot * fix launch * config fix * move file * fix: tsconfig things * fix: extension loading * launching * fix: drivers * fix: others * fix: icons * fix: file uploads * fs fixes * fix: fs api * fix: dev-center * config * add back telemetry * lint stuff * husky hooks * fix: fs oss * fix: config migration * config migration * migrate scripts + replicate * runner * fix: merge defafult config * fix: default region * fix: api domain * fix paths in readfile * fix fs entry default s3 * NS: Remove Referral && Entri Service * dep cleanups * fix: static assets * fix: kv and perms * fix: driver registrations * fix: home mapping * fix: rao * adding back 500 alarm * fix: build paths * fix: fs and kv shapes * fix: kv shape * more kv coercing and ai chat matching format as prior * fix: private app gates * private app caches * fix: whole bunch of legacy shape issues * update template jsonc * fix caching partial oidc and fs signed paths * more oidc fixes * fix: wip * fix: private apps * admin route fixes * fix: last few things hopefully * claude uploads * fix security for app only routes * fix kv system namespace * stuff * fix: app and kv and suggested apps * fix:open item * fix: FS operations * fix: default app icons * add back token-read and WSL support * metering fixes * fix: fsEntry * perm scanners and implicators * proper download endpoint * fix: download * fix anti csrft on v2 * fix file extensions, app icons * fold in v1 fixes from origin/main into v2 equivalents Re-applies the v1 fixes that landed on origin/main into their v2 counterparts since the v1 files were deleted on DS/wip during the v2 migration. v1 commits referenced below. - SQLBatcher: flush immediately when queue hits maxBatchSize instead of racing the timer (v1 12f48238). - RedisClient: drop maxRetriesPerRequest from 2 to 1 to shrink failure window (v1 |