Commit Graph
7 Commits
Author SHA1 Message Date
Daniel Salazar 9ff224e22f perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259)
Backend Tests / changes (push) Canceled after 0s
Backend Tests / typecheck (push) Canceled after 0s
Backend Tests / test shard 1/4 (push) Canceled after 0s
Backend Tests / test shard 2/4 (push) Canceled after 0s
Backend Tests / test shard 3/4 (push) Canceled after 0s
Backend Tests / test shard 4/4 (push) Canceled after 0s
Backend Tests / test (main) (push) Canceled after 0s
Backend Tests / coverage (push) Canceled after 0s
Maintain Release Merge PR / update-release-pr (push) Canceled after 0s
Notify HeyPuter / notify (push) Canceled after 0s
release-please / release-please (push) Canceled after 0s
Pipeline
- CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s.
- `req.ua` is parsed on first read instead of on every request.
- JSON `rawBody` reuses the parsed buffer instead of copying it.
- `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call.
- SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and
  waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under
  load is unchanged.

Auth and permissions
- Token-to-actor auth starts the user, session and app lookups together and checks them in the
  same order: fewer sequential Redis round trips per authed request.
- get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select
  after its own insert and doesn't wait on the cache write for replica rows.
- getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer
  hands out a session that was never written.
- whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged.
- OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight
  per URL, and the Apple key is parsed once.
- gui-token and sync-cookie use `req.actor.user` instead of re-reading the user.
- listSessions batches app lookups. App icon probes run together, and the original-icon probe
  skips the lineage fallback.
- Default-granted permissions are answered from config before the scan cache, and exploded
  options are deduped: one less Redis read per /drivers/call.
- ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node
  first, then the ancestors concurrently on a miss.

Filesystem
- Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about
  1.3 s to about 40 ms per 100 new files.
- MySQL single-file inserts return the row built from the written values, falling back to the
  primary read: one less primary round trip per new file.
- Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row
  fallback.
- storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries
  `userUuid`: quota checks and /df no longer serialize the listener reads.
- readdir cache backfill is one pipeline instead of 3xN SETEX.
- Profile reads skip the lineage fallback; the own profile is read alongside the user lookup.
- Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently.
- Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream
  at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream
  upstream error.
- mkdir looks up target and parent together.
- Upload completion reuses the session from the access check (status is still re-read before
  completing), and multipart signing reads it once.
- Moving a shared file to Trash skips two lookups that always returned the entry itself.

Drivers and events
- Subscription resolvers run concurrently and are folded in registration order.
- SubdomainDriver and WorkerDriver overlap independent lookups.
- The events "anyone subscribed?" check is one Redis pipeline when the region is warm.
- Handler publish overlaps the count with the size check.
- Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays
  cross-region invalidations to the others.

Sites, share and AI
- Hosted sites resolve the owner first, then the app and root together. Site config loads only
  on 404s. The hosted-app query selects only the columns it uses and moves into AppStore.
- shareFlags asks about the plan only when an item is shared by link only.
- Share listings run the count, teams and plan reads alongside their page query.
- Claude attachment cleanup no longer blocks the response.
- New util `awaitLater` for promises started early and awaited later.
2026-10-17 12:03:35 -07:00
Daniel Salazar c815dc82a1 refactor(core-util): shared bounded map, cheaper route lifecycle, DDB/DB/rate-limit/gate dedup (#4185)
* refactor(util): add BoundedTtlMap for core caches and log throttles

One size-capped, LRU-evicting map with an optional TTL and a once-per-window
shouldEmit helper. Replaces the hand-rolled maps in the DynamoDB throttle and
repair loggers (the repair one was unbounded), the Slack repeat throttle, the
auth probe's reauth log throttle, and the card-verification status memo.

* perf(http): skip route lifecycle work when nothing listens

The per-route lifecycle middleware built a payload, opened an
event.emitAndWait span and hooked finish/close on every request, listeners
or not. Each phase now checks hasListeners first. EventClient's three copies
of the key-prefix walk become one memoized #matchKeys.

* perf(ddb): write batch chunks concurrently and retry the TTL sweep

batchPut/batchDel sent their 25-item chunks one after another; they now run
up to four at a time through runWithConcurrencyLimit, and a failed chunk
stops the ones not yet started. batchGet and the batch write share one
#sendWithUnprocessedRetry instead of two copies of the backoff loop, and the
TTL sweep goes through #batchWrite, so the unprocessed deletes it used to
drop are retried. The AWS and local clients are built by one #bindClient.

* refactor(db): share migrations and tryHardRead in the base client

MySQL and Postgres each carried a copy of the migration-path runner and of
the replica-racing tryHardRead; both now live on AbstractDatabaseClient
(applyMigrationPaths, and tryHardRead gated on hasReadReplica), and SQLite
drops its own tryHardRead override. Removes MySQL reinitPrimary,
reinitReplica, isRetriableError and readWithRetry and the base requireRead,
which had no callers outside tests. SQLite prepared statements and the
Postgres SQL rewrite are memoized per query text in a bounded map.

* refactor(rate-limit): route every entry point through one charge()

The nine rate/peek/acquire entry points each repeated key and limit
resolution, the backend call and the fail-open catch. They now share
charge(), routeKey() and guardSlot(); log lines are unchanged. Driver
concurrency slots release at most once like the other slot handles, and
checkRateLimit takes { failOpen: false } for limits that must hold when the
backend is down.

* refactor(util): add secretsEqual for constant-time secret comparison

One HMAC-then-timingSafeEqual helper, safe for inputs of any length, so
callers stop hand-rolling the length guard. File signature verification
uses it.

* refactor(fs): infer signed-file MIME types with mime-types

mimeFromName looks extensions up in mime-types instead of a 35-entry inline
table, keeping the two types the table reported differently (ico, wav). The
purpose-labelled signing key is derived once per secret instead of on every
signature, where signFile derived it twice.

* refactor(http): build route gates from one gate() helper

Each gate repeated a try/catch -> next(err) wrapper or a .then(next, next)
chain around an assertion. gate(check) does that once: a throw or rejection
goes to next(err), and a synchronous check still calls next synchronously.
The credits, subscription and reputation gate files held nothing but such a
wrapper; their factories move into gates.ts and the files go.

* perf(hosting): build the puter-site hosting config once

createPuterSiteMiddleware rebuilt the normalized hosting-domain config on
every hosted-site request; it is derived from static config, like the domain
lists the middleware already builds at construction.

* fix(db): tryHardRead's replica leg doesn't fail over to the primary

MySQL's read() retries on the primary after a load-shed or transient
replica error, so routing tryHardRead through it read the primary twice
on that path. The replica leg now reads the replica alone.

* fix(ddb): settle in-flight batch chunks before rethrowing

A failed chunk stopped unstarted chunks but rejected while others were
still retrying, so writes could land after the caller had handled the
error and invalidated its cache.
2026-10-11 21:53:33 -07:00
Daniel Salazar dcb61bc4fd fix: PUT-1824 + PUT-1884 + PUT-1952 + PUT-1827 (#3951) 2026-09-25 17:47:03 -07:00
Daniel Salazar 132cae9f6a Bug bash fixes (PUT-1876, 1877, 1880, 1881, 1948, 1949, 1951) and KV event fan-out by plan (#3943)
* fix(kv): keep disableSharing entries out of other apps' events (PUT-1876)

Private kv keys and values are no longer delivered to cross-app subscriptions, including forwarded and durable deliveries.

* fix(fs): refuse usernames whose home path still holds another account's rows (PUT-1877)

Username claims (signup, save_account, change-username, OIDC, team provisioning) now also check for leftover rows under /<name>/, and the root listing only returns the actor's own home.

* fix(puter-js): reconnect the events channel after a server-side disconnect (PUT-1881)

A server hang-up is retried with a bounded backoff; persistent handlers survive it and get an optional onError when the connection can't be restored.

* fix(backend): let the graceful shutdown finish before telemetry exits (PUT-1948)

The telemetry preload no longer exits on signals; the entry point drains, tears down top-down, flushes telemetry last, then exits. DB pools stay open until teardown.

* fix(fs): reserve quota for signed uploads in progress (PUT-1880)

startWrite/startBatchWrite hold each upload's declared size in a per-owner cache reservation until it completes, aborts or expires, so a burst of starts can't all read the same committed usage. Caps in-flight uploads at 10,000 per account.

* fix(fs): refuse to complete a signed upload whose object never arrived (PUT-1951)

Completion now 400s when the object store reports the key missing (transient errors stay lenient), abort no longer deletes an object a live entry still uses, and a ghost entry is only removed after a second miss at its recorded location.

* fix(metering): move usage to v2 keys with a small totals item, per-model detail shards and a 3-month TTL (PUT-1949)

The billed item holds only totals; per-model detail lives in 100 hash-sharded items read in one batch and cached for 60s, and more than 5,000 distinct usage types in a month fold into other. Global and per-app aggregates store totals only, appTotals comes from a prefix listing, a refused path only skips the key that refused it, exact reads near the allowance are throttled per key, and every metering record expires after 3 months. September usage restarts at deploy; the monthly-charge claim stays on the v1 key through September so recurring charges don't fire twice.

* test(events): let the worker backoff test tolerate a loaded run

* feat(events): scale KV event fan-out with the key owner's plan

A KV change delivers to up to 512 matching subscriptions (2,048 filter evaluations) for a paid key owner and 128 (512) otherwise, inline values are dropped once more than 128 match, and paid accounts may hold up to 512 share handles per app. The owner's plan is only looked up when a change has more candidates than any cap.

* test(metering): seed the 150-app listing test directly so it fits under coverage
2026-09-24 16:00:52 -07:00
Daniel Salazar 116d6e6663 tests: big test push for better coverage (#3490) 2026-08-01 14:29:59 -07:00
Daniel Salazar 645409eddb fix: misc hardening (#3418) 2026-07-21 13:55:00 -07:00
Daniel SalazarandClaude Opus 4.7 ac5eecb7f3 feat (put-1019 put-1021): v2 auth revoke endpoints + silent v1->v2 to… (#3158)
* feat (put-1019 put-1021): v2 auth revoke endpoints + silent v1->v2 token migration

PUT-1019 (AUTH-5): full revoke-endpoint coverage
- /logout: soft-revoke web session + its asset cookies via revokeCascade
  (app sessions and access tokens survive)
- POST /auth/revoke-session: cascade per row kind (web/app/access_token/asset)
- POST /auth/revoke-all-sessions: revoke all web rows for user; optional
  include_apps=true nuclear option; gated by userProtected (cookie-only)
- revokeAccessToken: soft-revoke matching access_token row in addition to
  removing access_token_permissions
- All revokes are UPDATE revoked_at = now(); no DELETE statements remain

PUT-1021 (SDK-1): backend POST /auth/migrate-token
- v1 access_token/app -> mint matching-kind v2 token, idempotent on
  (auth_id, kind, token_uid)
- v1 web/session -> 409 { code: "reauth_required" } (interactive relogin only)
- Same-origin / signed-referer hardening; rate-limited per IP and auth_id
- Gated by auth.allow_v1_tokens; emits puter_token_v2 cookie for app-in-browser

DB migrations: mysql_mig_10, sqlite 0053 (sessions.access_token_uid column)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(put-1019): reject self-revoke; enrich list-sessions response

- handleRevokeSession refuses uuid === req.actor.session.uid (use /logout
  instead). The cookie that authenticated the call should never be the
  target of a self-revoke — the response can't write fresh auth state
  and the client ends up with an ambiguous identity. revoke-all-sessions
  still has the explicit include_current opt-in for the nuclear case.

- AuthService.listSessions now joins the apps table for kind='app' rows
  (returning { uid, name, title, icon } so the manage-sessions UI can
  render the authorizing app without a second round trip), surfaces
  kind / expires_at / label / last_ip / created_via, and filters out
  asset rows (per-cookie children of web rows, revoked transitively via
  cascade — surfacing them as standalone entries would be confusing).

- Sort order: current session first, then most-recently-active. UI
  relies on this to anchor "you are here" at the top.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(put-1019 put-1021): review nits — origin normalization, cookie fallback, types, migration order

B1: createAccessToken.options.expiresIn widened to string | number.
    The impl (#hardExpiryFromExpiresIn) and existing callers/tests use
    jsonwebtoken-style strings ('1h', '30d'); narrowing to number forced
    unsafe casts at every call site. Cast at the single sign() boundary
    where jsonwebtoken's typed template-literal SignOptions clashes
    with the wider runtime contract.

B2: Inline comment on the DELETE in access_token_permissions. The
    AUTH-5 "no DELETE on revoke" rule scoped to the `sessions` table
    (where the cascade graph + audit trail matter). Permissions rows
    are the grant manifest for an active token — once its session is
    soft-revoked they're dead-weight cache entries. A future audit
    requirement would land as a `revoked_at` column on this table,
    not a behavior change in this PR.

B3: handleMigrateToken now sets the puter_token_v2 cookie (with the
    shared sessionCookieFlags + httpOnly) when the migration result
    is kind='app'. The endpoint is already gated on Origin so the
    caller is by definition in a browser; access tokens deliberately
    skip the cookie since they're programmatic.

B4: #isMigrateTokenOriginAllowed normalizes both incoming origin and
    config.origin / allowlist entries (trim + strip trailing slash +
    lowercase) before equality. A misconfigured `config.origin =
    "https://puter.com/"` would otherwise reject every same-origin
    browser call.

B5: Replaced 4x `this.config.cookie_name!` non-null assertions in
    AuthController with `(this.config.cookie_name ?? 'puter_token')`.
    IConfig is `Partial<IConfigOptional>` so cookie_name is undefined
    at runtime in some deployments / test setups; the fallback matches
    the pattern in userProtected / OIDCController / puterSite.

B6: MySQLDatabaseClient sorts migrations numerically by trailing
    integer instead of lexically. Existing files use unpadded names
    (`mysql_mig_<N>.sql`), so plain `.sort()` ran mysql_mig_10 before
    mysql_mig_2 — a future migration that depended on _2..9 running
    first would break. Non-numeric filenames fall through to
    localeCompare for determinism.

B7: Restored the docstring for SessionStore.getOrCreateApp's
    `opts.auth_id` ("Stable per-user identity (survives re-login);
    carried on every v2 JWT so manage-sessions can group by identity")
    — the previous edit truncated it to a fragment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test + feat: backend test coverage for PUT-1019/1021 review fixes + worker session methods

Tests
-----
- compareMigrationFilenames (new) covering B6 numeric-sort: confirms
  mysql_mig_10.sql lands after mysql_mig_9.sql; non-numeric files sort
  after numbered ones; stable for already-ordered input.
- listSessions (AuthService.test.ts): excludes kind="asset" rows;
  enriches with kind/expires_at/last_ip/created_via; joins kind="app"
  rows with the apps table; sorts current first then by last_activity
  desc.
- handleRevokeSession (AuthController.test.ts): refuses self-revoke
  with 400; still allows revoking a sibling session.
- handleMigrateToken (AuthController.test.ts): rejects missing/
  disallowed Origin; tolerates trailing slash and uppercase Origin
  (B4 normalization); returns 409 reauth_required for v1 web tokens;
  does NOT set the cookie for access-token migration; DOES set the
  puter_token_v2 cookie (httpOnly + sessionCookieFlags) for
  app-under-user migration.
- SessionStore tests updated to import APP_WINDOW_SECONDS /
  WEB_WINDOW_SECONDS rather than hardcoded 30/90 day values — the
  windows just got bumped to 1y and the assertions need to follow
  the constant.

Refactor
--------
- MySQLDatabaseClient exports compareMigrationFilenames so the sort
  logic is unit-testable in isolation.

Worker tokens
-------------
- AuthService.createWorkerSessionToken(user, meta?): mints a new
  kind="web" row tagged meta.worker=true, expires_at =
  WORKER_WINDOW_SECONDS, returns { session, token, gui_token }
  with worker: true on each JWT.
- AuthService.createWorkerAppToken(actor, appUid): mints a new
  kind="app" row tagged meta.worker=true, expires_at =
  WORKER_WINDOW_SECONDS, returns an app-under-user JWT with
  worker: true. Note the existing idx_sessions_user_app_active
  unique index will collide with an existing non-worker app
  session for the same (user, app) — future schema work can
  carve workers out of that uniqueness.

SessionStore.js: WEB/APP_WINDOW_SECONDS now 1y;
WORKER_WINDOW_SECONDS = 99y added for the worker path.

Full backend suite: 2172 passed / 16 skipped / 0 failed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:49:36 -07:00