Daniel Salazar
360f6b1079
* fix(events): bill reached deliveries only, bound dispatch reads, seal cursors - A broadcast socket copy is billed only when a connection for its holder exists on this node, anywhere in this region (connection count) or, for a durable row, in a region presence names. Without peers it bills as before. - Compiled match filters are held in a bounded LRU with a TTL, and dropped when another process reports a subscription ended. - GET /events/fetch reads the continuation from `cursor`, falling back to `after`. - Cursors from /events/fetch, /events/subscriptions and /events/kv-handles are encrypted with sealCursor/openCursor; plain cursors still read. - A durable subscribe counts again after its insert and removes its own row when over the cap; the generation bump for that removal is published. - Subscribe checks access before reading the anchor's owner, and an anchor gone in between answers with the subject the caller sent. - Dispatch reads at most the rows per token routing can use, scanning large token hashes; a removal still settles every row on the removed anchor. - puter.events.unsubscribe() stops routing after the server ends the subscription or reports it gone; fetch() accepts `cursor`. * fix(events): forward kv values only to regions with rows that ask - A region counts its session rows asking for kv values per token and announces the token's value variant (`v#<token>`) over the existing `watch` item on the first such row, withdrawing it with the last. Add, remove, reap, reanchor and refresh all keep it, reading `includeValue` off the row as it is dropped. - A kv write reads the value variants alongside the watch index and attaches the value only to forwards bound for a region that announced one. * fix(pagination): seal id-keyed cursors on every list endpoint Apps, subdomains, workers, team (members, directory, teams, audit, own audit), shares (inbound, outbound), the user-to-user permission audit and readdir/descendant listings now hand out cursors sealed with sealCursor and read them with openCursor, so a cursor no longer carries a row id. Plain cursors issued before still read. * test(events): pass socket ids to noteConnect in the billing tests
4.5 KiB
Pagination convention for list APIs
Every list endpoint follows one wire contract. This document is the source of truth for adding pagination to a new or existing list surface.
Wire contract
Requests accept:
| Param | Type | Meaning |
|---|---|---|
limit |
number | Maximum items per page. Each endpoint documents its cap and default. |
cursor |
string | null | Opaque continuation token. null (or any presence of the key) requests the first page. |
offset |
number | Skip N items. Legacy/discouraged — see below. Cannot be combined with cursor. |
includeTotal |
boolean | Adds total to the response. |
Paginated responses are an envelope:
{ "items": [...], "cursor": "…", "total": 123 }
cursoris present only while more pages exist. Clients iterate until it is absent.totalis present only when the request setincludeTotal.- Pages may be short. Post-query filtering (TTL expiry, permission
checks) can shrink a page below
limit— or even to zero — while acursoris still returned. Never useitems.length < limitas an end-of-list signal.
New request params are camelCase (includeTotal, fetchUntilFull).
Pre-existing snake_case params stay for compatibility.
Backward compatibility
Requests without pagination params keep returning the full result in the legacy shape (bare array) forever — old clients never break.
The envelope trigger depends on the endpoint's history:
- Endpoints where
limitpre-dates the convention (readdir, subdomainselect) return the envelope only when the request containscursor(includingnull) orincludeTotal;limit/offset-only requests keep the bare array. - Endpoints where every pagination param is new (kv
list, workersgetFilePaths) return the envelope when any pagination param is present.
Cursors
Cursors are opaque base64-encoded JSON, produced and consumed only by the
backend (src/backend/util/pagination.ts). A cursor over a global sequence
(an auto-increment id) is sealed with sealCursor/openCursor instead, so
two pages can't be decoded into how fast the table grows; openCursor still
reads a plain cursor. What a cursor wraps is an implementation detail per
store:
- DynamoDB-backed lists wrap
LastEvaluatedKey. - SQL-backed lists wrap a keyset position —
(sortValue, id)of the last row — and the query seeks past it (WHERE (col, id) > (?, ?) ORDER BY col, id). Any SQL list gaining a cursor must have a deterministicORDER BYending in a unique tiebreaker (id).
Cursors that carry a sort also pin it: a request that passes a cursor plus a conflicting sort is rejected with 400.
Offset
SQL-backed endpoints support offset natively. DynamoDB-backed endpoints
(kv) emulate it by advancing past skipped items with Select: COUNT queries
— no item data is transferred, but read capacity is still consumed for
everything skipped, and the caller is metered for it. Offset is supported
for parity, not recommended — cost grows linearly with the offset, so it is
capped (kv: 5000). Use cursors.
Totals (includeTotal)
- SQL:
SELECT COUNT(*)with the same WHERE clause as the listing. - DynamoDB: a
Select: COUNTloop over the query (TTL-filtered), metered to the caller. Cost is proportional to the total item count — request the total on the first page only, not every page. - Where visibility is decided per-actor after the query (protected apps in
the catalog listing),
totalapproximates the visible set: it counts non-protected plus caller-owned rows, and misses rows visible only through explicit permission grants.
fetchUntilFull (kv only)
DynamoDB applies Limit before its filter expression, so TTL-filtered pages
are structurally short. fetchUntilFull: true makes the backend keep
fetching (bounded number of continuation queries) until the page holds
limit items or the keyset is exhausted. Requires limit. If the bound is
hit, the response simply carries a cursor — still convention-legal.
Adding pagination to a new endpoint
- Use
encodeCursor/decodeCursor/normalizeLimit/normalizeOffsetfromsrc/backend/util/pagination.ts—sealCursor/openCursorwhen the cursor wraps a global id. - Push equality filters into the query so pages and counts operate on the true result set; only genuinely per-actor filtering may remain post-query (short-pages rule covers it).
- Fetch
limit + 1rows to detect whether a next page exists (SQL), or useLastEvaluatedKey(DynamoDB). - Keep the no-params request returning the legacy full result.