mirror of
https://github.com/HeyPuter/puter.git
synced 2026-11-17 11:17:46 +00:00
perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259)
Backend Tests / changes (push) Canceled after 0s
Backend Tests / typecheck (push) Canceled after 0s
Backend Tests / test shard 1/4 (push) Canceled after 0s
Backend Tests / test shard 2/4 (push) Canceled after 0s
Backend Tests / test shard 3/4 (push) Canceled after 0s
Backend Tests / test shard 4/4 (push) Canceled after 0s
Backend Tests / test (main) (push) Canceled after 0s
Backend Tests / coverage (push) Canceled after 0s
Maintain Release Merge PR / update-release-pr (push) Canceled after 0s
Notify HeyPuter / notify (push) Canceled after 0s
release-please / release-please (push) Canceled after 0s
Backend Tests / changes (push) Canceled after 0s
Backend Tests / typecheck (push) Canceled after 0s
Backend Tests / test shard 1/4 (push) Canceled after 0s
Backend Tests / test shard 2/4 (push) Canceled after 0s
Backend Tests / test shard 3/4 (push) Canceled after 0s
Backend Tests / test shard 4/4 (push) Canceled after 0s
Backend Tests / test (main) (push) Canceled after 0s
Backend Tests / coverage (push) Canceled after 0s
Maintain Release Merge PR / update-release-pr (push) Canceled after 0s
Notify HeyPuter / notify (push) Canceled after 0s
release-please / release-please (push) Canceled after 0s
Pipeline - CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s. - `req.ua` is parsed on first read instead of on every request. - JSON `rawBody` reuses the parsed buffer instead of copying it. - `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call. - SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under load is unchanged. Auth and permissions - Token-to-actor auth starts the user, session and app lookups together and checks them in the same order: fewer sequential Redis round trips per authed request. - get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select after its own insert and doesn't wait on the cache write for replica rows. - getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer hands out a session that was never written. - whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged. - OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight per URL, and the Apple key is parsed once. - gui-token and sync-cookie use `req.actor.user` instead of re-reading the user. - listSessions batches app lookups. App icon probes run together, and the original-icon probe skips the lineage fallback. - Default-granted permissions are answered from config before the scan cache, and exploded options are deduped: one less Redis read per /drivers/call. - ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node first, then the ancestors concurrently on a miss. Filesystem - Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about 1.3 s to about 40 ms per 100 new files. - MySQL single-file inserts return the row built from the written values, falling back to the primary read: one less primary round trip per new file. - Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row fallback. - storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries `userUuid`: quota checks and /df no longer serialize the listener reads. - readdir cache backfill is one pipeline instead of 3xN SETEX. - Profile reads skip the lineage fallback; the own profile is read alongside the user lookup. - Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently. - Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream upstream error. - mkdir looks up target and parent together. - Upload completion reuses the session from the access check (status is still re-read before completing), and multipart signing reads it once. - Moving a shared file to Trash skips two lookups that always returned the entry itself. Drivers and events - Subscription resolvers run concurrently and are folded in registration order. - SubdomainDriver and WorkerDriver overlap independent lookups. - The events "anyone subscribed?" check is one Redis pipeline when the region is warm. - Handler publish overlaps the count with the size check. - Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays cross-region invalidations to the others. Sites, share and AI - Hosted sites resolve the owner first, then the app and root together. Site config loads only on 404s. The hosted-app query selects only the columns it uses and moves into AppStore. - shareFlags asks about the plan only when an item is shared by link only. - Share listings run the count, teams and plan reads alongside their page query. - Claude attachment cleanup no longer blocks the response. - New util `awaitLater` for promises started early and awaited later.
This commit is contained in:
1 parent
9ad7b99607
commit
9ff224e22f
89 files changed
+9275
-720
No files matched your search
@@ -435,6 +435,105 @@ describe('MySQLDatabaseClient — query interface', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ method: 'pread', poolIndex: 0, maxTimeInQueue: 30 },
|
||||
{ method: 'read', poolIndex: 1, maxTimeInQueue: 10 },
|
||||
] as const)(
|
||||
'$method: flushes lone reads at once until 8 are in flight, then waits $maxTimeInQueue ms and coalesces',
|
||||
async ({ method, poolIndex, maxTimeInQueue }) => {
|
||||
const client = await startClient({ replica: { host: 'replica' } });
|
||||
const pool = createdPools[poolIndex];
|
||||
const held: Array<() => void> = [];
|
||||
pool.respond = (sql) =>
|
||||
new Promise((resolve) => {
|
||||
const sets = sql
|
||||
.split(';')
|
||||
.slice(0, -1)
|
||||
.map((stmt) => [{ stmt }]);
|
||||
held.push(() => resolve(rowSets(...sets)));
|
||||
});
|
||||
vi.useFakeTimers({
|
||||
toFake: [
|
||||
'setTimeout',
|
||||
'clearTimeout',
|
||||
'setImmediate',
|
||||
'clearImmediate',
|
||||
],
|
||||
});
|
||||
try {
|
||||
const reads: Promise<unknown>[] = [];
|
||||
for (let i = 0; i < 8; i++) {
|
||||
reads.push(client[method](`SELECT ${i}`));
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
}
|
||||
expect(pool.calls).toHaveLength(8);
|
||||
|
||||
reads.push(client[method]('SELECT 8'));
|
||||
await vi.advanceTimersByTimeAsync(maxTimeInQueue - 1);
|
||||
reads.push(client[method]('SELECT 9'));
|
||||
expect(pool.calls).toHaveLength(8);
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(pool.calls.map((c) => c.sql).at(-1)).toBe(
|
||||
'SELECT 8;SELECT 9; SELECT 1',
|
||||
);
|
||||
|
||||
for (const release of held.splice(0)) release();
|
||||
const rows = await Promise.all(reads);
|
||||
expect(rows.at(-2)).toEqual([{ stmt: 'SELECT 8' }]);
|
||||
expect(rows.at(-1)).toEqual([{ stmt: 'SELECT 9' }]);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
{ setup: 'single node', replica: undefined, waits: true },
|
||||
{ setup: 'replica', replica: { host: 'replica' }, waits: false },
|
||||
])(
|
||||
'$setup: 8 primary reads in flight hold back a lone read: $waits',
|
||||
async ({ replica, waits }) => {
|
||||
const client = await startClient(replica ? { replica } : {});
|
||||
const held: Array<() => void> = [];
|
||||
for (const pool of createdPools) {
|
||||
pool.respond = (sql) =>
|
||||
new Promise((resolve) => {
|
||||
const sets = sql
|
||||
.split(';')
|
||||
.slice(0, -1)
|
||||
.map((stmt) => [{ stmt }]);
|
||||
held.push(() => resolve(rowSets(...sets)));
|
||||
});
|
||||
}
|
||||
const sent = () => createdPools.flatMap((pool) => pool.calls);
|
||||
vi.useFakeTimers({
|
||||
toFake: [
|
||||
'setTimeout',
|
||||
'clearTimeout',
|
||||
'setImmediate',
|
||||
'clearImmediate',
|
||||
],
|
||||
});
|
||||
try {
|
||||
const reads: Promise<unknown>[] = [];
|
||||
for (let i = 0; i < 8; i++) {
|
||||
reads.push(client.pread(`SELECT ${i}`));
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
}
|
||||
reads.push(client.read('SELECT 8'));
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
expect(sent()).toHaveLength(waits ? 8 : 9);
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(sent()).toHaveLength(9);
|
||||
|
||||
for (const release of held.splice(0)) release();
|
||||
await Promise.all(reads);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it('sends a write on its own, outside a transaction', async () => {
|
||||
const client = await startClient();
|
||||
const pool = createdPools[0];
|
||||
@@ -472,6 +571,23 @@ describe('MySQLDatabaseClient — query interface', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('passes the warning count through, so callers can tell a clean write', async () => {
|
||||
const client = await startClient();
|
||||
createdPools[0].respond = () =>
|
||||
writeHeader({ insertId: 9, affectedRows: 1, warningStatus: 1 });
|
||||
|
||||
await expect(
|
||||
client.write('INSERT IGNORE INTO `user` (`username`) VALUES (?)', [
|
||||
'ada',
|
||||
]),
|
||||
).resolves.toEqual({
|
||||
insertId: 9,
|
||||
affectedRows: 1,
|
||||
anyRowsAffected: true,
|
||||
warningStatus: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it('reports no rows affected when the header omits the counters', async () => {
|
||||
const client = await startClient();
|
||||
createdPools[0].respond = () => writeHeader({});
|
||||
|
||||
Reference in new issue
Block a user