refactor(stores): one row cache and keyset page helper, batched cache round trips, dead store methods removed (#4191)

* refactor(stores): one read-through RowCache for UserStore and AppStore

UserStore and AppStore each carried the same multi-key row cache: key
builder, cache read, tombstone-guarded backfill, broadcast refresh,
invalidate and tombstone-on-delete. They now share RowCache on PuterStore,
which also single-flights concurrent misses for one key and batches
getByIds/getByUids: one pipelined tombstone check for the loaded rows and
one pipelined backfill instead of per-row awaits. Cache keys and TTLs are
unchanged.

* perf(apps): invalidate cached app lists by generation, not by tracker scan

Every app write SMEMBERS'd a global set of every cached list key, parsed
each key back into its filters and deleted the matches. List pages are now
keyed by their owner's generation token (or a shared one for unscoped
lists), and a write replaces the tokens it affects: O(1) per write, and a
page loaded before a write can no longer be cached after it.

* refactor(stores): one keyset page helper for store listings

Thirteen store listings each cut a LIMIT n+1 read to a page and sealed the
last row's position, and each opened an id cursor its own way. They now use
keysetPage() and openIdCursor() from util/pagination. Cursor payloads and
sealing are unchanged; a cursor without a usable id still fails with 400 in
ShareStore and still restarts the listing elsewhere (strict option).

* refactor(stores): delete store methods nothing calls

AppStore.existsByIndexUrl, OIDCStore.unlinkByUserId,
TeamStore.listByOwner/listActiveSeatUuids, SubdomainStore.getByDomain/
listByDomain, FSEntryStore.ensureDirectoriesForUser,
ShareStore.listByIssuer/listByRecipientEmail/deleteByRecipientEmail/
getDailyShareCount/listByHolderAndFsentries and
PermissionStore.invalidateScanCache/deleteUserUserPermsByPermissionPrefix
had no production callers. ShareStore.listByFsentries was a copy of
listReaching; its callers use the latter.

SubdomainStore SMEMBERS'd a per-user prefix-list tracker on every write and
broadcast its deletion, but nothing ever wrote that set or cached a prefix
listing. The reads are gone.

* refactor(stores): shared escapeLike and bumpGeneration helpers

escapeLike replaces the LIKE escaping copied into PermissionStore (twice),
ShareStore and FSEntryStore, and now also guards SubdomainStore's prefix
filters, which passed caller prefixes into LIKE unescaped.

bumpGeneration sends INCR and EXPIRE in one pipeline instead of two awaited
commands in PermissionStore, EventSubscriptionStore and PresenceStore.

* refactor(stores): read affected rows off WriteResult

Every database client returns affectedRows and anyRowsAffected, so the
`affectedRows ?? changes ?? 0` fallbacks in ShareStore, NotificationStore,
UserStore and UserBlockStore read those directly.

* perf(stores): batch flat-permission KV deletes and durable cache drops

PermissionStore deleted each flat KV entry with its own kv.del; one
kv.batchDel now covers the set. DurableSubscriptionStore's suspend and reap
dropped each row from the cache separately; EventSubscriptionStore gains
dropDurables, which drops them per owner in one pass, and dropDurable calls
it. suspend keeps one conditional UPDATE per row, which is how concurrent
settles learn which rows each of them suspended.

* refactor(fs): fold FSEntryStore's duplicate getters, sort maps and cache keys

- getEntryByUuid and getEntryById share one by-column getter.
- listChildren and listChildrenPage read one sort table; the offset listing
  now coalesces NULL modified/size like the cursor listing, so both order
  NULLs the same way on every engine.
- Entry cache keys come from one builder instead of hand-built strings.
- getEntriesByIds, getEntriesByPaths and the per-user path reader fetch
  their cache hits in one pipelined round trip instead of a GET per key,
  through readCacheValues, which RowCache.readMany now uses too.

* fix(apps): cache the row a lost createFromOrigin race reads from the primary

Keeps AppStore's private #writeCache as a RowCache write, so code that
still calls it (#4166's read-back) loads in either merge order.

src/backend/util/pagination.test.ts

@@ -2,9 +2,11 @@ import { describe, expect, it } from 'vitest';
 import {
     decodeCursor,
     encodeCursor,
+    keysetPage,
     normalizeLimit,
     normalizeOffset,
     openCursor,
+    openIdCursor,
     sealCursor,
 } from './pagination';
 import { HttpError } from '../core/http';
@@ -139,4 +141,65 @@ describe('pagination util', () => {
             expect(normalizeOffset(5000, { cap: 5000 })).toBe(5000);
         });
     });
+
+    describe('openIdCursor', () => {
+        const secret = 'keyset-secret';
+
+        it('returns null without a cursor', () => {
+            expect(openIdCursor(undefined, secret)).toBeNull();
+            expect(openIdCursor('', secret)).toBeNull();
+        });
+
+        it('reads the id from a sealed or plain cursor', () => {
+            expect(openIdCursor(sealCursor({ id: 41 }, secret), secret)).toBe(
+                41,
+            );
+            expect(openIdCursor(encodeCursor({ id: 7 }), secret)).toBe(7);
+        });
+
+        it('reads a cursor without a usable id as the first page', () => {
+            expect(
+                openIdCursor(encodeCursor({ appUid: 'app-1' }), secret),
+            ).toBeNull();
+            expect(openIdCursor(encodeCursor({ id: -3 }), secret)).toBeNull();
+        });
+
+        it('refuses a cursor without a usable id when strict', () => {
+            expect(() =>
+                openIdCursor(encodeCursor({ id: 'x' }), secret, {
+                    label: 'share cursor',
+                    strict: true,
+                }),
+            ).toThrowError('invalid share cursor');
+        });
+
+        it('refuses an unreadable cursor either way', () => {
+            expect(() => openIdCursor('!!!', secret)).toThrowError(HttpError);
+        });
+    });
+
+    describe('keysetPage', () => {
+        const secret = 'keyset-secret';
+        const rows = [{ id: 3 }, { id: 5 }, { id: 9 }];
+
+        it('returns every row and no cursor when nothing follows', () => {
+            expect(keysetPage(rows, 3, secret)).toEqual({ rows });
+        });
+
+        it('cuts to the limit and points the cursor at the last row kept', () => {
+            const page = keysetPage(rows, 2, secret);
+            expect(page.rows).toEqual([{ id: 3 }, { id: 5 }]);
+            expect(openIdCursor(page.cursor, secret)).toBe(5);
+        });
+
+        it('takes a custom position, plain without a secret', () => {
+            const page = keysetPage(
+                [{ app: 'a' }, { app: 'b' }],
+                1,
+                undefined,
+                (last) => ({ appUid: last.app }),
+            );
+            expect(decodeCursor(page.cursor)).toEqual({ appUid: 'a' });
+        });
+    });
 });

src/backend/util/pagination.ts

@@ -190,3 +190,52 @@ export const normalizeOffset = (
     }
     return floored;
 };
+
+// -- Keyset pages -----------------------------------------------------
+
+/**
+ * The id a keyset page resumes from, or null on the first page. A cursor that
+ * opens without a usable id (another listing's, say) throws when `strict`, and
+ * otherwise reads as the first page.
+ */
+export const openIdCursor = (
+    cursor: string | Record<string, unknown> | null | undefined,
+    secret: string | undefined,
+    {
+        label = 'cursor',
+        strict = false,
+    }: { label?: string; strict?: boolean } = {},
+): number | null => {
+    const decoded = openCursor(cursor, secret, label);
+    if (decoded === undefined) return null;
+    const id = Number(decoded.id);
+    if (Number.isSafeInteger(id) && id >= 0) return id;
+    if (strict) {
+        throw new HttpError(400, `invalid ${label}`, {
+            legacyCode: 'bad_request',
+        });
+    }
+    return null;
+};
+
+/**
+ * One page of `rows`, fetched with `LIMIT limit + 1`: the first `limit` rows,
+ * and a cursor at the last of them when the extra row says another page
+ * follows. `position` is the cursor payload; it defaults to the row's `id`.
+ * Without `secret` the cursor is a plain `encodeCursor` one.
+ */
+export const keysetPage = <R>(
+    rows: R[],
+    limit: number,
+    secret: string | undefined,
+    position: (last: R) => Record<string, unknown> = (last) => ({
+        id: Number((last as { id?: unknown }).id),
+    }),
+): { rows: R[]; cursor?: string } => {
+    if (rows.length <= limit) return { rows };
+    const page = rows.slice(0, limit);
+    return {
+        rows: page,
+        cursor: sealCursor(position(page[page.length - 1]!), secret),
+    };
+};

src/backend/util/redisGeneration.test.ts

@@ -0,0 +1,68 @@
+/*
+ * Copyright (C) 2024-present Puter Technologies Inc.
+ *
+ * This file is part of Puter.
+ *
+ * Puter is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published
+ * by the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+import MockRedis from 'ioredis-mock';
+import type { Cluster } from 'ioredis';
+import { describe, expect, it, vi } from 'vitest';
+import { bumpGeneration } from './redisGeneration';
+
+const redis = new MockRedis() as unknown as Cluster;
+const freshKey = () => `gen-test:${Math.random().toString(36).slice(2)}`;
+
+describe('bumpGeneration', () => {
+    it('increments from zero and renews the TTL each time', async () => {
+        const key = freshKey();
+        expect(await bumpGeneration(redis, key, 60)).toBe(1);
+        await redis.expire(key, 5);
+        expect(await bumpGeneration(redis, key, 60)).toBe(2);
+        expect(await redis.ttl(key)).toBeGreaterThan(5);
+    });
+
+    it('sends the increment and the expiry in one round trip', async () => {
+        const pipeline = vi.spyOn(redis, 'pipeline');
+        const incr = vi.spyOn(redis, 'incr');
+        try {
+            await bumpGeneration(redis, freshKey(), 60);
+            expect(pipeline).toHaveBeenCalledTimes(1);
+            expect(incr).not.toHaveBeenCalled();
+        } finally {
+            pipeline.mockRestore();
+            incr.mockRestore();
+        }
+    });
+
+    it('throws the error redis reports for either command', async () => {
+        const failing = {
+            pipeline: () => {
+                const chain = {
+                    incr: () => chain,
+                    expire: () => chain,
+                    exec: async () => [
+                        [new Error('WRONGTYPE not an integer'), null],
+                        [null, 1],
+                    ],
+                };
+                return chain;
+            },
+        } as unknown as Cluster;
+        await expect(bumpGeneration(failing, freshKey(), 60)).rejects.toThrow(
+            'WRONGTYPE',
+        );
+    });
+});

src/backend/util/redisGeneration.ts

@@ -0,0 +1,41 @@
+/*
+ * Copyright (C) 2024-present Puter Technologies Inc.
+ *
+ * This file is part of Puter.
+ *
+ * Puter is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published
+ * by the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+import type { Cluster } from 'ioredis';
+
+/**
+ * Increment a generation counter and renew its TTL in one round trip; the two
+ * commands travel together, so the counter can't be left without an expiry.
+ * Returns the new value.
+ */
+export const bumpGeneration = async (
+    redis: Pick<Cluster, 'pipeline'>,
+    key: string,
+    ttlSeconds: number,
+): Promise<number> => {
+    const results = await redis
+        .pipeline()
+        .incr(key)
+        .expire(key, ttlSeconds)
+        .exec();
+    for (const [error] of results ?? []) {
+        if (error) throw error;
+    }
+    return Number(results?.[0]?.[1]);
+};

src/backend/util/sqlLike.test.ts

@@ -0,0 +1,32 @@
+/*
+ * Copyright (C) 2024-present Puter Technologies Inc.
+ *
+ * This file is part of Puter.
+ *
+ * Puter is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published
+ * by the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+import { describe, expect, it } from 'vitest';
+import { escapeLike } from './sqlLike';
+
+describe('escapeLike', () => {
+    it('escapes LIKE wildcards and the escape character itself', () => {
+        expect(escapeLike('50%_off!')).toBe('50!%!_off!!');
+    });
+
+    it('leaves everything else alone', () => {
+        expect(escapeLike('workers.puter.')).toBe('workers.puter.');
+        expect(escapeLike('/user/a b\\c')).toBe('/user/a b\\c');
+    });
+});

src/backend/util/sqlLike.ts

@@ -0,0 +1,26 @@
+/*
+ * Copyright (C) 2024-present Puter Technologies Inc.
+ *
+ * This file is part of Puter.
+ *
+ * Puter is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published
+ * by the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+/**
+ * `value` as literal text inside a LIKE pattern, for use with `ESCAPE '!'`: `%`
+ * and `_` would otherwise widen the match. `!` rather than a backslash, which
+ * MySQL also reads as an escape inside the string literal itself.
+ */
+export const escapeLike = (value: string): string =>
+    value.replace(/[!%_]/g, '!$&');