fix(kv): KV share lifecycle hardening (#4429)

* fix(kv): harden the KV share lifecycle

- Retire a deleted app's live shares before an origin app is registered
  again under the same uid. The retire runs before the app row exists and
  throws on failure, so a failed launch leaves no row and the next one
  retries.
- Record deleted-app retires and uninstall cleanups (retire, then purge) as
  reserved items before they run. A lease keeps one node on a job; a failed
  run raises an alarm and backs off, and a run that hits its time budget or
  is cut short by a restart resumes on the next check. Each job is bounded
  to handles made before it was recorded.
- Uninstalling with deleteData records the job and returns; the retire and
  purge run off the request. The purge checks whether the app was
  authorized again before deleting each page, so data written after that
  is kept.
- Take a per-region lock when minting and re-check for a live handle under
  it, so concurrent shares of one region make one handle.
- Retire the handles an account holds before the account is deleted, so the
  owners' subscriptions through them end instead of staying active.

* fix(events): bound share-handle session rows by the per-connection cap

The count across a holder's connections also counted connections that
had dropped but weren't reaped yet (a crashed node, a deploy, a network
blip the server hasn't noticed), so resubscribes after a reconnect could
be refused with `events_subscription_limit`. Rows made through a share
handle now count toward the 50-per-connection cap like any other, and
with the open-connections cap per account that bounds what one holder
can put in the owner's keyspace.

The handle index stays for ending rows on retire. Rows are added to it
with the row itself and taken out when the row is dropped, and members
whose row is gone are cleared when a retire reads it.

* fix(events): keep a retire's durable settle running if ending sessions fails

Ending a retired handle's session subscriptions reads Redis. A failure
there is now logged and the durable suspension that follows still runs,
so those rows' backlogs are still purged.

* perf(events): ask once per handle when a refresh screens rows, and share handle reads

A socket refresh asks whether a share handle still stands once per
handle, holder and app, rather than once per row: rows through one
handle share its grant. Concurrent cache misses on one handle share a
single read; a forget drops the shared read so later callers start a
fresh one.

* fix(permissions): chunk the exact-match subtree delete

The holder-scoped subtree delete names every matched grant, so a large
subtree could exceed an engine's limit on bound parameters. It now
deletes 500 grants per statement.

* fix(kv): give each holder a share of a namespace's handle budget

One holder could spend a namespace's whole handle budget and starve the
others. Each holder now gets at most 3,750 of the namespace's 15,000
units per 10 seconds, and the owner's own calls through a handle spend
none. The holder's budget is checked before the grant lookups, so
throttled retries cost none; the namespace's is only told to callers the
grant admits. Multi-unit charges use a new counted claim that keeps two
counters per key instead of a sorted-set member per unit.

* fix(kv): refuse handles older than their app and claim cleanup jobs globally

- A share handle created before its app row reads as gone: in the KV
  driver, in the handle cache, on the events subscribe path, and in the
  delegate implicator. An origin app registered again under a deleted
  app's uid can't reach its predecessor's shares, whatever created the row.
  The app-token launch no longer retires inline; it records a background
  retire and never waits on it or fails because of it. The clock-skew rule
  is shared with app sessions.
- Cleanup job records now carry their lease, attempts and retry time, and
  a node claims a job by compare-and-set on its record. A job stays with
  the deployment that recorded or last ran it, and another one adopts it
  only after it has gone unclaimed well past due.
- The job check pages through records from where the last look stopped, so
  waiting records don't hide due ones. After 10 failed runs a job stops
  retrying with a warning alarm, and its record expires after 30 days.
- A mint that waits out the region lock looks for the share again and
  returns it, or goes ahead, instead of failing with too_many_requests.
- The record-failure alarm dedups; an unfinished run is a log line.

* fix(kv): hide shares older than their app from re-shares and listings

After an app is registered again under a deleted app's uid, its
predecessor's live handles stayed visible: sharing the same region again
handed back the old, dead handle, and both sides' listings showed the old
handles and labels. findLive and both listing queries now leave out a live
handle created before its app row, the same rule handle resolution uses,
so a re-share makes a new handle and the listings, totals included, skip
the old ones until their retire lands.

A new share also retires, grants and all, the handles its owner gave the
same holder for the earlier app first, so withdrawing those later can't
take the new grant with it.

* fix(events): keep a loose cap on subscriptions through one share handle

Without a count across connections, a holder could put up to 20,000 rows
in the owner's keyspace (50 per connection, 400 connections), each one
evaluated on every owner write. The cap is back at 1,000 per holder per
handle, far above what connections that dropped without being reaped
leave behind. The index is only read row by row once it's past the cap.

* fix(kv): harden the share cleanup jobs

- Job records are keyed by domain, so a deployment that shares the table
  but not the database or broadcast (staging beside production) never
  runs another's jobs.
- A job whose record can't be written runs unrecorded instead of not at
  all, and a failed check for live handles records the job anyway so it
  retries. Failures that used to be swallowed are logged.
- Recording a job again keeps its attempts and backoff; one that gave up
  gets one more attempt.
- A system retire gives back the grant of a live share it doesn't cover
  inside a withdrawn region, and a share made under a reused uid grants
  again once its row stands, so a concurrent retire can't leave a live
  handle without its grant.

* fix(kv): don't cache a live handle a retire overtook during its app lookup

The live read decided whether it could cache before awaiting the app
lookup, so a retire's forget that landed during that lookup was
overwritten with the handle as live for up to 60 s. It's now checked
after the lookup.

* chore(kv): word the job key comment for any install

src/backend/clients/database/DatabaseClient.test.ts

@@ -275,6 +275,16 @@ describe('AbstractDatabaseClient — SQL generation', () => {
         expect(postgres().timestampFromUnixSql()).toBe('to_timestamp(?)');
     });
 
+    it('reads a timestamp column back as unix seconds in each engine', () => {
+        expect(sqlite().unixSecondsSql('`t`')).toBe(
+            "CAST(strftime('%s', `t`) AS INTEGER)",
+        );
+        expect(mysql().unixSecondsSql('`t`')).toBe('UNIX_TIMESTAMP(`t`)');
+        expect(postgres().unixSecondsSql('`t`')).toBe(
+            'EXTRACT(EPOCH FROM `t`)::bigint',
+        );
+    });
+
     it('describes a stored timestamp the way each driver reads it back', () => {
         const seconds = Date.UTC(2026, 9, 11, 8, 5, 9) / 1000;
         expect(sqlite().timestampValue(seconds)).toBe('2026-10-11 08:05:09');

src/backend/clients/database/DatabaseClient.ts

@@ -388,6 +388,19 @@ export class AbstractDatabaseClient extends PuterClient {
         });
     }
 
+    /**
+     * A timestamp column as unix seconds, computed by the database: the mysql
+     * driver reads a stored UTC timestamp as local time, so parsing it in JS
+     * can be off by the server's UTC offset.
+     */
+    unixSecondsSql(column: string): string {
+        return this.case({
+            postgres: `EXTRACT(EPOCH FROM ${column})::bigint`,
+            mysql: `UNIX_TIMESTAMP(${column})`,
+            otherwise: `CAST(strftime('%s', ${column}) AS INTEGER)`,
+        });
+    }
+
     /**
      * What a read of a timestamp column holding `seconds` returns: sqlite gives
      * back the UTC text it stored, mysql2 and pg a `Date` read in node's time

src/backend/controllers/auth/AuthController.ts

@@ -3836,9 +3836,10 @@ export class AuthController extends PuterController {
             // An origin's uid is a deterministic uuidv5, so a deleted app
             // reappears here under the identical uid. Withdraw any cross-app
             // data grants left pointing at it before this new row can inherit
-            // consent the user gave its predecessor. Only *this* path can reuse
-            // a uid: `AppStore.create` mints a random uuid4, which no deleted
-            // app can ever hold again.
+            // consent the user gave its predecessor. Only `createFromOrigin`
+            // reuses a uid, here or wherever else an origin app is bootstrapped:
+            // `AppStore.create` mints a random uuid4, which no deleted app can
+            // ever hold again.
             //
             // Called directly rather than through `app.changed`: the token is
             // issued below, so this has to be able to stop that, and
@@ -3849,6 +3850,9 @@ export class AuthController extends PuterController {
                 app_uid,
                 'uid reused by a new app',
             );
+            // Key-value shares the predecessor left read as gone already,
+            // being older than this row; this only cleans them up.
+            this.services.kvShare.retireSharesOlderThanApp(app);
         }
         if (!app) {
             throw new HttpError(404, `App ${app_uid} does not exist`, {

src/backend/drivers/kv/KVStoreDriver.ts

@@ -57,6 +57,7 @@ import {
     sealCursor,
     unsealCursor,
 } from '../../util/pagination.js';
+import { predatesApp } from '../../util/appGeneration.js';
 import { KV_COSTS } from './costs.js';
 import {
     KV_HANDLE_NAMESPACE_BUDGET,
@@ -507,11 +508,15 @@ export class KVStoreDriver extends PuterDriver {
         )
             throw handleBudgetSpent();
 
-        // Side by side, so the app lookup costs no round trip of its own.
+        // Side by side, so the app lookup costs no round trip of its own. A
+        // handle older than its app was made for an earlier app under the
+        // same uid, and is gone with it.
         const [appGone, allowed] = await Promise.all([
             row.appUid === KV_GLOBAL_APP_KEY
                 ? false
-                : this.stores.app.getByUid(row.appUid).then((app) => !app),
+                : this.stores.app
+                      .getByUid(row.appUid)
+                      .then((app) => !app || predatesApp(row.createdAt, app)),
             this.services.permission.check(
                 actor,
                 kvSharePermission(ownerUuid, row.appUid, row.keyPrefix, level),

src/backend/services/auth/AuthService.ts

@@ -28,6 +28,7 @@ import {
 import { HttpError } from '../../core/http/HttpError.js';
 import { WEB_AND_EXTENSION_PROTOCOLS } from '../../util/validation.js';
 import { awaitLater } from '../../util/concurrency.js';
+import { appGenerationStart } from '../../util/appGeneration.js';
 import {
     ASSET_WINDOW_SECONDS,
     GODMODE_TOKEN_MAX_AGE_SECONDS,
@@ -57,9 +58,6 @@ import type {
 const APP_ORIGIN_UUID_NAMESPACE = '33de3768-8ee0-43e9-9e73-db192b97a5d8';
 // Successor uids an origin can derive after its earlier apps were repointed.
 const MAX_ORIGIN_UID_GENERATIONS = 8;
-// Allowed gap between a session's creation time (server clock) and its app's
-// (database clock) before the session counts as older than the app.
-const APP_SESSION_CLOCK_SKEW_SECONDS = 60;
 
 // Revoked and expired sessions are kept this long, so a stale token still gets
 // `session_revoked`/`session_expired` rather than `invalid`, then deleted. Web
@@ -562,7 +560,7 @@ export class AuthService extends PuterService {
                 appUid,
                 workerName,
                 auth_id,
-                notBefore: this.#appSessionsNotBefore(app),
+                notBefore: appGenerationStart(app),
             },
         );
         if (!session) {
@@ -1808,7 +1806,7 @@ export class AuthService extends PuterService {
             appUid,
             {
                 auth_id: this.#authIdFor(actor.user as UserRow),
-                notBefore: this.#appSessionsNotBefore(app),
+                notBefore: appGenerationStart(app),
             },
         );
 
@@ -2685,7 +2683,7 @@ export class AuthService extends PuterService {
         // uid returns with the next app on that origin. A session older than
         // the app was made for the earlier one. Checked before `touch()` so a
         // rejected token doesn't slide the old session's expiry.
-        const notBefore = this.#appSessionsNotBefore(app);
+        const notBefore = appGenerationStart(app);
         const createdAt = Number(session.created_at);
         if (notBefore !== null && createdAt > 0 && createdAt < notBefore) {
             return { reauth: { reason: 'session_revoked' } };
@@ -2714,19 +2712,6 @@ export class AuthService extends PuterService {
         return block.blocked ? { reason: block.reason } : null;
     }
 
-    /**
-     * Unix seconds before which an app or worker session for `app` belongs to
-     * an earlier app with the same uid; null when the creation time is
-     * unknown.
-     */
-    #appSessionsNotBefore(
-        app: { created_epoch?: unknown } | null | undefined,
-    ): number | null {
-        const created = Number(app?.created_epoch);
-        if (!Number.isFinite(created) || created <= 0) return null;
-        return created - APP_SESSION_CLOCK_SKEW_SECONDS;
-    }
-
     #createdBefore(row: SessionRow | null, notBefore: number | null): boolean {
         if (notBefore === null || !row) return false;
         const createdAt = Number(row.created_at);
@@ -2802,7 +2787,7 @@ export class AuthService extends PuterService {
             if (blocked) return { blocked };
             // As with app sessions: a token minted under an earlier app that
             // held this uid doesn't authenticate as the current one.
-            const notBefore = this.#appSessionsNotBefore(app);
+            const notBefore = appGenerationStart(app);
             if (
                 this.#createdBefore(session, notBefore) ||
                 this.#createdBefore(parentSession, notBefore)

src/backend/services/events/EventsService.test.ts

@@ -205,6 +205,7 @@ let handles: Map<string, KvShareHandle>;
 
 const kvShareHandleStore = {
     getByHandle: async (handle: string) => handles.get(handle) ?? null,
+    predatesItsApp: async () => false,
     getLive: async (handle: string) => {
         const row = handles.get(handle);
         if (!row || row.revokedAt !== null) return null;

src/backend/services/events/EventsService.ts

@@ -3173,7 +3173,11 @@ export class EventsService extends PuterService {
         compileMatch(key, { separator: KV_MATCH_SEPARATOR });
 
         const share = await this.stores.kvShareHandle.getByHandle(handle);
-        if (!share || share.revokedAt !== null)
+        if (
+            !share ||
+            share.revokedAt !== null ||
+            (await this.stores.kvShareHandle.predatesItsApp(share))
+        )
             throw unknownKvShareHandle(handle);
         if (
             actor.user?.id !== share.granteeUserId &&

src/backend/services/kvShare/kvShareLevels.integration.test.ts

@@ -1123,7 +1123,16 @@ describe('an app deletion that fails partway', () => {
         }
         expect((await handles().getByHandle(handle))?.revokedAt).toBeNull();
 
-        await deleted();
+        // The next check after its backoff retries it.
+        const now = Date.now.bind(Date);
+        const clock = vi
+            .spyOn(Date, 'now')
+            .mockImplementation(() => now() + 2 * 60_000);
+        try {
+            await shares().resumeKvShareJobs();
+        } finally {
+            clock.mockRestore();
+        }
         expect(await grantRows(guest, prefix, app.uid)).toEqual([]);
         expect(await handles().getByHandle(handle)).toMatchObject({
             revokedBy: 'system',

src/backend/services/kvShare/kvShareLifecycle.integration.test.ts

@@ -0,0 +1,833 @@
+/*
+ * Copyright (C) 2024-present Puter Technologies Inc.
+ *
+ * This file is part of Puter.
+ *
+ * Puter is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published
+ * by the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+/**
+ * What happens to shares when what they hang off goes away: an app deleted and
+ * registered again under the same uid, a deletion whose retire fails or runs
+ * long, a holder's account deleted, and racing shares of one region.
+ */
+
+import { v4 as uuidv4 } from 'uuid';
+import {
+    afterAll,
+    beforeAll,
+    beforeEach,
+    describe,
+    expect,
+    it,
+    vi,
+} from 'vitest';
+import { EVENTS_KV_HANDLE_LIMIT } from '../../controllers/events/limits.js';
+import { makeActor, type Actor } from '../../core/actor.js';
+import { runWithContext } from '../../core/context.js';
+import { KV_HANDLE_RETIRE_BATCH } from '../../stores/events/KvShareHandleStore.js';
+import {
+    createTestUser,
+    setupPuterTestEnv,
+    type PuterTestEnv,
+} from '../../testUtil.js';
+import type { IConfig } from '../../types.js';
+import { kvSharePermission, type KvShareAccess } from './kvShares.js';
+import { seedKvShareHandles } from './kvShareTestUtil.js';
+
+const BOOT_TIMEOUT_MS = 120_000;
+
+interface TestUser {
+    actor: Actor;
+    username: string;
+    id: number;
+    uuid: string;
+}
+
+let env: PuterTestEnv;
+let owner: TestUser;
+let guest: TestUser;
+
+const shares = () => env.server.services.kvShare;
+const handles = () => env.server.stores.kvShareHandle;
+
+const userFor = async (username: string): Promise<TestUser> => {
+    const row = await env.server.stores.user.getByUsername(username);
+    return {
+        actor: makeActor({ user: row as never }),
+        username,
+        id: row!.id,
+        uuid: row!.uuid as string,
+    };
+};
+
+const newUser = async (label: string): Promise<TestUser> => {
+    const username = `${label}-${uuidv4().slice(0, 8)}`;
+    await createTestUser(env.server, { username, password: 'pw-test-1234' });
+    return userFor(username);
+};
+
+const region = (label = 'r') => `${label}-${uuidv4().slice(0, 8)}:`;
+
+const share = (
+    prefix: string,
+    access: KvShareAccess,
+    opts: { holder?: TestUser; appUid?: string } = {},
+) =>
+    shares().mintKvHandle(
+        owner.actor,
+        {
+            prefix,
+            recipient: { username: (opts.holder ?? guest).username },
+            access,
+            ...(opts.appUid ? { appUid: opts.appUid } : {}),
+        },
+        { defaultAccess: 'read' },
+    );
+
+const makeApp = async () => {
+    const name = `kv-life-${uuidv4().slice(0, 8)}`;
+    return env.server.stores.app.create(
+        {
+            name,
+            title: 'Lifecycle App',
+            index_url: `https://${name}.example.test/index.html`,
+        },
+        { ownerUserId: owner.id },
+    );
+};
+
+/** An app deleted the way the app driver does it: the row, then the event. */
+const deleteApp = async (app: { id: number; uid: string }) => {
+    await env.server.stores.app.delete(app.id);
+    await env.server.clients.event.emitAndWait(
+        'app.changed',
+        { app_uid: app.uid, app: null, old_app: app, action: 'deleted' },
+        {},
+    );
+};
+
+const liveInApp = async (appUid: string) =>
+    (await handles().listLiveForApps([appUid], 5_000)).length;
+
+const guestGet = (handle: string, key: string, actor = guest.actor) =>
+    runWithContext({ actor }, () =>
+        env.server.drivers.kvStore.get({ key, optConfig: { handle } }),
+    );
+
+const ownerWrites = (key: string, value: unknown, appUid: string) =>
+    runWithContext({ actor: owner.actor }, () =>
+        env.server.drivers.kvStore.set({
+            key,
+            value,
+            optConfig: { appUuid: appUid },
+        }),
+    );
+
+/** Make a handle look minted `seconds` ago, past the clock-skew allowance. */
+const age = (handle: string, seconds = 2 * 60 * 60) =>
+    env.server.clients.db.write(
+        'UPDATE `kv_share_handles` SET `created_at` = `created_at` - ? WHERE `handle` = ?',
+        [seconds, handle],
+    );
+
+/** The job record for one deleted app, as every node of this domain reads it. */
+const appJobKey = (appUid: string, domain = new URL(env.origin).hostname) =>
+    `kvShareJob#${domain}#app#${appUid}`;
+
+const suspendedReason = async (subId: string) => {
+    const [row] = await env.server.clients.db.pread(
+        'SELECT `suspended_reason` FROM `event_subscriptions` WHERE `sub_id` = ?',
+        [subId],
+    );
+    return (row as { suspended_reason: unknown } | undefined)?.suspended_reason;
+};
+
+/** Run `fn` with the clock `ms` ahead, past any retry wait shorter than that. */
+const later = async <T>(ms: number, fn: () => Promise<T>): Promise<T> => {
+    const now = Date.now.bind(Date);
+    const clock = vi.spyOn(Date, 'now').mockImplementation(() => now() + ms);
+    try {
+        return await fn();
+    } finally {
+        clock.mockRestore();
+    }
+};
+
+/** Check for recorded jobs until `check` passes. */
+const resumeUntil = (check: () => Promise<void>) =>
+    vi.waitFor(
+        async () => {
+            await shares().resumeKvShareJobs();
+            await check();
+        },
+        { timeout: 4_000, interval: 100 },
+    );
+
+beforeAll(async () => {
+    env = await setupPuterTestEnv({
+        events: { enabled: true, kvHandles: true },
+        kv: { shareHandles: true },
+        unlimitedMetering: true,
+    } as IConfig);
+    owner = await userFor(env.users.user.username);
+    guest = await userFor(env.users.other.username);
+}, BOOT_TIMEOUT_MS);
+
+afterAll(async () => {
+    await env?.shutdown();
+});
+
+beforeEach(async () => {
+    for (const user of [owner, guest])
+        await env.server.clients.redis.del(
+            `rate:${EVENTS_KV_HANDLE_LIMIT.scope}:${user.id}`,
+        );
+});
+
+describe('an origin app registered again under a deleted app`s uid', () => {
+    const call = (body: unknown) =>
+        fetch(new URL('/auth/get-user-app-token', env.apiOrigin), {
+            method: 'POST',
+            headers: {
+                authorization: `Bearer ${env.users.user.token}`,
+                'content-type': 'application/json',
+            },
+            body: JSON.stringify(body),
+        });
+
+    const originFor = (label: string) =>
+        `https://${label}-${uuidv4().slice(0, 8)}.example.com`;
+
+    /**
+     * An origin app with an old share in it, deleted without its retire
+     * running, so the handle and its grant are still live.
+     */
+    const deletedWithShare = async () => {
+        const origin = originFor('kv-reuse');
+        const appUid = await env.server.services.auth.appUidFromOrigin(origin);
+        const app = await env.server.stores.app.createFromOrigin(
+            appUid,
+            origin,
+        );
+        const prefix = region('reuse');
+        const { handle } = await share(prefix, 'read', { appUid });
+        await age(handle);
+        await env.server.stores.app.delete(app.id);
+        return { origin, appUid, prefix, handle };
+    };
+
+    it('refuses a handle older than the app, through the driver and the delegate, with nothing retired', async () => {
+        const { origin, appUid, prefix, handle } = await deletedWithShare();
+        // Registered again the way any bootstrap does, with no retire.
+        const app = await env.server.stores.app.createFromOrigin(
+            appUid,
+            origin,
+        );
+        await env.server.services.permission.grantUserAppPermission(
+            guest.actor,
+            appUid,
+            'flag:app-is-authenticated',
+        );
+        const guestInApp = makeActor({
+            user: guest.actor.user as never,
+            app: { uid: appUid, id: app.id },
+        });
+
+        for (const actor of [guest.actor, guestInApp])
+            await expect(guestGet(handle, 'k', actor)).rejects.toMatchObject({
+                legacyCode: 'kv_handle_not_found',
+            });
+        expect(
+            await env.server.services.permission.check(
+                guestInApp,
+                kvSharePermission(owner.uuid, appUid, prefix, 'read'),
+            ),
+        ).toBe(false);
+        expect((await handles().getByHandle(handle))?.revokedAt).toBeNull();
+
+        // A share made for the new app works as usual.
+        const fresh = await share(region('fresh'), 'read', { appUid });
+        expect(await guestGet(fresh.handle, 'k', guestInApp)).toBeNull();
+        expect(
+            await env.server.services.permission.check(
+                guestInApp,
+                kvSharePermission(owner.uuid, appUid, fresh.prefix, 'read'),
+            ),
+        ).toBe(true);
+    });
+
+    it('makes a new share when the owner shares an old region again', async () => {
+        const { origin, appUid, prefix, handle } = await deletedWithShare();
+        const app = await env.server.stores.app.createFromOrigin(
+            appUid,
+            origin,
+        );
+
+        const again = await share(prefix, 'read', { appUid });
+
+        expect(again.handle).not.toBe(handle);
+        expect(await handles().getByHandle(handle)).toMatchObject({
+            revokedBy: 'system',
+        });
+        await ownerWrites(`${prefix}k`, 'v', appUid);
+        expect(await guestGet(again.handle, 'k')).toBe('v');
+
+        // The cleanup a launch starts leaves the new share's grant alone.
+        shares().retireSharesOlderThanApp(app);
+        await new Promise((resolve) => setTimeout(resolve, 300));
+        expect(await guestGet(again.handle, 'k')).toBe('v');
+    });
+
+    it('leaves the old shares out of both sides` listings', async () => {
+        const { origin, appUid, handle } = await deletedWithShare();
+        const app = await env.server.stores.app.createFromOrigin(
+            appUid,
+            origin,
+        );
+        const guestInApp = makeActor({
+            user: guest.actor.user as never,
+            app: { uid: appUid, id: app.id },
+        });
+
+        for (const actor of [guest.actor, guestInApp]) {
+            const held = await shares().listKvSharesWithMe(actor, {
+                appUid,
+                includeTotal: true,
+            });
+            expect(held.items).toEqual([]);
+            expect(held.total).toBe(0);
+        }
+        for (const state of ['all', 'live'] as const) {
+            const mine = await shares().listKvHandles(owner.actor, {
+                appUid,
+                state,
+                includeTotal: true,
+            });
+            expect(mine.items).toEqual([]);
+            expect(mine.total).toBe(0);
+        }
+
+        // Once the cleanup retires it, the owner sees when it stopped.
+        shares().retireSharesOlderThanApp(app);
+        await vi.waitFor(async () =>
+            expect(
+                (await shares().listKvHandles(owner.actor, { appUid })).items,
+            ).toEqual([
+                expect.objectContaining({ handle, revokedBy: 'system' }),
+            ]),
+        );
+    });
+
+    it('launches without waiting on the retire, which cleans up in the background', async () => {
+        const { origin, appUid, handle } = await deletedWithShare();
+
+        let release!: () => void;
+        const held = new Promise<void>((resolve) => {
+            release = resolve;
+        });
+        const store = handles();
+        const retireMany = store.retireMany.bind(store);
+        const stalled = vi
+            .spyOn(store, 'retireMany')
+            .mockImplementation(async (...args) => {
+                await held;
+                return retireMany(...args);
+            });
+        try {
+            const res = await call({ origin });
+            expect(res.status).toBe(200);
+            expect(((await res.json()) as { app_uid: string }).app_uid).toBe(
+                appUid,
+            );
+            await expect(guestGet(handle, 'k')).rejects.toMatchObject({
+                legacyCode: 'kv_handle_not_found',
+            });
+        } finally {
+            release();
+            stalled.mockRestore();
+        }
+        await vi.waitFor(async () =>
+            expect(await handles().getByHandle(handle)).toMatchObject({
+                revokedBy: 'system',
+            }),
+        );
+    });
+
+    it('launches when the retire fails', async () => {
+        const { origin, handle } = await deletedWithShare();
+
+        const failing = vi
+            .spyOn(
+                env.server.services.permission,
+                'revokeUserUserPermissionSubtree',
+            )
+            .mockRejectedValue(new Error('database went away'));
+        try {
+            expect((await call({ origin })).status).toBe(200);
+            await expect(guestGet(handle, 'k')).rejects.toMatchObject({
+                legacyCode: 'kv_handle_not_found',
+            });
+        } finally {
+            failing.mockRestore();
+        }
+    });
+});
+
+describe('a deleted app`s retire', () => {
+    it('alarms when it fails, and the next check retries it from its record', async () => {
+        const app = await makeApp();
+        const { handle } = await share(region('retry'), 'read', {
+            appUid: app.uid,
+        });
+
+        const alarm = vi.spyOn(env.server.clients.alarm, 'create');
+        const failing = vi
+            .spyOn(
+                env.server.services.permission,
+                'revokeUserUserPermissionSubtree',
+            )
+            .mockRejectedValueOnce(new Error('database went away'));
+        try {
+            await deleteApp(app);
+            expect(alarm).toHaveBeenCalledWith(
+                expect.stringContaining(app.uid),
+                expect.any(String),
+                expect.objectContaining({ error: expect.any(Error) }),
+                'warning',
+                { dedup: true },
+            );
+        } finally {
+            failing.mockRestore();
+            alarm.mockRestore();
+        }
+        expect((await handles().getByHandle(handle))?.revokedAt).toBeNull();
+
+        // Backing off after the failure, so an immediate check leaves it.
+        await shares().resumeKvShareJobs();
+        expect((await handles().getByHandle(handle))?.revokedAt).toBeNull();
+
+        await later(2 * 60_000, () =>
+            resumeUntil(async () =>
+                expect(await handles().getByHandle(handle)).toMatchObject({
+                    revokedBy: 'system',
+                }),
+            ),
+        );
+    });
+
+    it('runs anyway, with an alarm, when its record can`t be written', async () => {
+        const app = await makeApp();
+        const { handle } = await share(region('unrecorded'), 'read', {
+            appUid: app.uid,
+        });
+        const alarm = vi.spyOn(env.server.clients.alarm, 'create');
+        const unwritable = vi
+            .spyOn(env.server.stores.kv, 'updateReservedItemIf')
+            .mockRejectedValueOnce(new Error('throttled'));
+        try {
+            await deleteApp(app);
+            expect(alarm).toHaveBeenCalledWith(
+                `kv_share_job_record_failed:${appJobKey(app.uid)}`,
+                expect.any(String),
+                expect.objectContaining({ error: expect.any(Error) }),
+                'warning',
+                { dedup: true },
+            );
+        } finally {
+            unwritable.mockRestore();
+            alarm.mockRestore();
+        }
+        expect(await handles().getByHandle(handle)).toMatchObject({
+            revokedBy: 'system',
+        });
+    });
+
+    it('keeps its backoff when it`s recorded again', async () => {
+        const app = await makeApp();
+        const { handle } = await share(region('backoff'), 'read', {
+            appUid: app.uid,
+        });
+        const failing = vi
+            .spyOn(
+                env.server.services.permission,
+                'revokeUserUserPermissionSubtree',
+            )
+            .mockRejectedValueOnce(new Error('database went away'));
+        try {
+            await deleteApp(app);
+        } finally {
+            failing.mockRestore();
+        }
+
+        // What a periodic sweep of deleted apps does.
+        await shares().retireForDeletedApps([app.uid]);
+        expect((await handles().getByHandle(handle))?.revokedAt).toBeNull();
+
+        await later(2 * 60_000, () =>
+            resumeUntil(async () =>
+                expect(await handles().getByHandle(handle)).toMatchObject({
+                    revokedBy: 'system',
+                }),
+            ),
+        );
+    });
+
+    it('never runs a record written for another domain', async () => {
+        const app = await makeApp();
+        const { handle } = await share(region('other-domain'), 'read', {
+            appUid: app.uid,
+        });
+        await env.server.stores.app.delete(app.id);
+        const key = appJobKey(app.uid, 'staging.example.test');
+        await env.server.stores.kv.putReservedItem(key, {
+            kind: 'app',
+            appUid: app.uid,
+            cutoff: Math.floor(Date.now() / 1000),
+            state: 'pending',
+            attempts: 0,
+            runAfter: 0,
+            home: env.server.services.broadcast.regionId,
+            touched: 0,
+            leaseUntil: 0,
+        });
+        try {
+            await later(60 * 60_000, async () => {
+                for (let look = 0; look < 3; look++)
+                    await shares().resumeKvShareJobs();
+            });
+            expect((await handles().getByHandle(handle))?.revokedAt).toBeNull();
+        } finally {
+            await env.server.stores.kv.deleteReservedItemIf(
+                key,
+                'attribute_exists(#c)',
+                {},
+                { '#c': 'cutoff' },
+            );
+        }
+    });
+
+    it('gives a share made meanwhile inside a retired region its grant back', async () => {
+        const app = await makeApp();
+        const outer = region('outer');
+        const before = await share(outer, 'read', { appUid: app.uid });
+        // Dated after the deletion, as one made while the retire runs would be.
+        const after = await share(`${outer}inner:`, 'read', {
+            appUid: app.uid,
+        });
+        await env.server.clients.db.write(
+            'UPDATE `kv_share_handles` SET `created_at` = `created_at` + 60 WHERE `handle` = ?',
+            [after.handle],
+        );
+
+        await deleteApp(app);
+
+        expect(await handles().getByHandle(before.handle)).toMatchObject({
+            revokedBy: 'system',
+        });
+        const kept = await handles().getByHandle(after.handle);
+        expect(kept?.revokedAt).toBeNull();
+        expect(
+            await env.server.services.permission.hasUserUserGrant(
+                guest.id,
+                owner.id,
+                [kept!.permission],
+            ),
+        ).toBe(true);
+    });
+
+    it('carries on from its record when one run runs out of time', async () => {
+        const app = await makeApp();
+        // Spread over holders, as the per-holder cap would spread them.
+        const holders = await Promise.all(
+            [0, 1, 2].map(() => newUser('kv-seed')),
+        );
+        const perHolder = Math.ceil((KV_HANDLE_RETIRE_BATCH + 1) / 3);
+        for (const holder of holders)
+            await seedKvShareHandles(env.server, {
+                ownerUserId: owner.id,
+                ownerUuid: owner.uuid,
+                granteeUserId: holder.id,
+                count: perHolder,
+                appUid: () => app.uid,
+            });
+        const seeded = perHolder * holders.length;
+
+        // The first batch takes the run past its deadline.
+        const store = handles();
+        const retireMany = store.retireMany.bind(store);
+        const now = Date.now.bind(Date);
+        const clock = vi.spyOn(Date, 'now');
+        const slow = vi
+            .spyOn(store, 'retireMany')
+            .mockImplementationOnce(async (...args) => {
+                await retireMany(...args);
+                clock.mockImplementation(() => now() + 2 * 60_000);
+            });
+        try {
+            await deleteApp(app);
+        } finally {
+            slow.mockRestore();
+            clock.mockRestore();
+        }
+        expect(await liveInApp(app.uid)).toBe(seeded - KV_HANDLE_RETIRE_BATCH);
+
+        await resumeUntil(async () => expect(await liveInApp(app.uid)).toBe(0));
+    });
+
+    it('leaves alone shares made after the deletion under the same uid', async () => {
+        const app = await makeApp();
+        const before = await share(region('old'), 'read', { appUid: app.uid });
+        // Dated after the deletion, as one made by an app registered again
+        // under the uid would be.
+        const after = await share(region('new'), 'read', { appUid: app.uid });
+        await env.server.clients.db.write(
+            'UPDATE `kv_share_handles` SET `created_at` = `created_at` + 60 WHERE `handle` = ?',
+            [after.handle],
+        );
+
+        await deleteApp(app);
+
+        expect(await handles().getByHandle(before.handle)).toMatchObject({
+            revokedBy: 'system',
+        });
+        expect(
+            (await handles().getByHandle(after.handle))?.revokedAt,
+        ).toBeNull();
+    });
+
+    it('runs once when two runners race for it', async () => {
+        const app = await makeApp();
+        const { handle } = await share(region('race-job'), 'read', {
+            appUid: app.uid,
+        });
+        await env.server.stores.app.delete(app.id);
+
+        const store = handles();
+        const retireMany = vi.spyOn(store, 'retireMany');
+        try {
+            await Promise.all([
+                shares().retireForDeletedApps([app.uid]),
+                shares().retireForDeletedApps([app.uid]),
+            ]);
+            const runs = retireMany.mock.calls.filter(([named]) =>
+                named.includes(handle),
+            );
+            expect(runs).toHaveLength(1);
+        } finally {
+            retireMany.mockRestore();
+        }
+        expect(await handles().getByHandle(handle)).toMatchObject({
+            revokedBy: 'system',
+        });
+    });
+
+    it('is left to the deployment that holds it until it goes unclaimed', async () => {
+        const app = await makeApp();
+        const { handle } = await share(region('elsewhere'), 'read', {
+            appUid: app.uid,
+        });
+        const failing = vi
+            .spyOn(
+                env.server.services.permission,
+                'revokeUserUserPermissionSubtree',
+            )
+            .mockRejectedValueOnce(new Error('database went away'));
+        try {
+            await deleteApp(app);
+        } finally {
+            failing.mockRestore();
+        }
+        // Another deployment ran it last, a moment ago, and it's due now.
+        await env.server.stores.kv.updateReservedItemIf(
+            appJobKey(app.uid),
+            { set: { home: 'elsewhere', touched: Date.now(), runAfter: 0 } },
+            'attribute_exists(#c)',
+            {},
+            { '#c': 'cutoff' },
+        );
+
+        await later(5 * 60_000, async () => {
+            await shares().resumeKvShareJobs();
+            await shares().resumeKvShareJobs();
+        });
+        expect((await handles().getByHandle(handle))?.revokedAt).toBeNull();
+
+        await later(31 * 60_000, () =>
+            resumeUntil(async () =>
+                expect(await handles().getByHandle(handle)).toMatchObject({
+                    revokedBy: 'system',
+                }),
+            ),
+        );
+    });
+
+    it('stops retrying after repeated failures, with an alarm', async () => {
+        const app = await makeApp();
+        const { handle } = await share(region('gave-up'), 'read', {
+            appUid: app.uid,
+        });
+        const alarm = vi.spyOn(env.server.clients.alarm, 'create');
+        const failing = vi
+            .spyOn(
+                env.server.services.permission,
+                'revokeUserUserPermissionSubtree',
+            )
+            .mockRejectedValue(new Error('database went away'));
+        const gaveUp = () =>
+            alarm.mock.calls.some(
+                ([id]) =>
+                    id.startsWith('kv_share_job_gave_up:') &&
+                    id.includes(app.uid),
+            );
+        try {
+            await deleteApp(app);
+            // Each check comes after the longest wait between retries.
+            for (let hours = 7; hours < 7 * 20 && !gaveUp(); hours += 7)
+                await later(hours * 60 * 60_000, async () => {
+                    for (let look = 0; look < 5; look++)
+                        await shares().resumeKvShareJobs();
+                });
+            expect(gaveUp()).toBe(true);
+
+            failing.mockClear();
+            await later(1_000 * 60 * 60_000, async () => {
+                for (let look = 0; look < 5; look++)
+                    await shares().resumeKvShareJobs();
+            });
+            expect(failing).not.toHaveBeenCalled();
+        } finally {
+            failing.mockRestore();
+            alarm.mockRestore();
+        }
+        expect((await handles().getByHandle(handle))?.revokedAt).toBeNull();
+
+        // A new deletion event records it again.
+        await deleteApp(app);
+        expect(await handles().getByHandle(handle)).toMatchObject({
+            revokedBy: 'system',
+        });
+    });
+
+    it('reaches a due job behind a page of ones that are waiting', async () => {
+        const waiting: string[] = [];
+        for (let i = 0; i < 150; i++) {
+            const appUid = `app-0000-wait-${String(i).padStart(4, '0')}`;
+            waiting.push(appJobKey(appUid));
+            await env.server.stores.kv.putReservedItem(appJobKey(appUid), {
+                kind: 'app',
+                appUid,
+                cutoff: 0,
+                state: 'pending',
+                attempts: 1,
+                runAfter: Date.now() + 1_000 * 60 * 60_000,
+                home: env.server.services.broadcast.regionId,
+                touched: Date.now(),
+                leaseUntil: 0,
+            });
+        }
+        try {
+            const app = await makeApp();
+            const { handle } = await share(region('behind'), 'read', {
+                appUid: app.uid,
+            });
+            const failing = vi
+                .spyOn(
+                    env.server.services.permission,
+                    'revokeUserUserPermissionSubtree',
+                )
+                .mockRejectedValueOnce(new Error('database went away'));
+            try {
+                await deleteApp(app);
+            } finally {
+                failing.mockRestore();
+            }
+
+            await later(2 * 60_000, async () => {
+                for (let look = 0; look < 4; look++)
+                    await shares().resumeKvShareJobs();
+            });
+            expect(await handles().getByHandle(handle)).toMatchObject({
+                revokedBy: 'system',
+            });
+        } finally {
+            for (const key of waiting)
+                await env.server.stores.kv.deleteReservedItemIf(
+                    key,
+                    'attribute_exists(#c)',
+                    {},
+                    { '#c': 'cutoff' },
+                );
+        }
+    });
+});
+
+describe('shares of one region made at the same time', () => {
+    it('make one handle, and every call gets it', async () => {
+        const prefix = region('race');
+
+        const minted = await Promise.all(
+            [0, 1, 2].map(() => share(prefix, 'read')),
+        );
+
+        expect(new Set(minted.map((one) => one.handle)).size).toBe(1);
+        const live = (
+            await handles().listLiveForOwnerAndGrantee(owner.id, guest.id)
+        ).filter((row) => row.keyPrefix === prefix);
+        expect(live.map((row) => row.handle)).toEqual([minted[0].handle]);
+    });
+
+    it('hands a call that waited out the lock the share made meanwhile', async () => {
+        const prefix = region('slow');
+        const store = handles();
+        const mint = store.mint.bind(store);
+        const pause = (ms: number) =>
+            new Promise((resolve) => setTimeout(resolve, ms));
+        // The first mint inserts after the second has looked and found
+        // nothing, then holds the lock well past how long the second waits.
+        const slow = vi
+            .spyOn(store, 'mint')
+            .mockImplementationOnce(async (input) => {
+                await pause(300);
+                const row = await mint(input);
+                await pause(2_000);
+                return row;
+            });
+        try {
+            const first = share(prefix, 'read');
+            await pause(100);
+            const second = await share(prefix, 'read');
+            expect(second.handle).toBe((await first).handle);
+        } finally {
+            slow.mockRestore();
+        }
+    });
+});
+
+describe('deleting an account', () => {
+    const subscribe = (user: TestUser, handle: string) =>
+        env.server.services.events.subscribeDurable(user.actor, {
+            subject: `kv:${handle}:*`,
+        });
+
+    it('ends what the owner subscribed to through a share the account held', async () => {
+        const holder = await newUser('kv-holder');
+        const { handle } = await share(region('held'), 'read', { holder });
+        const { sub } = await subscribe(owner, handle);
+
+        await env.server.services.userAccount.cascadeDelete(holder.id);
+
+        expect(await suspendedReason(sub.subId)).toBe('permission_revoked');
+    });
+});

src/backend/services/kvShare/kvShareUninstall.integration.test.ts

@@ -110,6 +110,17 @@ const revokedBy = async (handle: string) =>
     (await env.server.stores.kvShareHandle.getByHandle(handle))?.revokedBy ??
     null;
 
+/** Run `fn` with the clock `ms` ahead, past any retry wait shorter than that. */
+const later = async <T>(ms: number, fn: () => Promise<T>): Promise<T> => {
+    const now = Date.now.bind(Date);
+    const clock = vi.spyOn(Date, 'now').mockImplementation(() => now() + ms);
+    try {
+        return await fn();
+    } finally {
+        clock.mockRestore();
+    }
+};
+
 describe('uninstalling an app', () => {
     it('keeps the data and the shares made out of it without deleteData', async () => {
         const app = await makeApp();
@@ -151,7 +162,9 @@ describe('uninstalling an app', () => {
 
         await uninstall(app.uid, true);
 
-        expect(await revokedBy(owned.handle)).toBe('system');
+        await vi.waitFor(async () =>
+            expect(await revokedBy(owned.handle)).toBe('system'),
+        );
         expect(await revokedBy(ownedElsewhere.handle)).toBeNull();
         expect(await revokedBy(held.handle)).toBeNull();
 
@@ -194,4 +207,149 @@ describe('uninstalling an app', () => {
         }
         expect(await keysIn(owner, app.uid)).toEqual(['after-reinstall']);
     });
+
+    it('returns before the shares are retired and the data deleted', async () => {
+        const app = await makeApp();
+        await env.server.stores.kv.set(
+            { key: 'k', value: 1 },
+            kvOpts(owner, app.uid),
+        );
+        const owned = await share(owner, guest, app.uid);
+
+        let release!: () => void;
+        const held = new Promise<void>((resolve) => {
+            release = resolve;
+        });
+        const store = env.server.stores.kvShareHandle;
+        const retireMany = store.retireMany.bind(store);
+        const stalled = vi
+            .spyOn(store, 'retireMany')
+            .mockImplementation(async (...args) => {
+                await held;
+                return retireMany(...args);
+            });
+        try {
+            const returned = (async () => {
+                await uninstall(app.uid, true);
+                return 'returned';
+            })();
+            const waited = new Promise((resolve) =>
+                setTimeout(() => resolve('waited'), 2_000),
+            );
+            const outcome = await Promise.race([returned, waited]);
+            expect(outcome).toBe('returned');
+            expect(await revokedBy(owned.handle)).toBeNull();
+        } finally {
+            release();
+            stalled.mockRestore();
+        }
+
+        await vi.waitFor(async () =>
+            expect(await revokedBy(owned.handle)).toBe('system'),
+        );
+        await vi.waitFor(async () =>
+            expect(await keysIn(owner, app.uid)).toEqual([]),
+        );
+    });
+
+    it('alarms when the cleanup fails, and the next check finishes it', async () => {
+        const app = await makeApp();
+        await env.server.stores.kv.set(
+            { key: 'k', value: 1 },
+            kvOpts(owner, app.uid),
+        );
+
+        const alarm = vi.spyOn(env.server.clients.alarm, 'create');
+        const failing = vi
+            .spyOn(env.server.stores.kv, 'purgeNamespace')
+            .mockRejectedValueOnce(new Error('table went away'));
+        try {
+            await uninstall(app.uid, true);
+            await vi.waitFor(() =>
+                expect(alarm).toHaveBeenCalledWith(
+                    expect.stringContaining(app.uid),
+                    expect.any(String),
+                    expect.objectContaining({ error: expect.any(Error) }),
+                    'warning',
+                    { dedup: true },
+                ),
+            );
+        } finally {
+            failing.mockRestore();
+            alarm.mockRestore();
+        }
+        expect(await keysIn(owner, app.uid)).toEqual(['k']);
+
+        await later(2 * 60_000, () =>
+            vi.waitFor(
+                async () => {
+                    await env.server.services.kvShare.resumeKvShareJobs();
+                    expect(await keysIn(owner, app.uid)).toEqual([]);
+                },
+                { timeout: 4_000, interval: 100 },
+            ),
+        );
+    });
+
+    it('keeps what the app writes once the user authorizes it again mid-purge', async () => {
+        const app = await makeApp();
+        // More than one page of the purge.
+        await env.server.stores.kv.batchPut(
+            {
+                items: Array.from({ length: 1_200 }, (_, i) => ({
+                    key: `item-${String(i).padStart(4, '0')}`,
+                    value: i,
+                })),
+            },
+            kvOpts(owner, app.uid),
+        );
+
+        // The user opens the app again while the first page is deleted, and
+        // the app writes a key that sorts after every page.
+        const dynamo = env.server.clients.dynamo;
+        const batchDel = dynamo.batchDel.bind(dynamo);
+        let reinstalled = false;
+        const deleting = vi
+            .spyOn(dynamo, 'batchDel')
+            .mockImplementation(async (params) => {
+                if (
+                    !reinstalled &&
+                    params.some((p) =>
+                        String(p.key.namespace).endsWith(app.uid),
+                    )
+                ) {
+                    reinstalled = true;
+                    await env.server.stores.permission.upsertUserAppPerm(
+                        owner.id,
+                        app.id as number,
+                        'flag:app-is-authenticated',
+                        {},
+                    );
+                    await env.server.stores.kv.set(
+                        { key: 'zz-after-reinstall', value: 1 },
+                        kvOpts(owner, app.uid),
+                    );
+                }
+                return batchDel(params);
+            });
+        try {
+            await uninstall(app.uid, true);
+            await vi.waitFor(
+                async () => {
+                    expect(reinstalled).toBe(true);
+                    const keys = await keysIn(owner, app.uid);
+                    expect(keys).toContain('zz-after-reinstall');
+                    expect(keys).toHaveLength(1_200 - 1_000 + 1);
+                },
+                { timeout: 4_000, interval: 100 },
+            );
+            // And stays that way: the purge stopped rather than paused.
+            await new Promise((resolve) => setTimeout(resolve, 500));
+            expect(await keysIn(owner, app.uid)).toHaveLength(
+                1_200 - 1_000 + 1,
+            );
+        } finally {
+            deleting.mockRestore();
+        }
+    });
 });

src/backend/services/user/UserAccountService.ts

@@ -120,6 +120,15 @@ export class UserAccountService extends PuterService {
             ownerUserId: userId,
         });
 
+        // Handles the account holds cascade with the row and nothing is told,
+        // so what their owners subscribed to through them would stay active
+        // and silent. Retiring them first ends those subscriptions.
+        try {
+            await this.services.kvShare.retireForDeletedHolder(userId);
+        } catch (e) {
+            console.warn('[cascade-delete-user] kv share retire failed:', e);
+        }
+
         // Sessions FK is SET NULL, so delete explicitly to avoid dangling rows.
         await this.clients.db.write(
             'DELETE FROM `sessions` WHERE `user_id` = ?',

src/backend/stores/app/AppStore.js

@@ -1335,18 +1335,9 @@ export class AppStore extends PuterStore {
         return this.#normalizeRow(rows[0]);
     }
 
-    /**
-     * The row's creation time as unix seconds, computed by the database: the
-     * mysql driver reads a stored UTC timestamp as local time, so parsing
-     * `timestamp` in JS can be off by the server's UTC offset.
-     */
+    /** The row's creation time as unix seconds. */
     #createdEpochColumn(alias = '') {
-        const column = `${alias}\`timestamp\``;
-        const epoch = this.clients.db.case({
-            postgres: `EXTRACT(EPOCH FROM ${column})::bigint`,
-            mysql: `UNIX_TIMESTAMP(${column})`,
-            otherwise: `CAST(strftime('%s', ${column}) AS INTEGER)`,
-        });
+        const epoch = this.clients.db.unixSecondsSql(`${alias}\`timestamp\``);
         return `${epoch} AS \`created_epoch\``;
     }
 

src/backend/stores/events/KvShareHandleStore.integration.test.ts

@@ -33,7 +33,11 @@ import {
     vi,
 } from 'vitest';
 import { isHttpError } from '../../core/http/HttpError.js';
-import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
+import {
+    createTestUser,
+    setupPuterTestEnv,
+    type PuterTestEnv,
+} from '../../testUtil.js';
 import type { IConfig } from '../../types.js';
 import { KV_SHARE_HANDLE_WIDTHS } from './columnWidths.js';
 import type { MintKvShareHandleInput } from './KvShareHandleStore.js';
@@ -304,6 +308,66 @@ describe('live handles by owner and app', () => {
     });
 });
 
+describe('a cutoff on the retire listings', () => {
+    it('leaves out handles made after it', async () => {
+        const appUid = 'app-cutoff';
+        const old = await store().mint(input({ appUid, keyPrefix: 'old:' }));
+        const fresh = await store().mint(
+            input({ appUid, keyPrefix: 'fresh:' }),
+        );
+        await env.server.clients.db.write(
+            'UPDATE `kv_share_handles` SET `created_at` = ? WHERE `handle` = ?',
+            [old.createdAt - 10, old.handle],
+        );
+        const cutoff = old.createdAt - 5;
+
+        expect(
+            (await store().listLiveForApps([appUid], 10, cutoff)).map(
+                (row) => row.handle,
+            ),
+        ).toEqual([old.handle]);
+        expect(
+            (
+                await store().listLiveForOwnerApp(
+                    ownerUserId,
+                    appUid,
+                    10,
+                    cutoff,
+                )
+            ).map((row) => row.handle),
+        ).toEqual([old.handle]);
+        expect(await store().listLiveForApps([appUid], 10)).toHaveLength(2);
+        await store().retireMany([old.handle, fresh.handle], 'owner');
+    });
+});
+
+describe('live handles one user holds', () => {
+    it('come a batch at a time after an id, live ones only', async () => {
+        const holderName = `kv-held-${Math.random().toString(36).slice(2, 10)}`;
+        await createTestUser(env.server, {
+            username: holderName,
+            password: 'pw-test-1234',
+        });
+        const holderId = (await env.server.stores.user.getByUsername(
+            holderName,
+        ))!.id;
+        const held = [];
+        for (const keyPrefix of ['held-a:', 'held-b:', 'held-c:'])
+            held.push(
+                await store().mint(
+                    input({ granteeUserId: holderId, keyPrefix }),
+                ),
+            );
+        await store().retireMany([held[1].handle], 'owner');
+
+        const first = await store().listLiveHeldBy(holderId, 0, 1);
+        expect(first.map((row) => row.handle)).toEqual([held[0].handle]);
+        const rest = await store().listLiveHeldBy(holderId, first[0].id);
+        expect(rest.map((row) => row.handle)).toEqual([held[2].handle]);
+        expect(rest[0].ownerUserId).toBe(ownerUserId);
+    });
+});
+
 describe('the live cache across nodes', () => {
     const handleReads = (spy: ReturnType<typeof vi.spyOn>) =>
         spy.mock.calls.filter(([sql]) =>
@@ -363,6 +427,26 @@ describe('the live cache across nodes', () => {
         );
     });
 
+    it('doesn`t cache a row a retire overtook during the app lookup', async () => {
+        const row = await store().mint(
+            input({ appUid: 'app-forget', keyPrefix: 'forget-race:' }),
+        );
+        const apps = env.server.stores.app;
+        const getByUid = apps.getByUid.bind(apps);
+        const lookup = vi
+            .spyOn(apps, 'getByUid')
+            .mockImplementationOnce(async (...args) => {
+                await store().retireMany([row.handle], 'owner');
+                return getByUid(...args);
+            });
+        try {
+            await store().getLive(row.handle);
+        } finally {
+            lookup.mockRestore();
+        }
+        expect(await store().getLive(row.handle)).toBeNull();
+    });
+
     it('drops an owner\'s cached rows when the owner is suspended, here and on other nodes', async () => {
         const row = await store().mint(
             input({ appUid: 'app-forget', keyPrefix: 'forget-owner:' }),

src/backend/stores/events/KvShareHandleStore.ts

@@ -24,12 +24,17 @@ import {
     mintKvHandleId,
     type KvShareAccess,
 } from '../../services/kvShare/kvShares.js';
+import {
+    APP_CLOCK_SKEW_SECONDS,
+    predatesApp,
+} from '../../util/appGeneration.js';
 import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
 import {
     keysetPage,
     openIdCursor,
     type PageResult,
 } from '../../util/pagination.js';
+import { KV_GLOBAL_APP_KEY } from '../systemKv/SystemKVStore.js';
 import { PuterStore } from '../types.js';
 import { assertColumnWidths, KV_SHARE_HANDLE_WIDTHS } from './columnWidths.js';
 
@@ -325,7 +330,8 @@ export class KvShareHandleStore extends PuterStore {
      * primary on a replica miss: that is for read-after-mint, and legacy ids
      * are no longer minted. A handle or owner forgotten in the last minute,
      * before or during the read, is read from the primary too. Concurrent
-     * misses on one handle share one read.
+     * misses on one handle share one read. A handle older than its app reads as
+     * gone.
      */
     async getLive(handle: string): Promise<LiveKvShareHandle | null> {
         const kind = kvHandleIdKind(handle, this.config.jwt_secret_v2);
@@ -365,8 +371,9 @@ export class KvShareHandleStore extends PuterStore {
             rows = await this.clients.db.pread(LIVE_QUERY, [handle]);
         }
 
-        // A forget that landed during the read may postdate what it returned.
-        const cacheable =
+        // A forget that landed during the read, or the app lookup after it,
+        // may postdate what it returned.
+        const cacheable = () =>
             !forgottenSince(this.#forgotten.get(handle), before) &&
             !forgottenSince(
                 this.#forgottenOwners.get(Number(rows[0]?.owner_user_id)),
@@ -375,19 +382,52 @@ export class KvShareHandleStore extends PuterStore {
 
         const found = rows[0];
         if (!found || found.revoked_at !== null || !found.owner_uuid) {
-            if (cacheable) this.#missing.set(handle, true);
+            if (cacheable()) this.#missing.set(handle, true);
             return null;
         }
         const { label: _label, ...row } = toRow({ ...found, label: null });
+        if (await this.predatesItsApp(row)) {
+            if (cacheable()) this.#missing.set(handle, true);
+            return null;
+        }
         const live: LiveKvShareHandle = {
             row,
             ownerUuid: String(found.owner_uuid),
             ownerSuspended: Boolean(Number(found.owner_suspended ?? 0)),
         };
-        if (cacheable) this.#live.set(handle, live);
+        if (cacheable()) this.#live.set(handle, live);
         return live;
     }
 
+    /**
+     * Whether a handle was made for an earlier app under its app's uid: an
+     * origin-derived uid comes back verbatim when the app is registered again,
+     * and its predecessor's handles must not reach the new one. Such a handle
+     * reads as gone wherever a handle is resolved.
+     */
+    async predatesItsApp(
+        row: Pick<KvShareHandle, 'appUid' | 'createdAt'>,
+    ): Promise<boolean> {
+        if (row.appUid === KV_GLOBAL_APP_KEY) return false;
+        return predatesApp(
+            row.createdAt,
+            await this.stores.app.getByUid(row.appUid),
+        );
+    }
+
+    /**
+     * SQL that holds for a handle older than its app, the same rule as
+     * {@link predatesItsApp}. Binds one parameter, the clock-skew allowance.
+     */
+    #predatesAppSql(): string {
+        const created = this.clients.db.unixSecondsSql('`apps`.`timestamp`');
+        return (
+            'EXISTS (SELECT 1 FROM `apps` WHERE ' +
+            `\`apps\`.\`uid\` = \`${TABLE}\`.\`app_uid\` AND ` +
+            `${created} - ? > \`${TABLE}\`.\`created_at\`)`
+        );
+    }
+
     #forget(handle: string): void {
         this.#forgotten.set(handle, ++this.#forgets);
         this.#live.delete(handle);
@@ -474,7 +514,8 @@ export class KvShareHandleStore extends PuterStore {
      * The live handle already minted over this exact region, if one exists —
      * what makes minting the same `(grantee, appUid, keyPrefix)` twice hand
      * back the one capability rather than a second row over the same
-     * permission.
+     * permission. One older than its app doesn't count, so sharing the region
+     * again makes a new one.
      */
     async findLive(
         input: FindLiveKvShareHandleInput,
@@ -490,12 +531,13 @@ export class KvShareHandleStore extends PuterStore {
             `SELECT ${SELECT_COLUMNS} FROM \`${TABLE}\` WHERE ` +
                 '`owner_user_id` = ? AND `grantee_user_id` = ? AND ' +
                 `\`app_uid\` = ? AND ${exactPrefix} AND \`revoked_at\` IS NULL ` +
-                'LIMIT 1',
+                `AND NOT ${this.#predatesAppSql()} LIMIT 1`,
             [
                 input.ownerUserId,
                 input.granteeUserId,
                 input.appUid,
                 input.keyPrefix,
+                APP_CLOCK_SKEW_SECONDS,
             ],
         );
         return rows.length > 0 ? toRow(rows[0]) : null;
@@ -622,19 +664,27 @@ export class KvShareHandleStore extends PuterStore {
     /**
      * One batch of live handles in these apps' namespaces, oldest first. A
      * deleted app's handles are retired off this; the caller repeats until a
-     * batch comes back short.
+     * batch comes back short. `createdAtMost` (unix seconds) leaves out handles
+     * made later, which belong to an app registered since under the same uid.
      */
     async listLiveForApps(
         appUids: readonly string[],
         limit = KV_HANDLE_RETIRE_BATCH,
+        createdAtMost?: number,
     ): Promise<KvShareHandle[]> {
         const unique = [...new Set(appUids)];
         if (unique.length === 0) return [];
+        const created =
+            createdAtMost === undefined ? '' : 'AND `created_at` <= ? ';
         const rows = await this.clients.db.pread(
             `SELECT ${SELECT_COLUMNS} FROM \`${TABLE}\` ` +
                 `WHERE \`app_uid\` IN (${unique.map(() => '?').join(', ')}) ` +
-                'AND `revoked_at` IS NULL ORDER BY `id` LIMIT ?',
-            [...unique, limit],
+                `AND \`revoked_at\` IS NULL ${created}ORDER BY \`id\` LIMIT ?`,
+            [
+                ...unique,
+                ...(createdAtMost === undefined ? [] : [createdAtMost]),
+                limit,
+            ],
         );
         return rows.map(toRow);
     }
@@ -642,22 +692,50 @@ export class KvShareHandleStore extends PuterStore {
     /**
      * One batch of live handles one owner holds out in one app's namespace,
      * oldest first, over the `(owner_user_id, app_uid, id)` index. The caller
-     * repeats until a batch comes back short.
+     * repeats until a batch comes back short. `createdAtMost` as in
+     * {@link listLiveForApps}.
      */
     async listLiveForOwnerApp(
         ownerUserId: number,
         appUid: string,
         limit = KV_HANDLE_RETIRE_BATCH,
+        createdAtMost?: number,
     ): Promise<KvShareHandle[]> {
+        const created =
+            createdAtMost === undefined ? '' : 'AND `created_at` <= ? ';
         const rows = await this.clients.db.pread(
             `SELECT ${SELECT_COLUMNS} FROM \`${TABLE}\` ` +
                 'WHERE `owner_user_id` = ? AND `app_uid` = ? ' +
-                'AND `revoked_at` IS NULL ORDER BY `id` LIMIT ?',
-            [ownerUserId, appUid, limit],
+                `AND \`revoked_at\` IS NULL ${created}ORDER BY \`id\` LIMIT ?`,
+            [
+                ownerUserId,
+                appUid,
+                ...(createdAtMost === undefined ? [] : [createdAtMost]),
+                limit,
+            ],
         );
         return rows.map(toRow);
     }
 
+    /**
+     * One batch of live handles one user holds, by id after `afterId`, over the
+     * grantee index. Deleting the holder's account retires off this before its
+     * rows cascade away.
+     */
+    async listLiveHeldBy(
+        granteeUserId: number,
+        afterId = 0,
+        limit = KV_HANDLE_RETIRE_BATCH,
+    ): Promise<Array<KvShareHandle & { id: number }>> {
+        const rows = await this.clients.db.pread(
+            `SELECT \`id\`, ${SELECT_COLUMNS} FROM \`${TABLE}\` ` +
+                'WHERE `grantee_user_id` = ? AND `id` > ? ' +
+                'AND `revoked_at` IS NULL ORDER BY `id` LIMIT ?',
+            [granteeUserId, afterId, limit],
+        );
+        return rows.map((row) => ({ ...toRow(row), id: Number(row.id) }));
+    }
+
     /**
      * Change a live handle's level or label in place, scoped to its owner. A
      * compare-and-set on `revoked_at IS NULL`, so a handle retired meanwhile
@@ -697,7 +775,8 @@ export class KvShareHandleStore extends PuterStore {
     /**
      * Live handles held by one user, newest grants last, keyset-paginated on
      * the grantee index. Selected without the owner, prefix or grant string:
-     * this is the holder's listing, and none of those are theirs to see.
+     * this is the holder's listing, and none of those are theirs to see. One
+     * older than its app is left out.
      */
     async listLiveForGrantee(
         granteeUserId: number,
@@ -712,6 +791,8 @@ export class KvShareHandleStore extends PuterStore {
             filter.push('`app_uid` = ?');
             filterParams.push(options.appUid);
         }
+        filter.push(`NOT ${this.#predatesAppSql()}`);
+        filterParams.push(APP_CLOCK_SKEW_SECONDS);
         const where = [...filter];
         const params = [...filterParams];
         if (after !== null) {
@@ -771,6 +852,12 @@ export class KvShareHandleStore extends PuterStore {
         if (options.state === 'live') filter.push('`revoked_at` IS NULL');
         if (options.state === 'revoked')
             filter.push('`revoked_at` IS NOT NULL');
+        // A live one older than its app is left out until its retire lands
+        // and it lists as retired.
+        filter.push(
+            `(\`revoked_at\` IS NOT NULL OR NOT ${this.#predatesAppSql()})`,
+        );
+        filterParams.push(APP_CLOCK_SKEW_SECONDS);
 
         const where = [...filter];
         const params = [...filterParams];

src/backend/stores/systemKv/SystemKVStore.test.ts

@@ -897,6 +897,107 @@ describe('SystemKVStore', () => {
                 ).rejects.toThrow();
             }
         });
+
+        it('asks stillWanted before deleting a page, and stops on a no', async () => {
+            await seed(3);
+            let asked = 0;
+            expect(
+                await target.purgeNamespace(namespaceOf(), {
+                    maxItems: 10,
+                    stillWanted: async () => {
+                        asked++;
+                        return false;
+                    },
+                }),
+            ).toEqual({ deleted: 0, done: false, stopped: true });
+            expect(asked).toBe(1);
+            expect(
+                (await target.list({ as: 'keys' }, appOpts())).res,
+            ).toHaveLength(3);
+
+            expect(
+                await target.purgeNamespace(namespaceOf(), {
+                    maxItems: 10,
+                    stillWanted: async () => true,
+                }),
+            ).toEqual({ deleted: 3, done: true });
+        });
+    });
+
+    describe('reserved items', () => {
+        const prefix = () =>
+            `test-reserved#${Math.random().toString(36).slice(2)}#`;
+
+        it('deletes one only while its condition holds', async () => {
+            const key = `${prefix()}item`;
+            const ifVersion = (version: number) =>
+                target.deleteReservedItemIf(
+                    key,
+                    '#v = :v',
+                    { ':v': version },
+                    { '#v': 'version' },
+                );
+            await target.putReservedItem(key, { version: 1 });
+
+            expect(await ifVersion(2)).toBe(false);
+            expect(await target.getReservedItem(key)).toMatchObject({
+                version: 1,
+            });
+            expect(await ifVersion(1)).toBe(true);
+            expect(await target.getReservedItem(key)).toBeNull();
+            expect(await ifVersion(1)).toBe(false);
+        });
+
+        it('pages through them in key order from where the last page ended', async () => {
+            const shared = prefix();
+            for (const name of ['a', 'b', 'c'])
+                await target.putReservedItem(`${shared}${name}`, { n: 1 });
+
+            const first = await target.pageReservedItems(shared, { limit: 2 });
+            expect(first.items.map((item) => item.key)).toEqual([
+                `${shared}a`,
+                `${shared}b`,
+            ]);
+            expect(first.nextKey).toBe(`${shared}b`);
+            const rest = await target.pageReservedItems(shared, {
+                limit: 2,
+                afterKey: first.nextKey,
+            });
+            expect(rest.items.map((item) => item.key)).toEqual([`${shared}c`]);
+            expect(await target.queryReservedItems(shared)).toHaveLength(3);
+        });
+
+        it('updates one only while its condition holds, and returns it', async () => {
+            const key = `${prefix()}item`;
+            const claim = (owner: string) =>
+                target.updateReservedItemIf(
+                    key,
+                    { set: { owner }, setIfAbsent: { since: 1 } },
+                    'attribute_not_exists(#o)',
+                    {},
+                    { '#o': 'owner' },
+                );
+
+            expect(await claim('first')).toMatchObject({
+                owner: 'first',
+                since: 1,
+            });
+            expect(await claim('second')).toBeNull();
+            expect(
+                await target.updateReservedItemIf(
+                    key,
+                    { set: { since: 2 }, remove: ['owner'] },
+                    '#o = :o',
+                    { ':o': 'first' },
+                    { '#o': 'owner' },
+                ),
+            ).toEqual(
+                expect.not.objectContaining({ owner: expect.anything() }),
+            );
+            expect(await target.getReservedItem(key)).toMatchObject({
+                since: 2,
+            });
+        });
     });
 
     describe('expireAt / expire', () => {

src/backend/stores/systemKv/SystemKVStore.ts

@@ -1659,19 +1659,38 @@ export class SystemKVStore extends PuterStore {
     async queryReservedItems<T extends Record<string, unknown>>(
         prefix: string,
     ): Promise<T[]> {
+        return (await this.pageReservedItems<T>(prefix, { limit: 0 })).items;
+    }
+
+    /**
+     * Up to `limit` reserved items whose key starts with `prefix`, in key order
+     * after `afterKey` (a `limit` of 0 reads one unbounded page). `nextKey` is
+     * where the next page starts, while more may remain. Expired rows are left
+     * out, as in {@link queryReservedItems}.
+     */
+    async pageReservedItems<T extends Record<string, unknown>>(
+        prefix: string,
+        { limit, afterKey }: { limit: number; afterKey?: string },
+    ): Promise<{ items: T[]; nextKey?: string }> {
         const now = Date.now() / 1000;
         const response = await this.clients.dynamo.query(
             this.tableName,
             { namespace: SYSTEM_NAMESPACE },
-            0,
-            undefined,
+            limit,
+            afterKey === undefined
+                ? undefined
+                : { namespace: SYSTEM_NAMESPACE, key: afterKey },
             '',
             false,
             { beginsWith: { key: 'key', value: prefix } },
         );
-        return ((response.Items ?? []) as T[]).filter(
-            (item) => !item.ttl || (item.ttl as number) > now,
-        );
+        const last = response.LastEvaluatedKey?.key;
+        return {
+            items: ((response.Items ?? []) as T[]).filter(
+                (item) => !item.ttl || (item.ttl as number) > now,
+            ),
+            ...(last === undefined ? {} : { nextKey: String(last) }),
+        };
     }
 
     /**
@@ -1769,6 +1788,104 @@ export class SystemKVStore extends PuterStore {
         }
     }
 
+    /**
+     * Update one reserved item, creating it when missing, if `condition` holds.
+     * `set` attributes are written outright, `setIfAbsent` ones only where the
+     * item doesn't carry them, and `remove` ones are dropped. Returns the item
+     * as it now stands, or `null` when the condition refused it. Condition
+     * tokens must not start with `#u` or `:u`.
+     */
+    async updateReservedItemIf<T extends Record<string, unknown>>(
+        key: string,
+        change: {
+            set?: Record<string, unknown>;
+            setIfAbsent?: Record<string, unknown>;
+            remove?: string[];
+        },
+        condition: string,
+        conditionValues: Record<string, unknown> = {},
+        conditionNames: Record<string, string> = {},
+    ): Promise<T | null> {
+        assertKey(key);
+        const names: Record<string, string> = { ...conditionNames };
+        const values: Record<string, unknown> = { ...conditionValues };
+        const setParts: string[] = [];
+        const removeParts: string[] = [];
+        let idx = 0;
+        const token = (attr: string) => {
+            const name = `#u${idx}`;
+            names[name] = attr;
+            return { name, value: `:u${idx++}` };
+        };
+        for (const [attr, value] of Object.entries(change.set ?? {})) {
+            const t = token(attr);
+            values[t.value] = value;
+            setParts.push(`${t.name} = ${t.value}`);
+        }
+        for (const [attr, value] of Object.entries(change.setIfAbsent ?? {})) {
+            const t = token(attr);
+            values[t.value] = value;
+            setParts.push(`${t.name} = if_not_exists(${t.name}, ${t.value})`);
+        }
+        for (const attr of change.remove ?? [])
+            removeParts.push(token(attr).name);
+        const expression = [
+            setParts.length > 0 ? `SET ${setParts.join(', ')}` : '',
+            removeParts.length > 0 ? `REMOVE ${removeParts.join(', ')}` : '',
+        ]
+            .filter(Boolean)
+            .join(' ');
+        if (!expression)
+            throw new Error('kv: a reserved update needs a change');
+
+        try {
+            const response = await this.clients.dynamo.update(
+                this.tableName,
+                { namespace: SYSTEM_NAMESPACE, key },
+                expression,
+                values,
+                names,
+                { condition },
+            );
+            return (response.Attributes as T | undefined) ?? null;
+        } catch (err) {
+            if (isConditionRefused(err)) return null;
+            throw err;
+        }
+    }
+
+    /**
+     * Delete one reserved item, conditional on an attribute the caller read
+     * still holding. False means the item changed or is already gone.
+     */
+    async deleteReservedItemIf(
+        key: string,
+        condition: string,
+        conditionValues: Record<string, unknown>,
+        conditionNames: Record<string, string> = {},
+    ): Promise<boolean> {
+        assertKey(key);
+        try {
+            await this.clients.dynamo.del(
+                this.tableName,
+                { namespace: SYSTEM_NAMESPACE, key },
+                {
+                    condition: {
+                        expression: condition,
+                        ...(Object.keys(conditionValues).length > 0
+                            ? { values: conditionValues }
+                            : {}),
+                        names: conditionNames,
+                    },
+                },
+            );
+            return true;
+        } catch (err) {
+            if (isConditionRefused(err)) return false;
+            throw err;
+        }
+    }
+
     // -- Public API ---------------------------------------------------
 
     /**
@@ -2765,12 +2882,17 @@ export class SystemKVStore extends PuterStore {
     /**
      * Delete up to `maxItems` items of one user's app namespace, as the
      * platform: no mutation events and no usage. Idempotent; `done` once the
-     * namespace is empty.
+     * namespace is empty. `stillWanted` is asked before each page is deleted,
+     * after its keys are read, and a no ends the call with `stopped`: what's
+     * written once it would say no is kept.
      */
     async purgeNamespace(
         namespace: string,
-        { maxItems }: { maxItems: number },
-    ): Promise<{ deleted: number; done: boolean }> {
+        {
+            maxItems,
+            stillWanted,
+        }: { maxItems: number; stillWanted?: () => Promise<boolean> },
+    ): Promise<{ deleted: number; done: boolean; stopped?: boolean }> {
         const parsed = parseKvNamespace(namespace);
         if (
             !parsed ||
@@ -2787,7 +2909,9 @@ export class SystemKVStore extends PuterStore {
             const page = await this.#deletePage(namespace, pageKey, {
                 limit: Math.min(maxItems - deleted, PURGE_PAGE_ITEMS),
                 consistent: true,
+                beforeDelete: stillWanted,
             });
+            if (page.stopped) return { deleted, done: false, stopped: true };
             deleted += page.deleted;
             pageKey = page.pageKey;
             if (!pageKey) return { deleted, done: true };
@@ -2804,11 +2928,20 @@ export class SystemKVStore extends PuterStore {
     async #deletePage(
         namespace: string,
         pageKey: Record<string, unknown> | undefined,
-        { limit = 0, consistent }: { limit?: number; consistent: boolean },
+        {
+            limit = 0,
+            consistent,
+            beforeDelete,
+        }: {
+            limit?: number;
+            consistent: boolean;
+            beforeDelete?: () => Promise<boolean>;
+        },
     ): Promise<{
         deleted: number;
         usage: KVUsage;
         pageKey?: Record<string, unknown>;
+        stopped?: boolean;
     }> {
         const response = await this.clients.dynamo.query(
             this.tableName,
@@ -2823,6 +2956,8 @@ export class SystemKVStore extends PuterStore {
             response.ConsumedCapacity?.CapacityUnits as number | undefined,
         );
         const keys = (response.Items ?? []).map((entry) => String(entry.key));
+        if (keys.length > 0 && beforeDelete && !(await beforeDelete()))
+            return { deleted: 0, usage, stopped: true };
         if (keys.length > 0) {
             try {
                 const deleted = await this.clients.dynamo.batchDel(

src/backend/util/appGeneration.test.ts

@@ -0,0 +1,46 @@
+/*
+ * Copyright (C) 2024-present Puter Technologies Inc.
+ *
+ * This file is part of Puter.
+ *
+ * Puter is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published
+ * by the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+import { describe, expect, it } from 'vitest';
+import {
+    APP_CLOCK_SKEW_SECONDS,
+    appGenerationStart,
+    predatesApp,
+} from './appGeneration.js';
+
+describe('app generations', () => {
+    const app = { created_epoch: 10_000 };
+
+    it('starts a clock-skew allowance before the app row was created', () => {
+        expect(appGenerationStart(app)).toBe(10_000 - APP_CLOCK_SKEW_SECONDS);
+    });
+
+    it('has no start without a creation time', () => {
+        for (const row of [null, undefined, {}, { created_epoch: 0 }])
+            expect(appGenerationStart(row)).toBeNull();
+    });
+
+    it('counts only records made before that start as older', () => {
+        const start = 10_000 - APP_CLOCK_SKEW_SECONDS;
+        expect(predatesApp(start - 1, app)).toBe(true);
+        expect(predatesApp(start, app)).toBe(false);
+        expect(predatesApp(10_000 + 5, app)).toBe(false);
+        expect(predatesApp(1, null)).toBe(false);
+    });
+});

src/backend/util/appGeneration.ts

@@ -0,0 +1,53 @@
+/*
+ * Copyright (C) 2024-present Puter Technologies Inc.
+ *
+ * This file is part of Puter.
+ *
+ * Puter is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published
+ * by the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+// An origin-derived app uid comes back verbatim when its app is registered
+// again, so anything keyed on the uid alone can outlive the app it was for.
+
+/**
+ * Allowed gap between a record's creation time (server clock) and its app row's
+ * (possibly the database clock) before the record counts as older.
+ */
+export const APP_CLOCK_SKEW_SECONDS = 60;
+
+/** An app row as read from the store, with its creation time. */
+export interface AppGenerationRow {
+    created_epoch?: unknown;
+}
+
+/**
+ * Unix seconds before which a record tied to `app`'s uid belongs to an earlier
+ * app with the same uid, or `null` when the creation time is unknown.
+ */
+export const appGenerationStart = (
+    app: AppGenerationRow | null | undefined,
+): number | null => {
+    const created = Number(app?.created_epoch);
+    if (!Number.isFinite(created) || created <= 0) return null;
+    return created - APP_CLOCK_SKEW_SECONDS;
+};
+
+/** Whether a record made at `createdAt` (unix seconds) predates `app`. */
+export const predatesApp = (
+    createdAt: number,
+    app: AppGenerationRow | null | undefined,
+): boolean => {
+    const start = appGenerationStart(app);
+    return start !== null && createdAt > 0 && createdAt < start;
+};

src/docs/src/Events/subjects.md

@@ -126,7 +126,7 @@ await puter.events.onLocal(`kv:${handle}:*`, ({ event }) => render(event.key));
 - **Keys can't leave the handle.** A handle with no key after it, or a key using `..`, is refused with `invalid_kv_handle_key`.
 - **Writes from every side arrive.** A change the owner makes, and a change any holder makes through a `write` share, reach the owner's subscriptions and every holder's. On a holder's own write through the handle, `self` is `true`, so `self && !viaWorker` skips your own changes. See [the event](#the-event).
 
-When a share stops, deliveries through its handle stop, and every persistent subscription on it is suspended with `permission_revoked` and its backlog dropped. Every [`onLocal()`](/Events/onLocal/) subscription on it ends, and its `onError` gets `subscription_ended` with `permission_revoked`. A connection to another region can take up to 20 minutes to be told, and gets no deliveries through the handle in the meantime. If another share to the same holder still covers its keys, live deliveries through the stopped handle can continue for up to 60 seconds. This happens when the owner revokes it with [`puter.kv.unshare()`](/KV/unshare/), when the holder removes their own access the same way, and when the app whose data it is gets deleted. Where `watch` shares are off on the deployment, handle subjects fail with `events_kv_handles_disabled`.
+When a share stops, deliveries through its handle stop, and every persistent subscription on it is suspended with `permission_revoked` and its backlog dropped. Every [`onLocal()`](/Events/onLocal/) subscription on it ends, and its `onError` gets `subscription_ended` with `permission_revoked`. A connection to another region can take up to 20 minutes to be told, and gets no deliveries through the handle in the meantime. If another share to the same holder still covers its keys, live deliveries through the stopped handle can continue for up to 60 seconds. This happens when the owner revokes it with [`puter.kv.unshare()`](/KV/unshare/), when the holder removes their own access the same way, when the app whose data it is gets deleted, when the owner removes that app along with its data, and when the holder deletes their account. Where `watch` shares are off on the deployment, handle subjects fail with `events_kv_handles_disabled`.
 
 **Apps.** An app can share, list, and unshare its user's data with [`puter.kv.share()`](/KV/share/) and its sibling methods, but only inside its own namespace and with the user's consent. To ask for it, call [`puter.perms.request()`](/Perms/request/) with the `'kvShare'` resource, the prefix the shares will cover, and the highest level they'll have:
 

src/docs/src/KV/listSharedByMe.md

@@ -40,7 +40,7 @@ A `Promise` that resolves to:
     - `label` (String | null): The label the holder is shown.
     - `createdAt` (Number): When the share was made, in Unix seconds.
     - `revokedAt` (Number | null): When the share stopped, in Unix seconds, or `null` while it's live.
-    - `revokedBy` (String | null): Who stopped it. It's `'owner'` when you revoked it, `'holder'` when the holder removed their own access, and `'system'` when the app whose data it was got deleted. It's `null` while the share is live, and on some shares that stopped before Puter recorded who stopped them.
+    - `revokedBy` (String | null): Who stopped it. It's `'owner'` when you revoked it, `'holder'` when the holder removed their own access, and `'system'` when the app whose data it was got deleted or you removed that app along with its data. It's `null` while the share is live, and on some shares that stopped before Puter recorded who stopped them.
 - `cursor` (String): Pass it as `cursor` to get the next page. It's included only if there are more pages to load.
 - `total` (Number): The number of shares across every page that match `appUid` and `state`. Only included when `includeTotal` is `true`.
 

src/docs/src/KV/share.md

@@ -87,6 +87,8 @@ Sharing the same prefix of the same app's data with the same user again changes
 
 Leave out `label` to keep the current label, pass a new one to replace it, or pass `null` to clear it.
 
+Concurrent calls that share the same prefix with the same user usually make one share, and every call resolves to it with the `access` and `label` of the call that made it. This isn't guaranteed for calls that reach different Puter regions at the same moment, which can each make a share. [`puter.kv.listSharedByMe()`](/KV/listSharedByMe/) lists both, and you can stop the extra one with [`puter.kv.unshare()`](/KV/unshare/).
+
 ## What the holder sees
 
 The holder is never told who shared or what the prefix is. The keys they read and write are relative to the prefix. The label is the only thing about the share you choose to tell them.

src/puter-js/tests/api/suites/perms.suite.ts

@@ -82,12 +82,13 @@ export default suite('perms', {
         });
         t.assert.ok(!revoked.error, `revoke failed: ${JSON.stringify(revoked)}`);
 
-        // The data is deleted in the background after the call returns.
+        // The data is deleted in the background after the call returns. Read
+        // as the account: opening the app again would stop the deletion.
         let value: unknown = 'x';
         for (let i = 0; i < 50 && value !== null; i++) {
-            value = await asApp(t, app.uid, () =>
-                t.puter.kv.get('perms-suite-key'),
-            );
+            value = await t.puter.kv.get('perms-suite-key', {
+                appUuid: app.uid,
+            });
             if (value !== null) await new Promise((r) => setTimeout(r, 100));
         }
         t.assert.equal(value, null);