7037 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Neal Shah	b8bb6292d6	Fix webdav locks issue and tighten GUI ipc rules for setWindow* (#4207)	2026-10-12 17:39:09 -04:00
Filip Kujundžić	542ca9ee1c	Merge pull request #4202 from HeyPuter/filipkujundzic/speech-contract-fixes ... fix(ai-speech): honor ElevenLabs output formats, voice IDs and URL audio	2026-10-12 16:21:33 +02:00
404oopsandClaude Opus 5.5	8e92bf6803	fix(ai-speech): honor ElevenLabs output formats, voice IDs and URL audio ... - TTS sends output_format as the query parameter ElevenLabs reads; the body field was ignored, so every request came back as MP3. Unknown formats are rejected before credits are reserved, and the OpenAI-style names mp3, opus, pcm and wav map to native formats. - Voice IDs go straight to synthesis. The library listing omits voices that still synthesize, the default 21m00Tcm4TlvDq8ikWAM among them, and runs to 9 MB per call. Names resolve through /v2/voices search, and premade voices ("Bella - Professional, Bright, Warm") also match their short name. - speech2speech accepts HTTPS audio URLs through secureFetch, as documented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 12:40:43 +02:00
404oopsandClaude Opus 5.5	b61448c8b7	feat(ai-image): add Gemini Nano Banana 2.1 ... Catalog gemini-nano-banana-2.1 (aliases google/gemini-nano-banana-2.1 and nano-banana-2.1) at Google's standard rates: $1.50/1M input, $7.50/1M text and thinking output, $30/1M image output, with 1K/2K/4K tiers at 1120/1680/3780 image tokens. 3.1 Flash Image stays the provider default. Pre-flight credit checks assumed 50 text/thinking output tokens per image for every Gemini model; live samples run 270-1122 depending on the model, so each model now carries its own estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 12:39:49 +02:00
404oops	f9cde23937	Merge branch 'filipkujundzic/put-1954-elevenlabs-tts-passes-voice-names-through-as-voice-ids-so' into filipkujundzic/put-1907-ai-streaming-buffering-and-error-shape-cleanup ... # Conflicts: # src/backend/drivers/ai-chat/utils/inlineImages.test.ts # src/backend/drivers/ai-chat/utils/inlineImages.ts # src/backend/drivers/util/fileInput.ts	2026-10-12 12:39:35 +02:00
404oops	94e84a2819	Merge branch 'main' into filipkujundzic/put-1954-elevenlabs-tts-passes-voice-names-through-as-voice-ids-so	2026-10-12 12:36:26 +02:00
404oopsandClaude Opus 5.5	f0b55e7605	docs(ai): trim listModels return value and AI overview per review ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 12:33:05 +02:00
Daniel Salazar	d8f9aff3dd	fix: follow-ups to #4161–#4203 (rate limits, broadcast, AI routing and billing, fs subtree walks) (#4205) ... * fix(rate-limit): count the newest KV rate rows first The KV backend's rate check, peek and concurrent acquire list the key's rows with a limit + 1 cap that the store applies before dropping expired rows, so unswept rows from past windows (oldest keys) could fill the fetch and the live count read as zero. They now list newest first. * fix(broadcast): split webhook flushes under a byte budget, per-peer backlogs Flushes go out in bodies of at most `broadcast.outboundMaxBodyBytes` (default 4 MiB) instead of one POST, and each peer keeps its own backlog of what it missed, capped by count and four bodies' worth of bytes, oldest dropped first, so one failing peer can no longer grow every peer's body past their 50 MB limit. * fix(ai-chat): classify dropped upstream connections as provider failures A connection reset or refusal (SDK APIConnectionError, Mistral's ConnectionError, an undici reset on the cause) carries no status, so a chain of them fell through to 500 internal_error. Attempts now record connectionFailed; a chain of only resets and 5xx answers 400 upstream_provider_unavailable, and a reset counts as an upstream signal in a mixed chain. * fix(ai-chat): keep SDK retries on Anthropic count_tokens The Claude client is built with SDK retries off so the driver can fall back, but count_tokens has no fallback route. It now passes sdkRetryOptions, like the OpenAI moderation call, so a transient 429 or reset is retried. * fix(ai-chat): mark a route unhealthy for 60 s after a transient failure With SDK retries off, one 5xx, 429, timeout or dropped connection kept a route marked for 10 minutes, sending that model's traffic to resellers. Those failures now mark the route for 60 s; bad credentials, an exhausted account and other route-level failures keep 10 minutes. * fix(metering): don't memoise spent addons inside the force-exact window The spent-addons memo could cache a pre-grant record read from a lagging replica right after a credits-changed announcement, so overage skipped the new credit and requireCredits kept answering 402 for up to 15 s. A spent record is now only cached once the actor's force-exact window has ended. * fix(share): send the digest entries a flush claimed when another take fails Digest entries are taken concurrently, and a single rejected take threw away every entry the other takes had already removed from KV. The flush now logs the failed takes and sends what it claimed; an entry whose take failed stays queued for a later flush. * fix(fs): announce an overwriting move's replaced entry only when it took its name /move and v2 /fs/move looked up the entry at the destination name before moving and always announced its removal. A move into someone else's Trash renames instead of replacing, so the owner's desktop hid a live entry and legacy /move reported it as `overwritten`. Both routes now pass the target on only when the moved entry kept the name it was looked up by. * fix(ai): name the vendor in upstream error fields from media providers upstreamFetch now takes the provider id plus a display label, and UpstreamResponseError carries the id. translateProviderError copies a string `provider` off the error into `fields`, and the BytePlus video and Cloudflare image errors set one, so upstream_auth_failed and upstream_credits_exhausted alarms say which vendor failed again. * fix(teams): delete a provisioned seat whose home folders could not be created provisionUser logs a failed default-folder creation and still returns the account, so provisionAccount went on to add the membership, issue a password, mail the seat and emit team.account.created for an account with no home. It now cascade-deletes the new user and returns 500 internal_error before any of that. * fix(ai-chat): load model catalogs without holding up server start onServerStart awaited every gateway catalog (up to its 10s fetch timeout) while HTTP was already listening, so drivers started after it had no providers yet and chat requests found an empty model map. The first build now runs in the background; complete(), models() and token counting wait on it instead of failing. * fix(ai-chat): serve direct models while gateway catalogs load at boot The first model-map build now publishes a fresh map as each provider's catalog lands, so static catalogs are routable at once. complete() and token counting wait for the full first build only when the model, or the provider asked for, isn't in the map yet. Refreshes still swap once. * fix(ai-chat): split gateway-billed cost across input and output A completion billed at the cost OpenRouter, Infron or Neuralwatt reports was quoted to `ai.prompt.cost-calculated` as all input and no output. The ledger still records one `billedUsage` line; the reported per-key costs now spread that bill over the usage keys at the catalog's relative rates, in whole µ¢ that sum to it. Neuralwatt energy units take no share. * fix(ai-image): hold Replicate credits for the whole prediction The output and input-surcharge holds took the default 10-minute TTL, the same as the prediction window alone, so a slow prediction could outlive them once creation and input measurement were added. They now last 15 minutes, like the video job holds. * feat(ai-chat): report what a completion was charged as usage.cost Chat usage (the non-stream result and the stream's final usage line) now carries `cost`: the µ¢ the ledger records for the request after the cost factor, from the request's recorded charges and the gate's factor lookup. `usd_cents` is unchanged. A hooked recording now copies the usage object first, so keys the driver adds later never land in the ledger. * fix(ai): charge streamed speech once its first byte arrives Per-character TTS and the voice changer recorded their charge when the upstream's response headers arrived, so a body that failed before any audio was billed in full. A shared relay now charges on the first byte, or when the caller hangs up, and charges nothing for an upstream that fails or ends before sending audio. * fix(ai): release credit holds only after the request's usage lands A hold was released as soon as its operation returned, while the usage it stood for was still being written (behind the cost-factor lookup), so a concurrent request briefly saw neither. AI metering now tracks the request's usage writes and its holds (chat included) release once those settle, without delaying the response; a streamed charge chains its own. A failed write still releases the hold, and the hold's TTL backstops it. * test(ai-chat): wait for a stream's credit hold to come back after its usage lands * perf(fs): page subtree delete and copy by parent link, not by depth over the path range Each page of a recursive delete or copy sorted the whole remaining subtree by a computed depth, so a walk cost grew with the square of its size. It now pages the children of up to 100 sibling directories at a time by parent_uid, depth first, keyset on (parent_uid, id), so a page reads only its own rows; rows with no stored path are reached too. SQLite gets the parent_uid index the others have. * fix(fs): find children by parent_uid in the empty check and supplementary-name lookup Rows the old backend wrote carry parent_uid and no parent_id, so a non-recursive delete of a directory holding only such rows passed the empty check and orphaned them, and on MySQL the supplementary-character child lookup missed them, leaving their subtrees out of listings, counts and renames. Both now key on parent_uid; the empty check still probes parent_id too, which the delete cascade follows. * docs(fs): drop the single-POST note from the entry broadcast cap	2026-10-12 01:10:54 -07:00
Daniel Salazar	b88aa55b24	fix: drop moved descendants from the entry cache; read cursor=null as the first page (#4204) ... Renaming or moving a directory rewrote its descendants' paths but left their cached copies (by id, uuid and old path) live for up to 60 s. A lookup of an old path kept resolving, and a write beneath one landed at a path its parent no longer had. updatePathPrefixForUser and renameUserHome now drop those keys, here and, within the broadcast budget, in peer regions. puter.js sent `cursor=null` for `listShared`/`listSharedByMe({ cursor: null })`, which decoded to null and failed every keyset listing with a 500. decodeCursor reads it as no cursor, and the SDK no longer sends it.	2026-10-12 00:19:11 -07:00
Daniel Salazar	ff13edc19a	Merge branch 'main' into filipkujundzic/put-1954-elevenlabs-tts-passes-voice-names-through-as-voice-ids-so	2026-10-12 02:21:00 -04:00
Daniel Salazar	f07a786b41	fix: reserve storage before creating directories on signed write starts (#4161) ... startBatchWrite and startWrite created directory items and missing parents before (or alongside) the storage reservation, so an over-quota start paid for every directory write, left the directories behind, and waited on them before returning 413. File targets are now resolved and storage reserved first; directories are created only after, and the reservation is released if that fails.	2026-10-11 22:14:57 -07:00
Daniel Salazar	21131ad3d6	fix(kv): spread upload sessions across system KV partitions (#4162) ... Upload sessions shared the system namespace's single partition key. They now go to one of 16 system partitions hashed from the session id, via a new system-actor-only `systemPartition` KV option; partitions are never cached and emit no mutation events. Reads fall back to the unpartitioned key for sessions written before this (max session lifetime is one hour).	2026-10-11 22:14:11 -07:00
Daniel Salazar	38c986927c	fix(fs): page subtree walks over supplementary-character names on MySQL (#4203) ... listDescendantsInOrder, which drives paged recursive delete and copy, still used a bare `path LIKE` range, so on MySQL a child whose name starts with a supplementary character (and its subtree) was skipped: delete left it to the parent_id cascade and copy dropped it. It now ORs the subtree patterns like the other subtree queries, paging each pattern group with the same seek and merging the pages in order.	2026-10-11 22:11:39 -07:00
Daniel Salazar	590c87c00a	refactor(stores): one row cache and keyset page helper, batched cache round trips, dead store methods removed (#4191) ... * refactor(stores): one read-through RowCache for UserStore and AppStore UserStore and AppStore each carried the same multi-key row cache: key builder, cache read, tombstone-guarded backfill, broadcast refresh, invalidate and tombstone-on-delete. They now share RowCache on PuterStore, which also single-flights concurrent misses for one key and batches getByIds/getByUids: one pipelined tombstone check for the loaded rows and one pipelined backfill instead of per-row awaits. Cache keys and TTLs are unchanged. * perf(apps): invalidate cached app lists by generation, not by tracker scan Every app write SMEMBERS'd a global set of every cached list key, parsed each key back into its filters and deleted the matches. List pages are now keyed by their owner's generation token (or a shared one for unscoped lists), and a write replaces the tokens it affects: O(1) per write, and a page loaded before a write can no longer be cached after it. * refactor(stores): one keyset page helper for store listings Thirteen store listings each cut a LIMIT n+1 read to a page and sealed the last row's position, and each opened an id cursor its own way. They now use keysetPage() and openIdCursor() from util/pagination. Cursor payloads and sealing are unchanged; a cursor without a usable id still fails with 400 in ShareStore and still restarts the listing elsewhere (strict option). * refactor(stores): delete store methods nothing calls AppStore.existsByIndexUrl, OIDCStore.unlinkByUserId, TeamStore.listByOwner/listActiveSeatUuids, SubdomainStore.getByDomain/ listByDomain, FSEntryStore.ensureDirectoriesForUser, ShareStore.listByIssuer/listByRecipientEmail/deleteByRecipientEmail/ getDailyShareCount/listByHolderAndFsentries and PermissionStore.invalidateScanCache/deleteUserUserPermsByPermissionPrefix had no production callers. ShareStore.listByFsentries was a copy of listReaching; its callers use the latter. SubdomainStore SMEMBERS'd a per-user prefix-list tracker on every write and broadcast its deletion, but nothing ever wrote that set or cached a prefix listing. The reads are gone. * refactor(stores): shared escapeLike and bumpGeneration helpers escapeLike replaces the LIKE escaping copied into PermissionStore (twice), ShareStore and FSEntryStore, and now also guards SubdomainStore's prefix filters, which passed caller prefixes into LIKE unescaped. bumpGeneration sends INCR and EXPIRE in one pipeline instead of two awaited commands in PermissionStore, EventSubscriptionStore and PresenceStore. * refactor(stores): read affected rows off WriteResult Every database client returns affectedRows and anyRowsAffected, so the `affectedRows ?? changes ?? 0` fallbacks in ShareStore, NotificationStore, UserStore and UserBlockStore read those directly. * perf(stores): batch flat-permission KV deletes and durable cache drops PermissionStore deleted each flat KV entry with its own kv.del; one kv.batchDel now covers the set. DurableSubscriptionStore's suspend and reap dropped each row from the cache separately; EventSubscriptionStore gains dropDurables, which drops them per owner in one pass, and dropDurable calls it. suspend keeps one conditional UPDATE per row, which is how concurrent settles learn which rows each of them suspended. * refactor(fs): fold FSEntryStore's duplicate getters, sort maps and cache keys - getEntryByUuid and getEntryById share one by-column getter. - listChildren and listChildrenPage read one sort table; the offset listing now coalesces NULL modified/size like the cursor listing, so both order NULLs the same way on every engine. - Entry cache keys come from one builder instead of hand-built strings. - getEntriesByIds, getEntriesByPaths and the per-user path reader fetch their cache hits in one pipelined round trip instead of a GET per key, through readCacheValues, which RowCache.readMany now uses too. * fix(apps): cache the row a lost createFromOrigin race reads from the primary Keeps AppStore's private #writeCache as a RowCache write, so code that still calls it (#4166's read-back) loads in either merge order.	2026-10-11 22:07:53 -07:00
Daniel Salazar	ed22e2b285	docs: tighten contributor and architecture guides (#4194)	2026-10-11 22:07:02 -07:00
Daniel Salazar	9030b45459	refactor(ai-chat): one OpenAI-compatible provider, one pricing path, cached catalogs (#4192) ... * perf(ai-chat): cache gateway catalogs with one remote-catalog helper cachedRemoteCatalog serves a fetched catalog for 15 minutes, shares one in-flight fetch between callers, abandons a fetch after 10 s, remembers a failure for 30 s and keeps the last good value through a failed refresh. Together, OpenRouter, Infron, Neuralwatt (models and quota) and Ollama use it in place of their kv-singleton caches. The cached value is the converted catalog, so a completion no longer re-coerces the whole gateway catalog (OpenRouter did it twice per request). Ollama now refreshes instead of caching its first non-empty list forever, and Together's catalog failure no longer rejects models(). * refactor(ai-chat): build every chat SDK client with retries off sdkClientOptions() gives each OpenAI, Groq, Together and Anthropic client maxRetries: 0 and an explicit timeout. The SDK default of two retries, each up to the full timeout, stacked under the driver's three-route fallback chain; a failed attempt now goes straight back to the driver, which marks the route and moves to the next provider for the model. Timeouts keep what each provider already had: ten minutes, Groq's SDK default of one minute, and Claude's 10 m 0.6 s. Non-streaming Mistral calls, which had no timeout once a cancellation signal was passed, are bounded at ten minutes. The Azure providers are left for after the open Azure fixes land. * refactor(ai-chat): meter every OpenAI-family completion through one pricing path meterChatUsage records a completion's usage at buildCostsOverride's rates, or as one billedUsage line when the gateway reports what it billed, and returns the per-key costs it recorded. Providers hand those to the driver through the usageCosts channel, now carried by the Chat Completions handlers too, so the usd_cents and cost-calculated figures equal the ledger. The driver's own #computeCost is replaced by pricing.ts usageCost, which shares the rate resolution with buildCostsOverride. Billing changes, all toward what the upstream charges: - DeepSeek, Moonshot, Alibaba, Groq, MiniMax, Z.AI, Hoonify, BytePlus and the Neuralwatt catalog fallback took cached reads out of prompt_tokens nowhere, so a cached token was billed at input + cache rate. splitCachedPrompt bills it once, at the cache rate; usage.prompt_tokens on those routes now excludes cached reads, as it already did for OpenAI, Gemini and Meta. - Catalog entries that listed cached_tokens: 0 (38 Alibaba, 5 Groq) relied on that double count to bill cached reads at the input rate. The zero is dropped, so they keep exactly that price through the input-rate fallback. Neuralwatt models with no published cache price likewise. - An unpriced key falls back to the input/output rate instead of NaN (0), e.g. OpenRouter cached reads on its no-cost fallback path. - A gateway-reported cost of 0 records 0 µ¢ rather than 1 µ¢. Claude, Azure and xAI keep their metering; xAI's cached-token handling waits on the open Grok decision. * refactor(ai-chat): one OpenAICompatProvider for the OpenAI-compatible vendors DeepSeek, Moonshot, Alibaba, Groq, MiniMax, Z.AI, Hoonify, BytePlus, Together, Gemini, Meta, Ollama and the complete() paths of OpenRouter, Infron and Neuralwatt were copies of one Chat Completions call. Each is now an OpenAICompatOptions entry (client, catalog, metering prefix, wire-id prefix, max-token param and default, passthrough fields, openAICompatParams scope, message cleanup, image inlining, stream usage, context-length retry) plus the hooks its API needs: prepareMessages (DeepSeek, Groq, Gemini), vendorParams (MiniMax, Hoonify, Z.AI, BytePlus, Meta, OpenRouter, Infron, Neuralwatt), meteredUsage (Gemini and the gateways) and reportedUsage (Neuralwatt). xAI is left until the open xAI fix lands. Deliberate changes: - Tools go through make_openai_tools for every vendor (six did, ten sent them raw): Anthropic-only tool extras and server tools no longer reach these upstreams, and an empty tool list is omitted. - Model lookup is one indexed, case-insensitive pass per catalog for every vendor (Together and Neuralwatt already ignored case). - Per-provider console logging of upstream errors is gone; the driver logs every failed attempt. - Every vendor honours a configured apiBaseUrl. tool_choice forwarding stays as each vendor had it (MiniMax, Hoonify, Z.AI auto-only, BytePlus, Meta auto-only, OpenRouter); widening it needs each vendor's support confirmed. * perf(ai-chat): resolve the model once in the driver and keep its map fresh The driver hands each provider its own catalog entry for the route it chose, so providers stop re-resolving the id per request; the OpenAI-compatible providers only look an id up (indexed, case-insensitively) for a direct caller, and an unknown id is a 400 instead of silently becoming the default model. Gateways with mixed-case ids (Infron, Neuralwatt) used to miss the driver's lowercased id and fall through to that default. The model map is now built from every provider's catalog in parallel, with a failing catalog left out rather than aborting the build, swapped in whole, awaited in onServerStart, and rebuilt every 15 minutes so gateway catalog changes reach routing without a restart. * refactor(ai-chat): drop provider surface and OpenAIUtil code nothing uses - list() leaves IChatProvider and every provider that isn't Azure or xAI (both wait on the open fix for those files); only tests called it. - OpenAI Chat, OpenAI Responses and Meta no longer inline puter_path parts: the driver does it before every attempt, so theirs found nothing to do. The stores and FS service they took for it go too. - providers/ChatProvider.ts had no subclasses. - OpenAIUtil: one hoistToolUseBlocks for the Chat and Responses message passes (Responses keeps canonical_id), and the dead create_usage_calculator, unused Responses deviations and constant mode/toolblock are gone. checkModeration stays on IChatProvider: the open Azure moderation fix calls it through that interface. * refactor(ai-chat): register keyed providers from one table Sixteen near-identical blocks in #registerProviders, with three constructor argument orders, become a { name, configKey, Class } table read by one loop; every entry takes (config, metering) and gets apiBaseUrl when configured. Claude, Azure, OpenAI (sibling wiring), Ollama (enabled unless turned off) and the fake provider stay explicit. Gemini's constructor takes config first like the rest. * fix(ai-chat): send Ollama the model name it listed Ollama catalog ids are ollama:ollama/<name>, built from /api/tags. Only the ollama: part was stripped, so the local server was asked for ollama/<name>, a model it doesn't have. Strip the whole ollama:ollama/ namespace. * fix(ai-chat): keep SDK retries on the OpenAI moderation call The completion clients run with SDK retries off so the driver can fall back. A moderation check has no fallback, so it passes the SDK's default two retries per request.	2026-10-11 22:06:18 -07:00
Daniel Salazar	f3134b2530	refactor(ai-media): shared upstream fetch, provider registry and bounded inputs for AI media drivers (#4188) ... * refactor(ai): route TTS, STT and voice-changer REST calls through upstreamFetch upstreamFetch bounds a provider call with AbortSignal.timeout and throws an UpstreamResponseError carrying the upstream status and code, which the /drivers/call boundary maps like any SDK error. Replaces five hand-written status translators (ElevenLabs TTS, xAI TTS, Speechify, xAI STT, voice changer); caller-facing messages are now sanitized by the shared mapper. * perf(ai): bound input image fetches and share one data-URI parser fetchImageBytes now streams under a byte cap (30 MB by default) with a 30-second timeout, via a readBoundedBody helper shared with loadFileInput's web branch and Replicate's input measuring. assertInputImagesShape caps input_images at 16. Chat image inlining uses fetchImageBytes instead of buffering with no timeout. fileInput, imageInput and mediaParts parse data URIs through one parseDataUri. * refactor(ai-tts): one per-character metering path for TTS providers meterPerCharacter holds text length x price, runs the synthesis and meters the same amount once it succeeds; characterCostReport builds the per-character cost lines. Both sit on a new withAiCreditHold (reserve, run, release) that Gemini TTS uses directly. SAMPLE_AUDIO_URL is defined once. The TTSProvider base class is gone: its methods were stubs every provider overrode, and only Polly read its untyped providerConfig. * refactor(ai): one ProviderRegistry and key reader for the AI media drivers ProviderRegistry holds a driver's provider catalog (ids, default, aliases) and its configured instances: alias resolution, the not-found 400, default fallback and parallel 'all' listing. TTS, speech-to-text, OCR and the voice changer route through it. readProviderKey reads apiKey, secret_key, api_key and key in that order for every AI media driver, so providers.xai.secret_key now enables xAI TTS as it already did xAI image and speech-to-text. ElevenLabs caches its voice list for 10 minutes, as Polly does. * perf(ai): stream TTS and voice-changer audio instead of buffering it OpenAI, xAI and ElevenLabs TTS and the voice changer hand the upstream body straight to the caller through upstreamBodyStream; their costs are known before the call, so nothing waits on the download. A body that fails mid-stream ends the response early instead of raising an unhandled stream error. xAI speech-to-text keeps one copy of the audio (the upload Blob) instead of up to three. The voice changer and xAI STT hold credits through withAiCreditHold. * perf(ai-ocr): read Puter files only when the provider needs the bytes resolveFileInput resolves and access-checks a Puter file and defers the read to load(); loadFileInput is that plus the read. OCR resolves first: Textract reads the object straight from S3 and downloads it only if Textract can't reach the bucket, and only Mistral loads the bytes. A file over the size cap is refused from its recorded size before any read. countPdfPages inflates object streams on the libuv pool instead of the request thread. * perf(ai): build image and video model catalogs once and share output saving ModelCatalog holds what the public model routes serve (models, sorted names, cost lines), computed once after boot instead of on every listing. The video driver now awaits its model map before serving. resolveOutputPath and saveGeneratedMediaToFS replace the copies of puter_output_path handling in both drivers, using expandTildePath and normalizeAbsolutePath. The video driver normalizes a copy of the caller's args instead of mutating them, and its never-taken stream branch is gone. * fix(drivers): abort the response when a streamed driver result fails mid-body upstreamBodyStream now errors when the provider body fails instead of ending cleanly, and DriverController pipes stream results through pipeStreamResult (stream.pipeline): a failed source destroys the response so the client sees an aborted transfer, the failure is logged once, and nothing is left to raise an unhandled stream error. A client hanging up stops the source. * fix(drivers): drop a streamed driver result when the caller already hung up pipeline() throws synchronously on a closed destination, which surfaced as a 500 alarm on every early disconnect and left the upstream body open. * fix(ai): bound streamed upstream calls by time to first byte, not download time upstreamFetch takes a `stream` option: the timeout covers the request until the response (or a non-2xx error body) is read, then is cleared so a body piped to the client drains at the client's pace. ElevenLabs TTS, xAI TTS and the voice changer pass it; non-streamed calls keep the full-response bound. A stalled upstream body is still cut by undici's body idle timeout. * fix(drivers): type pipeStreamResult's response by the members it uses	2026-10-11 22:05:37 -07:00
Daniel Salazar	45f6e9d8ee	refactor(drivers-wiring): AppService views and launch gates, batched app lookups, shared driver policy and guards (#4186) ... * perf(workers): skip hot-reload lookups for files no worker is bound to The fs.write.file / fs.remove.node / fs.move.node listeners listed every workers.puter.* row of the file's owner on every event, for every user. The event's FSEntry already carries the worker rows bound to it, so a file with none returns before any I/O; a bound file reads just its own rows, batched, from the primary. * perf(apps): read the uid alias only after a direct lookup misses Every AppDriver uid lookup fired an uncached system-KV GetItem for the merge alias in parallel with the direct read, even when the direct read hit. The alias is now read only on a miss. * refactor(apps): one app view with the launch gates, built by AppService AppDriver's serializer and its private-app / hosted-backing gates move to AppService.views, built on a pure toAppView that is the one place index_url is withheld. AppDriver, the subdomains driver's embedded apps and the GUI shell view all use it. Visibility of protected apps (filterReadable) moves with it. Subdomain embeds (associated_app, app_owner) used to return index_url with no gate; they now withhold it from a viewer who may not open a private app or whose hosted backing is gone, and carry privateAccess, godmode and feedback_enabled like a direct app read. * refactor(apps): one toAppSummary; batch the recommended and taskbar lookups SuggestedApps, RecommendedApps, /get-launch-apps and the taskbar each built their own launch summary; they now share toAppSummary (the extra feedback_enabled / external flags appear on the lists that lacked them). getRecommendedApps awaited getByName once per recommended app (29) and getTaskbarItems did the same per taskbar entry on every whoami; both now make one batched call per key through the new AppStore.getByNames, which keeps getByName's case-insensitive and old-name matching. The taskbar's raw UPDATE of the user row goes through UserStore.update, and /get-launch-apps checks hosted backings in one batch and reaches the stores without casts. * refactor(apps): app routes read through AppService; /delete-site calls the subdomains driver AppController used the global driversContainers through a getter whose boot-order comment no longer held; it uses the injected drivers and AppService. GET /apps/:name looked each name up, read it again through the driver and resolved private access a second time, emitting app.privateAccess.resolveLaunch twice per private app; it now batches the names and resolves once. POST /query/app did a lookup, a driver read and a raw app_filetype_association SELECT per selector (up to 200, serially); it now makes one batched lookup per key, one grant check and one filetype read. The driver's read shape is a pure toAppReadView both use. HostingController's /delete-site re-implemented the subdomains driver's delete (protected check, delete, subdomain.delete event); it keeps its owner-scoped 404 and calls the driver. Hosting and Desktop controllers drop constructors that discarded the drivers argument, and trivial getters. * refactor(drivers): one per-method policy validator and resolver; drop @Driver Four policy blocks (rateLimit, concurrent, requireSubscription, requireReputation) share the { default, methods } shape but each had its own validator and resolver. They become validatePerMethod(value, label, policy) and resolvePerMethod(cfg, method); error messages are unchanged. The @Driver class decorator and the prototype keys it wrote had no users outside its own tests; drivers declare their fields, and resolveDriverMeta reads only those. DriverController takes the driver name from the meta it already resolved instead of probing the prototype. resolveDriverMethodRateLimit/Concurrent stay as deprecated aliases for the email extension. * refactor(drivers): shared actor guards and crud-q concurrency policy The apps, subdomains, workers and notifications drivers each carried a private #requireActor, two carried the same #requireVerified, and apps and subdomains the same owner/app_owner write predicate. These become requireContextActor, requireContextUserActor, assertActorEmailVerified and actorOwnsRow in core/actor.ts; each driver keeps its own error for the cases that differed (system actors still read apps; notifications still answer 403 for an actor without a user). The identical crud-q concurrency block is one CRUD_CONCURRENT constant. The metering, installedApps and appTelemetry extensions re-checked an actor their routes already require; those checks and the tests for them go. * refactor(extensions): store methods instead of raw SQL in OSS extensions installedApps and appTelemetry queried user_to_app_permissions directly and metering read apps by name with its own SELECT. PermissionStore gains listAppsGrantedByUser, listAppAuthenticatedUsers (with the email-share check folded in) and countAppUsers; metering uses AppStore.getByName. * refactor(server): one layer builder and one lifecycle hook runner PuterServer built each of its five layers with its own copy of the same loop (built-ins, then extension additions, into the layer and its export container), and ran each lifecycle hook with five near-identical loops per hook. Both are now one helper each: #instantiateLayer and #runHook(order, hook). Construction order, the args each layer receives, and hook order (start and prepare bottom-up, shutdown top-down) are unchanged. * fix(workers): tear down a deleted source's worker after its FK nulls root_dir_id	2026-10-11 22:04:23 -07:00
Daniel Salazar	d162549c1d	refactor(controllers): one signup, login, confirm and window.user path; userProtected/internalAuth route options (#4189) ... * refactor(auth): one signup path for form, temp, OIDC and team accounts Signup steps (validate hook, phone/card forcing, bonus verdict, provisioning, signup events) move from AuthController and OIDCService into SignupService. Username rules, availability and generation move to util/username.ts, and user-row + group + default-folder creation to provisionUser, which TeamService and DefaultUserService now share. Behaviour fixes: a provisioned team seat joins default_user_group like every other account, and generated temp usernames skip names whose home path is occupied. * refactor(auth): one web-session start for password and OIDC sign-in startWebSession creates the session with the request's IP, user agent, origin and host and sets the session cookie. OIDC sign-ins used to create their session without that metadata, so they showed up blank in the session list. * refactor(auth): one confirmEmail for the code and link confirmation routes Both routes now re-run the email checks, refuse an address another account confirmed, strip it from other rows, promote the group and push user.email_confirmed to the account's sockets (with the originating socket id on the code route). The link route's full-table UPDATE clearing other accounts' pending change-email requests is dropped: /change_email/confirm already refuses an address a confirmed account owns. * refactor(auth): build window.user once for whoami and sign-in buildUserDetails and scrubSensitive move from the whoami extension into util/userDetails.ts. /whoami calls them; sign-in responses now return the same user-actor payload, scrubbed, so the login copy no longer ships the phone number and a no-reload login gets oidc_only/otp/directories up front. Taskbar resolution is best-effort on both. * refactor(http): userProtected route option replaces AuthController's route walker copy The user-protected gate is materialized from a userProtected RouteOption (implies requireUserActor), so AuthController no longer re-implements the decorator walker to wire it per instance; those routes and the session routes are plain decorators now. AuthRequired also counts requireVerified and userProtected. * refactor(http): internalAuth route option and a shared bodyRecord EventsController and PeerController each checked x-puter-internal-auth by hand with different compares; an internalAuth RouteOption now picks the secret from config and one constant-time gate checks it. ShareController and EventsController share bodyRecord instead of private copies. * fix(fs): cap legacy /delete at 500 paths per request The bulk form resolved, ACL-checked and removed an unbounded paths[] list one by one. It now refuses more than FS_BATCH_WRITE_MAX_ITEMS (500), the cap the batch write routes already use, before touching any entry. * refactor(controllers): user and subdomain writes and reads through their stores AuthController's 2FA setup/enable/disable and recovery-code writes go through UserStore.update instead of raw UPDATEs plus a manual invalidate, and /readdir-subdomains pages SubdomainStore.listByUserId instead of an unbounded SELECT; the response shape is unchanged. * refactor(auth): note SignupService's call-time peers * fix(auth): type userDetails' taskbar deps independently of TaskbarDeps.clients	2026-10-11 22:00:53 -07:00
Daniel Salazar	576e893a5b	refactor(services-core): read-only socket reauth, one token-grant read per check, shared grant/revoke and hosted-asset paths (#4187) ... * perf(socket): re-check live sockets without touching their sessions The 5-minute sweep reused the request-path authenticate(), which slid every connected session's expiry and wrote user activity each tick, and could fill SessionStore's touch-throttle map. authenticate() now takes touch: false for re-checks; the sweep uses it, checks credentials through runWithConcurrencyLimitSettled, and a call while a sweep runs joins it instead of starting another. * perf(acl): read a scoped token's grants once per check ACLService.check did a Redis GET + JSON.parse of the token's grant list for every ancestor x mode, and the permission scan did the same per option. Both now read the list once (PermissionStore.listAccessTokenPerms, which replaces the per-permission hasAccessTokenPerm) and test it in memory. * refactor(metering): incrementUsage goes through batchIncrementUsages incrementUsage was a copy of the batch path that let an infinite amount through and skipped NaN, where the batch path counts either as one unit. It now calls batchIncrementUsages with a one-entry batch, so both apply the batch clamping (documented on batchIncrementUsages). The overuse alarm loses the single-path fields nothing sets any more. * perf(metering): memoize an actor's spent addons on the increment path Every increment and credit refresh read the actor's addons record from the KV store. A record with no purchased credit left can only change through a grant, and every grant announces credits-changed, so it is now kept per actor beside the credit memo and dropped with it. Records with credit left are still read fresh, since increments on other nodes draw them down without announcing it. A read that straddles a drop isn't kept. * refactor(services): route raw table writes through their stores - access_token_permissions: AuthService inserted grants one row at a time through a cast to the store's db client, read the authorizer, and deleted the manifest with raw SQL, then asked PermissionStore to drop its cache. PermissionStore now owns insertAccessTokenPerms (chunked multi-row insert), deleteAccessTokenPerms and getAccessTokenAuthorizerId, and invalidates inside the writes. - TeamService's 2FA reset clears the OTP columns through UserStore.update. - AppIconService's data-URL icon swap is AppStore.replaceDataUrlIcon. * fix(user): correct cascadeDelete's FK note; seat cleanup is best effort The header said most user_id foreign keys are ON DELETE SET NULL; the migrations cascade nearly all of them and keep SET NULL for audit and analytics history. forgetSeat ran unguarded after the row was already gone, so a cache failure there failed the whole delete; it is now logged and stepped over like the other teardown steps. * refactor(acl): one grant/revoke path per holder kind ACLService's user and group stat/set pairs were copies differing only in which PermissionService methods they called. They now share #stat and #set over a holder adapter ({ lockKey, query, grant, revoke }); lock keys and errors are unchanged. PermissionService's eleven grant/revoke methods repeated the same guard blocks. Each keeps its own order of checks, now through shared helpers (#requireUserId, #refuseAppActor, #assertCanManage, #requireUser, #requireApp, #assertFitsColumn, #audit), so every refusal keeps its status, legacyCode and wording. Audit writes stay fire-and-forget. * refactor(auth): one hosted-asset token API keyed by kind The private-asset and public hosted-actor cookies had separate create/ verify methods, four cookie-name getters and two identical options getters, differing only in the kind claim and what the host field was called. They are now createHostedAssetToken(kind, claims), verifyHostedAssetToken(kind, token, expected), getHostedAssetCookieOptions(), and the exported HOSTED_ASSET_COOKIES names. Token claims, cookie names and every error message are unchanged; puterSite and privateAppGate call the new methods. * refactor(auth): one revoked/expired check for every actor builder The session, app-under-user and access-token builders (and the access token's parent-session check) each spelled out the same revoked/expired tests. They now share #endedSession, which returns the same reauth results; only a session token passes its auth_id. Adds coverage for an access token whose parent session expires, is revoked, or is gone.	2026-10-11 21:55:20 -07:00
Daniel Salazar	ecef154630	refactor(services): page fs subtree walks, gate share fan-out, one fs ACL helper (#4190) ... * refactor(acl): read ancestor chains from FSEntryStore and share one fs ACL helper getAncestorChain (split a path, batch-read the entries) moves to FSEntryStore, with getAncestors for callers that want the rows. Services read it from the store instead of calling FSService sideways; FSService.getAncestorChain stays for drivers and extensions. ACLService gains fsDescriptor (memoized chain), assertFsAccess, assertFsCreate and refusal, plus the pure aclRefusal mapping. The eight hand-rolled descriptor and safe-error copies in FSService, ShareService, events/authorization, FSController, legacyFsHelpers, LegacyFSController, AuthController and WebDAVController use them. Each route keeps its status, legacyCode and message: fs routes answer access_denied on a 403, the legacy routes still hide refusals from apps as 404, share and events keep forbidden and their own 404 text. * perf(fs): page recursive deletes and batch their event dispatch A recursive delete read every descendant into memory, and an empty-check for a non-recursive one did the same. It now walks the subtree a page at a time in descending path order (children before the directory holding them, so the parent_id cascade never takes a row before it is handled), deleting objects and rows and publishing each page as it goes. A non-recursive delete probes with one LIMIT 1 query. Removals publish once per entry, after the row is gone (the same change as the open fix for the double emit), with one surviving-ancestor chain read at most once for the whole subtree. Events-API dispatch for a batch runs as one background task at bounded concurrency instead of a fire-and-forget promise per entry. FSEntryStore gains hasDescendants and listDescendantsInOrder (keyset paged by path, then id). * perf(fs): copy directories a level at a time A directory copy walked every descendant in memory and copied them one by one: an insert per directory and, per file, an S3 CopyObject, a batchCreateEntries of one and its read-back. It now pages the source in ascending path order (parents before children) and takes each page a depth at a time: rows for directories, links and empty files at bounded concurrency, then the depth's S3 copies at bounded concurrency and one batchCreateEntries for all of them. A single-file copy goes through the same path. A failed S3 copy still fails the call (a ghost file still answers 404), but only after the objects that did copy have their rows, so nothing is left in the bucket unreferenced. Copied entries publish per depth in one dispatch. * refactor(share): build masked share paths in one place sharePathMask exports maskedRoot(owner, uuid, name) and maskUnder(root, realPath). ShareService's maskedPathVia goes, its maskedSelfPath and shareDeepLink's maskedSharePath build on them, and the masker, maskUnderAnchor and learnShareRoots use maskedRoot. Output is byte-for-byte the same; tests pin it. * fix(share): send old_path on a recipient's item.moved The desktop's item.moved handler reads old_path; ShareService sent the former masked path only as from_path. Both are sent now. * perf(share): skip the audience lookup for owners with nothing shared Every fs write, create, rename, move and delete looked for share recipients: an ancestor walk plus two share queries, for an owner who in nearly every case has shared nothing. #sharesReaching now reads the chain through FSEntryStore.getAncestors (no third copy of the walk) and asks a per-owner "has anything shared" flag before the share queries. The flag is the per-user cache events already keeps for subscriptions (SubscriptionCache): one ShareStore.ownerHasShares read per owner per TTL, coalesced across a burst. Creating a share (user, team, claimed invite) or moving a subtree to a new owner bumps it locally and over outer.pubsub.share.ownerShared; the TTL covers a process that misses the bump. A failed lookup reads as shared, so the full lookup decides. * perf(share): batch the per-row reads behind share listings Share listings looked things up one row at a time: - teams by id: TeamStore.getByIdsIncludingDeleted (the single-id read calls it) for listSharedByMe, listSharesOf and #revokeGroupSharesBy - team shares per node and ancestor: ShareStore.listGroupOnFsentries, one query for the chain, in the same node-then-ancestor order - #liveGroupGrants: PermissionStore.queryIssuerGroupPermsByPrefixes reads every issuer/group pair under every prefix in one query - #revokeGroupSharesBy: the five canManagePermission checks per row run together instead of one after another - ShareNotificationService #unblocked: users via getByIds and blocks via UserBlockStore.blockersAmong, two queries instead of two per member - digest flush: entries are taken at bounded concurrency rather than one at a time * perf(events): ask background consent once per holder and app on a revoke settle Withdrawing events:background re-checked consent row by row, resolving the holder's actor and asking the permission service for every one of their background rows. Rows of one holder and app stand on one consent, so it is asked once and shared. * refactor(events): drop the registry's unused push fields notify, defaultDelivery, PushMessage, NotifyProjection and pushProjection were read only by registry.test. Every subject was unpushable and broadcast by default; the subscription's delivery class stays in DeliveryClass, which the stores use. * refactor(share): retire trashed shares through onEntryOwnerChanged onEntryTrashed was a copy of onEntryOwnerChanged: the same subtree share sweep, grant retire and row delete. * fix(fs): page subtree walks by depth, not path MySQL sorts a PAD SPACE string on its first max_sort_length (1024) bytes, so under utf8mb4_unicode_ci paths past about 512 characters tie and fall back to id. A file moved under a newer directory then sorted after it: the delete walk reached the directory first and the parent_id cascade took the file without its object delete or event, and the keyset skipped or repeated rows. Pages now order by segment count, then id, which no collation can reorder. * fix(share): cache an owner as shared when the bump arrives A bump only dropped the cached answer, so the next lookup could read a replica that hadn't applied the new share yet and cache "nothing shared" for another TTL. The bump now records "shared" outright. * fix(fs): probe a directory for children by parent_id, from the primary * fix(fs): keep the copied-size check in the batched directory copy A copied file records the size CopyObject actually duplicated; growth past the source row is charged against the allowance, and a copy that no longer fits is deleted and refused, as a single-file copy already does.	2026-10-11 21:54:13 -07:00
Daniel Salazar	c815dc82a1	refactor(core-util): shared bounded map, cheaper route lifecycle, DDB/DB/rate-limit/gate dedup (#4185) ... * refactor(util): add BoundedTtlMap for core caches and log throttles One size-capped, LRU-evicting map with an optional TTL and a once-per-window shouldEmit helper. Replaces the hand-rolled maps in the DynamoDB throttle and repair loggers (the repair one was unbounded), the Slack repeat throttle, the auth probe's reauth log throttle, and the card-verification status memo. * perf(http): skip route lifecycle work when nothing listens The per-route lifecycle middleware built a payload, opened an event.emitAndWait span and hooked finish/close on every request, listeners or not. Each phase now checks hasListeners first. EventClient's three copies of the key-prefix walk become one memoized #matchKeys. * perf(ddb): write batch chunks concurrently and retry the TTL sweep batchPut/batchDel sent their 25-item chunks one after another; they now run up to four at a time through runWithConcurrencyLimit, and a failed chunk stops the ones not yet started. batchGet and the batch write share one #sendWithUnprocessedRetry instead of two copies of the backoff loop, and the TTL sweep goes through #batchWrite, so the unprocessed deletes it used to drop are retried. The AWS and local clients are built by one #bindClient. * refactor(db): share migrations and tryHardRead in the base client MySQL and Postgres each carried a copy of the migration-path runner and of the replica-racing tryHardRead; both now live on AbstractDatabaseClient (applyMigrationPaths, and tryHardRead gated on hasReadReplica), and SQLite drops its own tryHardRead override. Removes MySQL reinitPrimary, reinitReplica, isRetriableError and readWithRetry and the base requireRead, which had no callers outside tests. SQLite prepared statements and the Postgres SQL rewrite are memoized per query text in a bounded map. * refactor(rate-limit): route every entry point through one charge() The nine rate/peek/acquire entry points each repeated key and limit resolution, the backend call and the fail-open catch. They now share charge(), routeKey() and guardSlot(); log lines are unchanged. Driver concurrency slots release at most once like the other slot handles, and checkRateLimit takes { failOpen: false } for limits that must hold when the backend is down. * refactor(util): add secretsEqual for constant-time secret comparison One HMAC-then-timingSafeEqual helper, safe for inputs of any length, so callers stop hand-rolling the length guard. File signature verification uses it. * refactor(fs): infer signed-file MIME types with mime-types mimeFromName looks extensions up in mime-types instead of a 35-entry inline table, keeping the two types the table reported differently (ico, wav). The purpose-labelled signing key is derived once per secret instead of on every signature, where signFile derived it twice. * refactor(http): build route gates from one gate() helper Each gate repeated a try/catch -> next(err) wrapper or a .then(next, next) chain around an assertion. gate(check) does that once: a throw or rejection goes to next(err), and a synchronous check still calls next synchronously. The credits, subscription and reputation gate files held nothing but such a wrapper; their factories move into gates.ts and the files go. * perf(hosting): build the puter-site hosting config once createPuterSiteMiddleware rebuilt the normalized hosting-domain config on every hosted-site request; it is derived from static config, like the domain lists the middleware already builds at construction. * fix(db): tryHardRead's replica leg doesn't fail over to the primary MySQL's read() retries on the primary after a load-shed or transient replica error, so routing tryHardRead through it read the primary twice on that path. The replica leg now reads the replica alone. * fix(ddb): settle in-flight batch chunks before rethrowing A failed chunk stopped unstarted chunks but rejected while others were still retrying, so writes could land after the caller had handled the error and invalidated its cache.	2026-10-11 21:53:33 -07:00
Daniel Salazar	b5eefaa9cc	fix(fs): include supplementary-character names in subtree path queries on MySQL (PUT-2135) (#4198) ... * fix(fs): include supplementary-character names in subtree path queries on MySQL (PUT-2135) MySQL ends the idx_fsentries_path range for `path LIKE 'p/%'` at 'p/' + U+FFFF, and utf8mb4_unicode_ci sorts supplementary characters above it, so a child whose name starts with one (most emoji) was skipped with its whole subtree by listing, delete, copy, rename/move rewrites, counts and sizes. On MySQL, FSEntryStore now looks up those children by parent_id and ORs a LIKE pattern for each onto the subtree match, recursing into directories. Every extra pattern starts with the prefix, so only the scanned ranges widen; rewrites past 500 patterns run in several statements. The prefix rewrite also passed a UTF-16 length to SUBSTR, which counts characters, so a rename under a path holding an emoji dropped characters from every descendant path on all engines. It now counts code points. * test(fs): seed the supplementary-name subtree with createNonFileEntry	2026-10-11 21:21:18 -07:00
Daniel Salazar	af9ebb0ef1	fix(ai): map BytePlus video and Cloudflare image upstream errors at the driver boundary (#4184)	2026-10-11 21:20:27 -07:00
Daniel Salazar	2131b980ce	fix(redis): release locks with an atomic compare-and-delete (#4183) ... * fix(redis): release locks with an atomic compare-and-delete ACLService, TeamService and the share digest flush each hand-rolled a SET NX lock whose release was a GET-then-DEL (or a blind DEL), so a holder whose TTL lapsed could delete the next holder's lock. They now share withRedisLock, which claims with a random token and releases via a Lua compare-and-delete. * chore(team): place the redis lock import with the other runtime imports	2026-10-11 21:19:37 -07:00
Daniel Salazar	96909b0800	fix(fs): send the same outer.gui.item.* payload from every fs controller (#4179) ... * fix(fs): send the same outer.gui.item.* payload from every fs controller The v2 /fs/move and /fs/delete routes spread the raw camelCase FSEntry into item.moved / item.removed and named the old path `from_path`, while the desktop reads the legacy snake_case shape and `old_path`. One helper, emitGuiItemEvent, now builds the payload for LegacyFSController, FSController and WebDAVController. v2 and WebDAV copy/move with overwrite now also emit item.removed for the entry they replace, as the legacy routes do, and v2 /fs/move forwards new_metadata to the service like the legacy /move. * fix(fs): announce a replaced entry before the one replacing it The desktop hides rows by path on item.removed, so a removal sent after the overwriting copy's added (or move's moved) hid the new row too. The shared emitter now takes the replaced entry and sends its removal first, awaited, on the v2, legacy and WebDAV copy/move paths.	2026-10-11 21:18:47 -07:00
Daniel Salazar	906d6b7886	fix(email): one canonicalization and blocklist for every address path (#4176) ... Share invites, share notifications and app feedback checked addresses through EmailClient.clean/validate, a second copy of the rules in util/email.ts that stripped dots on iCloud, stripped `+tags` on every domain, and matched the blocklist case-sensitively. They now use util/email.ts: isBlockedEmail for the blocklist, abuseKey for the per-address notification budget, and cleanEmail for the invite digest key. EmailClient keeps transport and templates only. isBlockedEmail also lowercases the configured suffixes, so a mixed-case blocklist entry still matches.	2026-10-11 18:24:28 -07:00
Daniel Salazar	6a938bd8e8	fix(db): detect duplicate keys with isUniqueViolation everywhere (#4175) ... FSService, OIDCStore, UserStore and TeamService each carried their own duplicate-key check. FSService only matched mysql and the bare SQLITE_CONSTRAINT code, so the mkdir/move race recovery never ran for a SQLite or Postgres unique violation. OIDCStore matched any message containing "UNIQUE" or the bare SQLITE_CONSTRAINT code, UserStore any SQLITE_CONSTRAINT_* code, and TeamService any message mentioning unique or duplicate. All four now use util/dbError's isUniqueViolation; UserStore still checks the index name on top.	2026-10-11 18:24:13 -07:00
Daniel Salazar	847a4fc031	fix(fs): route thumbnail writes through FSEntryStore (#4174) ... * fix(fs): route thumbnail writes through FSEntryStore Add FSEntryStore.updateEntryThumbnailByUuid, which sets a thumbnail with no owner check (callers authorize) and refreshes the entry cache, with an optional expected-value guard. The thumbnails extension's inline data: URL migration, its copy duplication, and the legacy /update-fsentry-thumbnail endpoint now write through it instead of raw UPDATEs, so cached entries stop serving the old thumbnail. The migration validates the image like the upload path, awaits the write, and only repoints a row that still holds the data URL; a read that loses that race deletes its upload. * fix(fs): report a thumbnail update by whether the row changed updateEntryThumbnailByUuid returned the refreshed entry, which is null when the cache misses and the replica hasn't seen the row yet. The read-time migration took that null for a lost race and deleted the object the row had just been pointed at. It now returns whether the UPDATE changed a row; the cache refresh still runs after it.	2026-10-11 18:23:58 -07:00
Daniel Salazar	3296526e8e	fix(ai-chat): derive Azure model pricing from the OpenAI and xAI catalogs (#4173) ... Azure entries that front an OpenAI or xAI model now come from a `mirror` helper that copies every non-identity field from the source catalog entry. The hand-copied entries had drifted: none carried `long_context_pricing`, so long prompts on Azure-served gpt-5.4 and Grok 4.3/4.20 were billed at 1x; grok-4.3 had lost `pdf` input and the Grok 4.20 deployments still listed the old 2M context. Models no longer in the direct catalogs stay inline.	2026-10-11 17:57:14 -07:00
Daniel Salazar	c553a380ae	fix(ai-chat): forward max_tokens to xAI; moderate Azure through OpenAI (#4172) ... xAI sent a fixed max_tokens of 1000, ignoring both the caller's value and the credit gate's cap. It now forwards max_tokens like the other OpenAI-compatible providers and omits it when unset. Azure chat's checkModeration threw "Method not implemented", so `moderation: true` on an Azure route metered the completion, then failed it as a route error: Azure was marked unhealthy and the fallback billed a second completion. Azure's v1 API has no /moderations endpoint, so both Azure providers now delegate moderation to the OpenAI chat provider (Azure chat via its Responses sibling, as it does for web_search). A moderation call that throws is now a 400 `moderation_unavailable` that the driver treats like `moderation_flagged`: final, no fallback, and the route stays healthy.	2026-10-11 17:55:38 -07:00
Daniel Salazar	cf9df8719a	fix(alarm): forget idle alarms and cap how many the client holds (#4171) ... AlarmClient kept every alarm id it had seen for the life of the process, and ids that embed a user, key or invoice grew it without bound. The registry is now ordered by last occurrence: alarms idle for 24 hours are dropped on the next create, and past 1000 the least recently raised goes first. Clearing or evicting an alarm only removes its short-id alias when the alias still points at it.	2026-10-11 17:54:26 -07:00
Daniel Salazar	7ad963bacf	fix(cache): broadcast fs entry and team invalidations to peer regions (#4169) ... * fix(cache): broadcast fs entry and team invalidations to peer regions FSEntryStore's invalidations called publishCacheKeys without broadcast, so a delete, move, rename or overwrite only cleared the writer's cluster; other regions served the old row until the 60 s TTL. They now broadcast. Batch paths (deleteEntries per chunk, the batch overwrite) send one event for the whole batch. Read-path backfills stay local. TeamStore's busts were a local redis.del, so a removed member kept passing getMembership in other regions until the TTL. They now go through publishCacheKeys with broadcast, and softDelete busts every key in one call. Remove FSEntryCacheInvalidationEventHandler. It re-derived the same invalidations from outer.gui.item.* with a replica SELECT per holder, and its fs.remove.node branch never ran (FSEntry has no get()). The store's own broadcast covers what it did. * fix(fs): cap how many entry invalidations a node broadcasts per window FSEntryStore now broadcasts at most 10,000 entries' cache invalidations per broadcast flush window (outbound_flush_ms, 2 s by default). Past that, keys are dropped locally only and peers fall back to the 60 s entry-cache TTL; the store logs once per window when this happens. Each flush goes to a peer as one POST that is re-sent whole on failure, so a large recursive delete could otherwise build a batch the peer rejects on every retry. The budget is per store instance, not per call, so a delete issued page by page is bounded too, and single-entry writes count against the same budget.	2026-10-11 17:53:26 -07:00
Daniel Salazar	0066d49f74	fix(fs): emit fs.remove.node once per entry on recursive delete (#4167) ... #removeDescendantsStorage emitted fs.remove.node for every file descendant while grouping S3 keys, and remove() emitted it again for every descendant after deleting the rows, so file listeners ran twice. Drop the storage-pass emit; remove() now emits once per entry after its row is deleted.	2026-10-11 17:52:27 -07:00
Daniel Salazar	b67acdbf6f	fix(stores): return written rows without reading them from a lagging replica (#4166) ... * fix(stores): read rows back from the primary after writing them ShareStore (create, upsertPending, upsertActive, upsertActiveGroup, applyPending, upsertAnyone), AppStore (create, createFromOrigin, update, claimOwnership), NotificationStore.create, SubdomainStore.update and NotificationDriver.create returned the row they had just written by reading it back through `db.read`, which goes to the replica. Under replica lag that came back null or stale. Those read-backs now go to the primary, through a `primary` option on the getters. AppStore caches the primary copy on create, and claimOwnership no longer reads through getById, which cached the replica's pre-claim row on a miss. * fix(stores): skip the notify read-back; read OIDC link conflicts from the primary NotificationStore gains `insert`, which writes and returns the uid; `create` is now `insert` plus the primary read-back. NotificationService .notify ignores the row, so it uses `insert` and no longer pays a primary round trip per recipient. NotificationDriver.create still reads the row back from the primary. OIDCStore.link reads the conflicting row from the primary after a unique violation, so a replica that has not seen another account's link can no longer make the call report success. * perf(stores): return written rows instead of reading them back ShareStore.create, NotificationStore.create, AppStore.create and AppStore.createFromOrigin build the row they return from the values they inserted and the insert id, instead of reading it back from the primary. The DB-defaulted creation timestamp is now written explicitly from unix seconds (`timestampFromUnixSql`), so the returned value matches what a read returns (`timestampValue`); postgres INSERTs add RETURNING id. The AppStore rows fill every column the INSERT leaves to the schema and are cached as before. ShareStore.upsertPending's re-invite branch selects the full pending row in its dedupe read and returns it with the new mode and data; it falls back to the primary only when the UPDATE matched nothing. The remaining primary read-backs stay and say why: the share upserts and the new-invite insert (an upsert's surviving uid/created_at and the entry-owner subquery are only known to the primary), applyPending and claimOwnership (only an id or uid in hand), AppStore.update and SubdomainStore.update (the base row is a replica copy that would be cached), NotificationDriver.create (notify returns only the uid) and the OIDC link conflict.	2026-10-11 17:51:38 -07:00
404oops	64db150a90	fix(ai): bound streaming buffers and normalize input errors	2026-10-11 21:14:16 +02:00
Daniel Salazar	f88cadb4b9	fix: a copy records the size of the bytes it duplicated (#4197) ... A copied file's row took its size from the source row the request had resolved, which can predate an overwrite whose bytes CopyObject then copied. The copy now reads the new object's size and records that. When it is larger than the source row said, the allowance is checked again against the real size; a copy that no longer fits is refused with 413 and its object deleted. If the size check fails, the copy falls back to the source row's size, as completing a signed upload already does.	2026-10-11 10:48:52 -07:00
Daniel Salazar	aef0a32d72	fix(ai): hold credits for image and video generation (#4164) ... Image and video providers checked the balance with hasEnoughCredits / getRemainingUsage and took no hold, so concurrent requests each saw the full balance until the first one recorded usage. They now reserve the estimate through the cost-factored reserveAiCredits and release it in a finally block once usage is recorded, like chat, speech and OCR. The video duration clamp now prices seconds at the model's cost factor and holds the clamped clip's cost for the job (poll window plus create time). Replicate holds the output-only estimate before fetching inputs and the input surcharge on top once it is measured.	2026-10-11 09:18:13 -07:00
404oops	800e300a8c	fix(ai-tts): resolve ElevenLabs voice names before synthesis	2026-10-11 16:33:05 +02:00
Daniel Salazar	491729e060	fix: backend test flakes and a socket shutdown hang (#4170) ... - SocketService's prepare-shutdown hook awaited io.close(), which also closes the http server and waits for every HTTP connection to end. A connection mid-request when close() runs stays open for the keep-alive timeout, so shutdown stalled before PuterServer severs connections. The hook no longer waits on the http close. - Test servers disable Ollama unless a test configures it. Its boot probe logged after files had ended, which failed shared-worker shards with EnvironmentTeardownError. - vitest ignores that teardown error for a console call still in flight. - setupPuterTestEnv boots again on a fresh port after EADDRINUSE, and a failed setupTestServer start shuts the server down. - UploadReservationStore expiry tests pin the clock instead of racing a 50 ms TTL.	2026-10-10 19:34:32 -04:00
Daniel Salazar	d4589caefe	fix: session store revokes (#4168)	2026-10-10 07:55:43 -07:00
Daniel Salazar	4fca4809b9	fix(health): discount total event-loop stall time from db liveness latency (#4160) ... The stall watcher kept only the longest gap, so a saturated loop with many sub-second stalls still failed database-liveness. It now sums blocked time across the probe (ignoring timer jitter) and reports total and longest.	2026-10-09 18:14:48 -07:00
Daniel Salazar	cc8b9858a2	feat: let extensions claim a subdomain (PUT-2100) (#4155) ... `extension.claimSubdomain(name, { authorize, checkRoute? })` lets an extension serve `<name>.<domain>` without a hard-coded reserved name. - A claimed host is dispatched where the unknown-subdomain 404 used to be, before CORS, the OPTIONS responder, body parsing, auth and every route (including `subdomain: '*'` ones). - `authorize` returning anything but true, or throwing a non-HttpError, yields exactly the unknown-subdomain 404. A thrown HttpError goes to the error handler. - Routes whose `subdomain` is the claimed name (decorated controllers and extension routes alike) are materialized onto the claim's own router, so they only run after `authorize` and never on the main app. Their options are limited to ones that don't rely on the main pipeline's actor; an optional `checkRoute` lets the claim refuse routes at boot. Case-variant labels and claimed names mixed into subdomain arrays fail the boot. - The unknown-subdomain check now compares the port-stripped hostname and requires a dot boundary, so a host with an explicit port behaves like one without it. Hosts served by the local worker proxy still pass through.	2026-10-09 18:14:04 -07:00
Daniel Salazar	6060de8fbd	fix(alerts): overspend alarm in credits at warning; no-choices completions stop paging (#4154) ... - The per-account overspend alarm now reads in credits and raises at warning so it reaches the pager instead of only chat. - A non-stream completion with no choices (an upstream answering 200 with an error body) threw a TypeError that classified as internal_error and paged critical. It now throws bad_response, which falls back like any other empty response and surfaces as a 400.	2026-10-09 18:13:44 -07:00
Daniel Salazar	0e0cedb02b	fix(auth): no email password recovery for built-in admin accounts (#4153) ... - /send-pass-recovery-email for `admin`/`system` (by username or email) returns the usual generic message, writes no token, sends no mail, and raises a critical alarm with the requester's IP, user agent and, when signed in, their user. - /verify-pass-recovery-token and /set-pass-using-token reject recovery tokens for those accounts as no longer valid.	2026-10-09 18:13:22 -07:00
Neal Shah	43094a84a0	PUT-1901 (#4159)	2026-10-09 20:12:45 -04:00
Neal Shah	5fb025c109	Misc subdomain and worker fixes (#4157) ... * PUT 1902 * LocalWorker fetch policy fixes for 127.0.0.1	2026-10-09 19:59:52 -04:00
Juan Fernando Castro	50bcdc9c6c	fix: let a revocation be seen by the reads and the nodes that outlive it (#4151) ... * fix: let a revocation be seen by the reads and the nodes that outlive it Four places where a revoke lands on the primary and the next reader can still be told the old answer. The u2a cache key was dropped but not re-filled, so the next miss read a replica that may still carry the row and cached it for the TTL -- and a revoke's own settling pass is usually that miss. It re-fills from the primary, as the u2u side already does. A revoked session dropped its keys the same way. It now leaves the revoked row in the cache, read from the primary, so there is nothing for a lagging replica to re-introduce. A link share is its own authority rather than an index over a permission, so a replica still carrying a withdrawn row answers with it. For a short window after one is withdrawn, those reads go to the primary. Socket eviction rode a local event, so it only reached the node that did the revoking. It is announced on the fan-out channel too, which is where the node actually holding the connection hears it. * refactor: carry the revocation on outer.* instead of outer.pubsub.* Eviction runs through `io.in(room).disconnectSockets()`, which goes via the socket.io Redis adapter a cluster shares, so one node handling the event disconnects that user on every node in it. Fanning out to siblings had each of them repeat the same cluster-wide disconnect. The adapter's Redis is per-cluster, so peer clusters still need the hop -- `outer.*` reaches them and skips the sibling fan-out. The local `auth.sessions.revoked` goes with it: `outer.*` fires on the emitting node too, and nothing else subscribed.	2026-10-09 16:36:26 -04:00
Juan Fernando Castro	ff35fbd66e	Share attribution and cascade: finish PUT-1856 and PUT-1857 (#4119) ... Replayed onto main after #4118 squash-merged. Same tree as the reviewed head (8d8bd871f); only the redundant merges from the old base are gone.	2026-10-09 16:23:18 -04:00
Juan Fernando Castro	1ff5d93244	fix: a signed overwrite lands on the live key only once its completion is accepted (#4136) ... * fix: a signed overwrite lands on the live key only once its completion is accepted A presigned overwrite URL was signed against the live object key, so the browser's PUT replaced the owner's bytes directly. That PUT never passes through Puter, which leaves the access check on `completeWrite` with nothing left to protect: by the time it runs, what /read serves has already changed. The upload now goes to a key of its own. `completeWrite` is already the authorized step, so that is where the staged object is copied onto the live key, server-side, and the staged one dropped. The entry keeps its uuid, so everything written against it still holds. Aborting deletes the staged object, which the old shared key made unsafe to do. `copyObjectOfAnySize` carries the copy past the single-request ceiling by copying part by part, so a large file is no different. Two behaviours follow from the key no longer being shared: completing an overwrite that uploaded nothing is refused rather than reported as an overwrite of unchanged bytes, and abort always drops the session's object. Only the two presigned paths changed; `prepareBatchWrites` uploads server-side within one authorized request and still writes in place. * fix: the review's findings on the staged overwrite The batch signed path still signed against the live key, so the route the SDK's signed upload actually takes was unchanged. It stages now, which also makes the abort simplification safe: the guard that had protected a shared key was removed while that path still shared one, so aborting a batch overwrite would have deleted the owner's object. The completion is allowed against a path but lands on the uuid captured at start, and a move between the two puts the bytes somewhere the caller was never authorized for. The promote refuses unless the entry is still where the check looked. The batch promote failing left every session pending with its lease held; it now fails them as the rest of the function does. The pending GUI tile and the thumbnail key were keyed on the staged uuid, which no entry ever carries, so neither correlated with what the completion emitted. Responses carry `entryUid` for that. * fix: do not treat a pre-staging session's key as this upload's own A session signed by the previous release points straight at the live key, since `objectKey` was `existingEntry?.uuid ?? uuidv4()`. Completing one is already handled -- the promote returns early when the two keys match -- but abort deleted `objectKey` unconditionally, which during a rolling deploy would take the owner's file rather than the upload's litter. Abort now deletes only a key the upload owns. The multipart branch was always safe: discarding parts never touches an object already in place.	2026-10-09 16:22:36 -04:00