Model sync against Mistral's listing, docs and a live call per id:
mistral-ocr-latest now serves OCR 4.1, so the catalog pins it (with
mistral-ocr-4 and the retired-but-redirected 2503) to mistral-ocr-4-1,
adds mistral-ocr-4-0 and OCR 3 (mistral-ocr-2512 / -3 / -3-0), and
rejects the retired mistral-ocr-2505 before any upstream call. Mistral
usage is metered per model at vendor rates: $4/1000 pages for OCR 4.x
with $5 annotations, $2 and $3 for OCR 3, replacing the flat $1 and $3.
Textract now calls DetectDocumentText instead of AnalyzeDocument with
LAYOUT. The layout feature cost $4/1000 pages while we billed the
$1.50 detect rate, and it only added text-less LAYOUT_* blocks that put
blank lines into img2txt output. Multi-page PDFs and unsupported
formats become a 400 that names Mistral as the alternative.
Request shape: a model name alone now selects its provider, so calls
differ only by model; an explicit provider still picks its default and
must agree with the model. Mistral options gain tableFormat,
extractHeader, extractFooter and documentAnnotationPrompt; annotation
formats accept the REST (json_schema.schema) or SDK
(jsonSchema.schemaDefinition) spelling and are validated before
billing. Non-image documents such as DOCX go as document_url.
Response shape: both providers return model, text and blocks with a
0-based page; Mistral also returns document_annotation, which was
billed but dropped. img2txt resolves to that annotation when
documentAnnotationFormat is set and to the text otherwise; the return
is a string for every model, so normalize has no effect on it.
Input size: OCR input is capped at the provider limit (10 MB Textract,
50 MB Mistral), and web inputs in loadFileInput are streamed under the
cap instead of buffered whole, which also bounds speech2txt URLs.
Docs: img2txt models, options, input limits, return value and errors;
OCR limits in the quotas page.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(ai): send stable, non-sequential user identifiers to AI providers
A precedence bug in the AI providers' identifier expression made every
request send `user: ":undefined"` (the ternary bound the app-uid suffix to
the whole `actor.user.id + actor.app?.uid` sum instead of just the suffix),
or read `actor.user.id` on a missing user. The same expression also shipped
the sequential internal user id, letting AI vendors correlate a single
account across apps and sessions.
All eight OpenAI-, Azure-, xAI-, Meta- and ZAI-style providers now build
the identifier through one shared helper, `aiUserIdentifier()`:
- `puter-<user-uuid>[-<app-token>]`: the random user UUID is always
preserved in full; `maxLength` constrains only the app-bearing form
- app attribution reads `effectiveApp`, so access-token requests name the
issuing app instead of looking like direct user traffic
- the app token is truncated to fit the budget, and omitted entirely when
the remaining budget is below 8 chars, where a truncation could collide
with another app's uid
- nothing is sent for the system actor
- Meta and ZAI keep a caller-supplied `safety_identifier` / `user_id`
override, applied before the helper result
`user` is deprecated by OpenAI; the SDK types direct callers to
`safety_identifier` (abuse detection) and `prompt_cache_key` (cache-hit
bucketing). The four OpenAI/Azure chat providers and MetaProvider now send
`prompt_cache_key` as well, defaulting it to the same per-user identifier
unless the caller supplies one; Azure's Grok branch drops both fields,
matching its rejection of unknown args. The cap comment cites only verified
limits: OpenAI's 64 for `safety_identifier` (from the SDK types) and Z.AI's
6-128 for `user_id` (from Z.AI's docs); Meta and xAI document none, so none
is claimed.
The xAI image `#edit` path now carries the identifier like generation, and
takes a named-options param so `user` cannot be transposed with the
adjacent same-typed `aspectRatio`.
Tests share a four-actor matrix (`user` / `user+app` / `access token` /
`system`) with `assertActorMatrixIdentifiers()` across the six
OpenAI-style suites; the helper has exact-string and boundary coverage
(size caps, zero-budget and sub-base cases, no dangling separator, UUID
never truncated, collision guard); the Azure Grok assertions run under a
real user actor so they cannot pass vacuously. 212 provider-suite tests
pass; typecheck and ESLint are clean.
* fix(ai): lock the vendor identifier down and keep vitest out of the test util
Meta and Z.AI no longer let `custom` override the abuse identifier; it is
Puter's attribution, not the caller's. The shared test util exposes pure
field pickers instead of importing vitest into a file the production
tsconfig compiles. The helper's length-cap comment now matches vendor docs
(Meta does cap `safety_identifier` at 64), the redundant budget branch and
the unused export are gone, and the per-user `prompt_cache_key` trade-off is
stated once in the helper instead of five times in providers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: 404oops <me@404oops.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(ai-image): sync image catalogs, unify request shape, document models
Model sync against every vendor listing and a live generation sweep:
Gemini stable ids replace the retired preview spellings (2.5 Flash Image
delisted ahead of its 2026-10-02 shutdown), OpenAI gpt-image-1/-1-mini/-1.5
are delisted but routable until their shutdown dates with gpt-image-2 as
the default, xAI gains grok-imagine-image-2.0 with its quality tiers,
BytePlus gains the 3K/4K tiers and per-model pixel bounds, Cloudflare
gains SDXL Lightning/Base and SD 1.5 Inpainting, Replicate gains a
schema-driven catalog of 88 additional models with version-pinned
community predictions, and every Together image route is excluded for
the third-party data-sharing requirement.
Request normalization: the driver collapses ratio/width/height/aspect_ratio
into one imageSize with aspect-versus-pixel intent, validates prompt,
quality and resolution once, resolves provider hints (short names or
full driver ids) and aliases with exact ids winning over resellers, and
hands each provider an immutable copy of the caller's args. Providers
share prompt validation, aspect snapping, and content-sniffed data URIs
so responses carry the right MIME type. Replicate predictions are created
once, cancelled on abort or deadline, and bounded in input fan-out.
SDK: txt2img copies caller options, rejects blank prompts with
prompt_required in every call form, and documents that normalize has no
effect because the image return shape is already uniform.
Docs: txt2img options per provider, provider defaults and discovery,
availability notes, a new image model catalog and pricing page, and the
image-generation limits in the quotas page.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(ai-image): sniff SVG outputs correctly, reject hex/exponent dimensions
imageDataUri only decoded the first 24 base64 chars (18 bytes) before
sniffImageMime, which cannot see an <svg> root behind an XML prolog -
SVG outputs from recraft-v*-svg models were being labeled image/png.
Decode enough of the payload to cover the 8 KB SVG sniff window.
dimension() accepted string number literals ('0x10', '1e3') as if they
were decimal dimensions; restrict to plain decimal notation.
* docs(ai): link to the model directory instead of a static catalog
The image model catalog and pricing page duplicated the always up to date
model directory at developer.puter.com/ai/models. Link to the directory
from txt2img-related docs and keep the Together data-sharing exclusion
note self-contained.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
OpenAI shuts the Sora Videos API down on 2026-09-24 and sora-2 was the
default txt2vid model, so the default moves to Veo 3.1 Lite on Gemini
and the OpenAI video provider goes. While there, the video catalogs are
brought in line with what each vendor serves today, the request options
are unified across providers, and the txt2vid docs are rewritten.
- driver: default provider gemini-video-generation with
veo-3.1-lite-generate-preview; a request under the generic `ai-video`
driver name lands on the default instead of the first-registered
provider; `WIDTHxHEIGHT` sizes map onto tier catalogs by the shorter
side and fill width/height
- openai video provider, the `openai-video-generation` alias, its
config template and migration entries, and Together's openai/sora-2*
rows removed
- gemini: Veo 3.1 Fast rates 10/12/30 cents per second for
720p/1080p/4K, Veo 3.1 Lite accepts reference images, URL image
inputs are fetched server-side through the SSRF-guarded fetch
- together: drop nine models retired upstream, add eighteen from the
live listing; per-second models are estimated from the catalog rate,
clamped to remaining credit and billed at the cost Together reports
on the job; tier-sized models take resolution/ratio;
input_reference/last_frame map onto keyframes; generate_audio is
forwarded
- byteplus: Seedance 2.5 (dreamina-seedance-2-5-260628) with per-model
reference-image caps
- util/imageInput: string-level image helpers shared by the image and
video drivers; ai-image/inputImage re-exports them unchanged
- puter.js types: provider and generate_audio options; docs: txt2vid
page rewritten with per-provider model tables, unified options and
four new playground examples
Known follow-up: Veo returns a key-protected Google file URL, so the
default clip cannot be played directly by a browser until the provider
fetches it server-side.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
- puter.email.sendTransactional is the new name; send stays as a
deprecated alias with the same arguments and result
- fileInput: openFileInputStream / resolveFileInputEntry expose the
ACL-checked FS read as a stream; loadFileInput wraps them
- EmailAttachment accepts a `path` the transport streams on its own
A video job that outlived its poll window, an SDK request that timed out,
or a Veo operation that finished with an error all reached the HTTP error
handler as plain Errors. Each became an unhandled 500 with critical
severity and paged on-call for what is the provider's pace or the
provider's fault.
Video providers now share one poll loop that gives up with a 504
`upstream_timeout`, treats a transient poll failure (timeout, dropped
connection, 408/429/5xx) as a missed poll rather than a failed job, and
stops polling with a 400 `client_aborted` when the caller disconnects, so
nothing is metered for a clip nobody will receive. The driver controller
exposes the disconnect as an `abortSignal` on the request context. The
window is ten minutes for every provider; Together and BytePlus move up
from five.
Failed jobs are classified: content-filter refusals become a 400
`bad_request` with `errorCode: moderation_flagged`, rejected parameters a
400 `upstream_bad_request`, and anything else a 502 `upstream_failed`,
each carrying the provider's own code. Veo's filtered output keeps
`disallowed_value` and gains the same `errorCode`. The sanitizer and
content-filter pattern move from the Replicate provider into a shared
util so image and video agree.
Status-less SDK connection timeouts are translated to a 504
`upstream_timeout` at the driver boundary, and the chat driver records
them per attempt so an all-timeout chain is a 504 and a mixed chain is
`upstream_failed` instead of an `internal_error` 500. The Together chat
client gets the same ten-minute request timeout as the other providers.
The OpenAI video provider is left alone beyond an import path: its API is
scheduled to shut down on 2026-09-24.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
OpenRouter and Together reject a request whose prompt plus max_tokens
overflows the model's context window. Both providers retried by deleting
max_tokens, which threw away the output cap the credit gate had sized to
the caller's remaining balance and let the retry run to the model's full
output limit with no second gate and no new hold.
The retry now goes through a shared helper that sizes a new cap from the
window and input count the rejection reports, falling back to the model's
declared context and a doubled prompt estimate, and never exceeds the cap
the gate set. When no window can be determined or no output fits, the
original rejection is rethrown instead of retrying uncapped. The rejected
params are copied rather than mutated, so the first attempt's record is
not rewritten after the fact.
Also corrects the estimator's own comment, which described the mean of
two approximations as a deliberate halving.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Covers `loadFileInput` (validation, data-URL decoding, FS-path resolution,
maxBytes enforcement, ACL gating) and `inferFilenameFromUrlOrPath`. Stubs
stores/fsService with prefab data so tests don't require a real server.
Closes#2957
