mirror of
https://github.com/HeyPuter/puter.git
synced 2026-09-12 00:05:38 +00:00
fix(ai): route Mistral streamed thinking to the reasoning channel, unconditionally
Third triple-check round. Ten findings were put through independent skeptics first; six did not survive — pre-existing on main, inert, or resting on a false premise — and are not acted on here. Mistral streamed thinking now goes to `reasoning` on every path. The previous commit gated the split, which made this the only place in the repo where chain-of-thought reached the visible text channel, and made Mistral the only provider whose streamed chunk *types* depend on a response-format flag. Every other reasoning path routes thinking to `reasoning` unconditionally — ClaudeProvider's thinking_delta, the DeepSeek/OpenRouter rename, and this branch's own Responses summary-delta handler. Removing the gate restores that uniformity and makes the documented promise that streaming is unaffected by normalization true again; the two opposing tests collapse into one that runs the same fixture with and without `normalize` and asserts identical event streams. Docs stop claiming older models are unchanged. Four reasoning fields were made consistent across all models, ungated, and one of them removes a field: on non-streaming responses `message.reasoning_content` is now `message.reasoning`. chat.md gains a table naming all four so a caller reading `reasoning_content` learns why it disappeared, instead of reading that nothing changed for them. Adds the driver-level fallback test. Writing it surfaced that the invariant it was meant to assert is false and always was: the driver rewrites string `content` into text blocks in place on the caller's own messages (`normalize_single_message`, pre-existing) before any provider runs. The test now asserts what is true and load-bearing — both attempts receive the same array reference, and the reasoning artifacts survive attempt 1 so attempt 2 can still replay them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
404oops
@@ -1150,6 +1150,71 @@ describe('ChatCompletionDriver.complete fallback and error envelope', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('hands every fallback attempt the same messages array, reasoning artifacts intact', async () => {
|
||||
// The hazard the providers' copy-on-write exists for. `complete` is
|
||||
// called per attempt with `{ ...args }` — a shallow spread — so
|
||||
// `args.messages` is the SAME array reference on every attempt. A
|
||||
// provider that strips replay fields in place therefore hands attempt 2
|
||||
// a message whose thinking signature is gone, and Anthropic rejects
|
||||
// that continuation.
|
||||
//
|
||||
// Note what this does NOT assert: the caller's message objects are not
|
||||
// pristine. `normalize_single_message` (utils/Messages.js, pre-existing)
|
||||
// rewrites string `content` into `[{type:'text'}]` blocks in place on
|
||||
// every inbound message before any provider runs. The invariant that
|
||||
// matters here is narrower and is the one the fix delivers: the
|
||||
// reasoning artifacts survive attempt 1 so attempt 2 can still replay
|
||||
// them.
|
||||
const freeRoute = {
|
||||
costs_currency: 'usd-cents',
|
||||
costs: { 'input-tokens': 0, 'output-tokens': 0 },
|
||||
max_tokens: 8192,
|
||||
};
|
||||
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
|
||||
{ id: 'route-a', aliases: ['shared-id'], ...freeRoute },
|
||||
{ id: 'route-b', aliases: ['shared-id'], ...freeRoute },
|
||||
] as never);
|
||||
const d = await makeDriver();
|
||||
|
||||
const completeSpy = vi
|
||||
.spyOn(FakeChatProvider.prototype, 'complete')
|
||||
.mockRejectedValueOnce(new Error('first route down'))
|
||||
.mockResolvedValueOnce({
|
||||
message: { role: 'assistant', content: 'from the fallback' },
|
||||
usage: {},
|
||||
finish_reason: 'stop',
|
||||
} as never);
|
||||
|
||||
const details = [
|
||||
{ type: 'thinking', thinking: 'step one', signature: 'sig_1' },
|
||||
];
|
||||
const messages = [
|
||||
{ role: 'user', content: 'hi' },
|
||||
{
|
||||
role: 'assistant',
|
||||
content: 'earlier reply',
|
||||
reasoning: 'step one',
|
||||
refusal: null,
|
||||
reasoning_details: details,
|
||||
},
|
||||
];
|
||||
|
||||
await withTestActor(() =>
|
||||
d.complete({ model: 'shared-id', messages: messages as never }),
|
||||
);
|
||||
|
||||
// Two attempts actually ran, which is what makes the reference shared.
|
||||
expect(completeSpy).toHaveBeenCalledTimes(2);
|
||||
expect(completeSpy.mock.calls[0]![0].messages).toBe(
|
||||
completeSpy.mock.calls[1]![0].messages,
|
||||
);
|
||||
// The replay material survived attempt 1 and reached attempt 2 intact.
|
||||
const secondAttempt = completeSpy.mock.calls[1]![0].messages as Array<
|
||||
Record<string, unknown>
|
||||
>;
|
||||
expect(secondAttempt[1]!.reasoning_details).toEqual(details);
|
||||
});
|
||||
|
||||
it('re-reads the balance between fallback attempts so a parallel request that drains the wallet aborts the chain', async () => {
|
||||
// The primary provider throws; the fallback loop runs the full gate
|
||||
// (one balance read per attempt) before its next upstream hit. We
|
||||
|
||||
@@ -680,146 +680,98 @@ describe('MistralAIProvider.complete streaming', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('flattens chunked delta.content into text and reasoning events', async () => {
|
||||
const { provider } = makeProvider();
|
||||
streamMock.mockReturnValueOnce(
|
||||
asAsyncIterable([
|
||||
{
|
||||
data: {
|
||||
choices: [
|
||||
{
|
||||
delta: {
|
||||
content: [
|
||||
{
|
||||
type: 'thinking',
|
||||
thinking: [
|
||||
{
|
||||
type: 'text',
|
||||
text: 'thinking…',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
it.each([
|
||||
['normalize: true', true],
|
||||
['normalize unset', undefined],
|
||||
])(
|
||||
'splits chunked delta.content into text and reasoning events (%s)',
|
||||
async (_label, normalize) => {
|
||||
// The reasoning split is unconditional: streamed chunk types must
|
||||
// not depend on the normalize policy, because chat.md promises
|
||||
// "Streaming is unaffected by normalization" and because every
|
||||
// other provider routes thinking to the reasoning channel
|
||||
// regardless. Both rows below assert the same event stream.
|
||||
const { provider } = makeProvider();
|
||||
streamMock.mockReturnValueOnce(
|
||||
asAsyncIterable([
|
||||
{
|
||||
data: {
|
||||
choices: [
|
||||
{
|
||||
delta: {
|
||||
content: [
|
||||
{
|
||||
type: 'thinking',
|
||||
thinking: [
|
||||
{
|
||||
type: 'text',
|
||||
text: 'thinking…',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
data: {
|
||||
choices: [
|
||||
{
|
||||
delta: {
|
||||
content: [
|
||||
{ type: 'text', text: 'answer' },
|
||||
],
|
||||
{
|
||||
data: {
|
||||
choices: [
|
||||
{
|
||||
delta: {
|
||||
content: [
|
||||
{ type: 'text', text: 'answer' },
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
data: {
|
||||
choices: [{ delta: {} }],
|
||||
usage: { promptTokens: 1, completionTokens: 1 },
|
||||
{
|
||||
data: {
|
||||
choices: [{ delta: {} }],
|
||||
usage: { promptTokens: 1, completionTokens: 1 },
|
||||
},
|
||||
},
|
||||
},
|
||||
]),
|
||||
);
|
||||
]),
|
||||
);
|
||||
|
||||
const result = await withTestActor(() =>
|
||||
provider.complete({
|
||||
model: 'magistral-small-latest',
|
||||
messages: [{ role: 'user', content: 'think' }],
|
||||
stream: true,
|
||||
normalize: true,
|
||||
}),
|
||||
);
|
||||
const result = await withTestActor(() =>
|
||||
provider.complete({
|
||||
model: 'magistral-small-latest',
|
||||
messages: [{ role: 'user', content: 'think' }],
|
||||
stream: true,
|
||||
...(normalize === undefined ? {} : { normalize }),
|
||||
}),
|
||||
);
|
||||
|
||||
const harness = makeCapturingChatStream();
|
||||
await (
|
||||
result as {
|
||||
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
|
||||
}
|
||||
).init_chat_stream({ chatStream: harness.chatStream });
|
||||
const harness = makeCapturingChatStream();
|
||||
await (
|
||||
result as {
|
||||
init_chat_stream: (p: {
|
||||
chatStream: unknown;
|
||||
}) => Promise<void>;
|
||||
}
|
||||
).init_chat_stream({ chatStream: harness.chatStream });
|
||||
|
||||
const events = harness.events();
|
||||
// The thinking chunk becomes a reasoning event, not stringified text.
|
||||
expect(
|
||||
events.filter((e) => e.type === 'reasoning').map((e) => e.reasoning),
|
||||
).toEqual(['thinking…']);
|
||||
expect(events.filter((e) => e.type === 'text').map((e) => e.text)).toEqual(
|
||||
['answer'],
|
||||
);
|
||||
});
|
||||
|
||||
it('never hands a chunk array to addText on the native path', async () => {
|
||||
// Splitting thinking into a `reasoning` delta is the dialect change and
|
||||
// is gated. Flattening the chunk array is not: the shared stream
|
||||
// handler passes `delta.content` straight to addText, so leaving an
|
||||
// array there would put stringified objects in the caller's text
|
||||
// stream. Native path keeps every chunk's text, thinking included.
|
||||
const { provider } = makeProvider();
|
||||
streamMock.mockReturnValueOnce(
|
||||
asAsyncIterable([
|
||||
{
|
||||
data: {
|
||||
choices: [
|
||||
{
|
||||
delta: {
|
||||
content: [
|
||||
{
|
||||
type: 'thinking',
|
||||
thinking: [
|
||||
{
|
||||
type: 'text',
|
||||
text: 'thinking…',
|
||||
},
|
||||
],
|
||||
},
|
||||
{ type: 'text', text: 'answer' },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
data: {
|
||||
choices: [{ delta: {} }],
|
||||
usage: { promptTokens: 1, completionTokens: 1 },
|
||||
},
|
||||
},
|
||||
]),
|
||||
);
|
||||
|
||||
const result = await withTestActor(() =>
|
||||
provider.complete({
|
||||
model: 'magistral-small-latest',
|
||||
messages: [{ role: 'user', content: 'think' }],
|
||||
stream: true,
|
||||
}),
|
||||
);
|
||||
|
||||
const harness = makeCapturingChatStream();
|
||||
await (
|
||||
result as {
|
||||
init_chat_stream: (p: { chatStream: unknown }) => Promise<void>;
|
||||
}
|
||||
).init_chat_stream({ chatStream: harness.chatStream });
|
||||
|
||||
const events = harness.events();
|
||||
// No reasoning channel on the native path...
|
||||
expect(events.some((e) => e.type === 'reasoning')).toBe(false);
|
||||
// ...and the text is text, not '[object Object]' or raw JSON.
|
||||
const text = events
|
||||
.filter((e) => e.type === 'text')
|
||||
.map((e) => e.text)
|
||||
.join('');
|
||||
expect(text).toBe('thinking…\n\nanswer');
|
||||
expect(text).not.toContain('object');
|
||||
expect(text).not.toContain('{');
|
||||
});
|
||||
const events = harness.events();
|
||||
// Thinking goes to the reasoning channel, never the text channel.
|
||||
expect(
|
||||
events
|
||||
.filter((e) => e.type === 'reasoning')
|
||||
.map((e) => e.reasoning),
|
||||
).toEqual(['thinking…']);
|
||||
const text = events
|
||||
.filter((e) => e.type === 'text')
|
||||
.map((e) => e.text)
|
||||
.join('');
|
||||
expect(text).toBe('answer');
|
||||
// And the array never reaches addText as an object.
|
||||
expect(text).not.toContain('object');
|
||||
expect(text).not.toContain('{');
|
||||
},
|
||||
);
|
||||
|
||||
it('builds a tool_use block from camelCase delta.toolCalls deltas', async () => {
|
||||
const { provider } = makeProvider();
|
||||
|
||||
@@ -191,9 +191,16 @@ export class MistralAIProvider implements IChatProvider {
|
||||
// policy resolution the driver's coercer uses rather than firing on
|
||||
// every Mistral call — a caller reading the SDK's native
|
||||
// `finishReason`/`toolCalls` keys keeps seeing them unless it asked
|
||||
// for the equalized shape. (The streaming half is not gated: feeding a
|
||||
// chunk array to `addText` is a plain bug, and streamed chunks are
|
||||
// provider-uniform by design.)
|
||||
// for the equalized shape.
|
||||
//
|
||||
// Streaming is deliberately NOT gated on this, and the deviation below
|
||||
// is uniform in both directions: streamed chunks are provider-uniform
|
||||
// by design, and every other reasoning path in this repo routes
|
||||
// thinking to the `reasoning` channel unconditionally (ClaudeProvider's
|
||||
// thinking_delta, the DeepSeek/OpenRouter rename in
|
||||
// `create_chat_stream_handler`, the Responses summary-delta handler).
|
||||
// Gating it would make Mistral the only provider whose streamed chunk
|
||||
// *types* depend on a response-format flag.
|
||||
const presentAsOpenAI = shouldPresentAsOpenAI(
|
||||
{ normalize, response },
|
||||
selectedModel.release_date,
|
||||
@@ -276,16 +283,19 @@ export class MistralAIProvider implements IChatProvider {
|
||||
|
||||
return snake_usage;
|
||||
},
|
||||
// Mistral wraps each event; unwrap it, then flatten a
|
||||
// reasoning model's chunked `delta.content`.
|
||||
// Mistral wraps each event; unwrap it, then split a
|
||||
// reasoning model's chunked `delta.content` into the two
|
||||
// channels the shared handler already understands: visible
|
||||
// text, and `reasoning`.
|
||||
//
|
||||
// Two concerns, gated differently. The shared stream handler
|
||||
// passes `delta.content` straight to `addText`, so a chunk
|
||||
// array would reach the caller as stringified objects —
|
||||
// flattening it to text is a correctness floor that applies on
|
||||
// every path. Splitting thinking out into a `reasoning` delta
|
||||
// is the dialect change, and that sits behind the policy gate
|
||||
// like the non-streaming remap.
|
||||
// Both halves are unconditional. Leaving the array on
|
||||
// `delta.content` would hand it to `addText` and reach the
|
||||
// caller as stringified objects, and putting the thinking text
|
||||
// into the visible channel would make this the only place in
|
||||
// the repo where chain-of-thought is answer text. So the split
|
||||
// matches every other provider and does not depend on the
|
||||
// normalize policy — streamed chunk types stay identical
|
||||
// whichever way that resolves.
|
||||
chunk_but_like_actually: (chunk: unknown) => {
|
||||
const data = (chunk as { data?: unknown }).data as
|
||||
| {
|
||||
@@ -301,15 +311,6 @@ export class MistralAIProvider implements IChatProvider {
|
||||
const { text, reasoning } = splitMistralContentChunks(
|
||||
delta.content,
|
||||
);
|
||||
if (!presentAsOpenAI) {
|
||||
// Native path: keep every chunk's text, thinking
|
||||
// included, so nothing is silently dropped — but
|
||||
// never hand an array to `addText`.
|
||||
delta.content = reasoning
|
||||
? [reasoning, text].filter(Boolean).join('\n\n')
|
||||
: text;
|
||||
continue;
|
||||
}
|
||||
delta.content = text;
|
||||
if (reasoning && delta.reasoning === undefined) {
|
||||
delta.reasoning = reasoning;
|
||||
|
||||
+16
-3
@@ -35,7 +35,7 @@ An object containing the following properties:
|
||||
- `tools` (Array) (Optional) - Function definitions the AI can call. See [Function Calling](#function-calling) for details.
|
||||
- `reasoning_effort` / `reasoning.effort` (String) (Optional) - Controls how much effort reasoning models spend thinking. Supported values: `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Lower values give faster responses with less reasoning. OpenAI models and Meta's Muse Spark models only; Muse Spark always reasons, so `none` is ignored for it.
|
||||
- `verbosity` / `text.verbosity` (String) (Optional) - Controls how long or short responses are. Supported values: `low`, `medium`, and `high`. Lower values give shorter responses. OpenAI models only.
|
||||
- `normalize` (Boolean) (Optional) - Controls the format of the non-streaming response. When `true`, the response is normalized to the OpenAI format regardless of the model's vendor: `message.content` is a string, tool calls appear as `message.tool_calls`, and `finish_reason` is one of `stop`, `length`, `tool_calls`, or `content_filter` — or the vendor's own stop reason, passed through unchanged when it has no OpenAI equivalent. When `false`, the response keeps the vendor's native format (for Anthropic models, an array of content blocks). When unset, `puter.ai.normalize` applies if you assigned it; otherwise **models released on or after September 1, 2026 return normalized (OpenAI-format) responses by default**, and older models keep their current behavior. Streaming responses are unaffected — chunks already share one format across vendors. See [Response normalization](#response-normalization).
|
||||
- `normalize` (Boolean) (Optional) - Controls the format of the non-streaming response. When `true`, the response is normalized to the OpenAI format regardless of the model's vendor: `message.content` is a string, tool calls appear as `message.tool_calls`, and `finish_reason` is one of `stop`, `length`, `tool_calls`, or `content_filter` — or the vendor's own stop reason, passed through unchanged when it has no OpenAI equivalent. When `false`, the response keeps the vendor's native format (for Anthropic models, an array of content blocks). When unset, `puter.ai.normalize` applies if you assigned it; otherwise **models released on or after September 1, 2026 return normalized (OpenAI-format) responses by default**, and for older models the default is unchanged — `message.content` keeps its vendor-native shape. (A handful of reasoning fields were made consistent across all models independently of this option; see [Reasoning fields on existing models](#reasoning-fields-on-existing-models).) Streaming responses are unaffected — chunks already share one format across vendors. See [Response normalization](#response-normalization).
|
||||
- `compaction` (Boolean | Object) (Optional) - Opt into inline context compaction for long conversations. Pass `true` to enable it with provider defaults, or `{ trigger_tokens: number }` to set the token threshold at which earlier context is summarized. When the model compacts, you receive a `compaction` chunk while streaming (or a `compaction` field on the result when not streaming) containing an opaque `encrypted_content` summary. Resend that item in `messages` on the next turn in place of the summarized history. The compaction chunk shape is identical across providers, so the same code works whether `model` is an OpenAI or Anthropic model. See [Compaction](#compaction).
|
||||
|
||||
#### `testMode` (Boolean) (Optional)
|
||||
@@ -119,7 +119,20 @@ We use different vendors for different models and try to use the best vendor ava
|
||||
|
||||
Most vendors respond in the OpenAI chat format, where `message.content` is a string and tool calls appear as `message.tool_calls`. Anthropic models historically respond in Anthropic's native format instead, where `message.content` is an array of content blocks such as `[{ type: "text", text: "..." }]`.
|
||||
|
||||
**Going forward, all models released on or after September 1, 2026 return responses in the OpenAI format**, no matter which vendor serves them — so the same response-handling code works across every new model. Models released before that date keep their historical behavior unless you opt in.
|
||||
**Going forward, all models released on or after September 1, 2026 return responses in the OpenAI format**, no matter which vendor serves them — so the same response-handling code works across every new model. For models released before that date, the `normalize` default does not change: leave the option unset and `message.content` keeps its vendor-native shape.
|
||||
|
||||
### Reasoning fields on existing models
|
||||
|
||||
Separately from the `normalize` default, four reasoning-related fields were made consistent across vendors. These apply to **every** model, including ones released before the cutoff, and are not affected by `normalize`:
|
||||
|
||||
| Field | Before | Now |
|
||||
| --- | --- | --- |
|
||||
| `message.reasoning_content` | Present on providers following the DeepSeek convention (DeepSeek, OpenRouter and others) | **Renamed to `message.reasoning`.** Read `reasoning` instead — `reasoning_content` is no longer present on non-streaming responses. |
|
||||
| `message.reasoning` on OpenAI Responses models | Always present as `null` | Absent when the model returned no reasoning summary; a string when it did. `if (msg.reasoning)` is unaffected; `'reasoning' in msg` changes. |
|
||||
| `message.reasoning_details` on OpenAI Responses models | Not present | Present when the model returned reasoning items, carrying their `id` and `encrypted_content` for replay. |
|
||||
| `finish_reason` on OpenAI Responses models | Always `"stop"` | `"tool_calls"` when the turn ended in tool calls, `"stop"` otherwise. |
|
||||
|
||||
If your code reads `message.reasoning_content` on a non-streaming response, that is the one change that removes a field — switch to `message.reasoning`.
|
||||
|
||||
You can control this per call with the `normalize` option:
|
||||
|
||||
@@ -150,7 +163,7 @@ One caveat. The release-date rule applies to the model that actually serves the
|
||||
|
||||
One thing to know about the release-date rule: a model's release date comes from the catalog of whichever provider serves it, and some providers report it from their own live listing. Models served through OpenRouter carry the date OpenRouter itself assigns, so a model newly listed there on or after September 1, 2026 is normalized by default without Puter shipping any change. Pin `normalize: false` if your code depends on a provider's native shape.
|
||||
|
||||
Streaming is unaffected by normalization: streamed [`ChatResponseChunk`](/Objects/chatresponsechunk) objects already share one format across all vendors.
|
||||
Streaming is unaffected by normalization: streamed [`ChatResponseChunk`](/Objects/chatresponsechunk) objects already share one format across all vendors, and the chunk types a model emits do not depend on the `normalize` option. Reasoning models stream their thinking as `reasoning` chunks on every provider, whether or not normalization applies.
|
||||
|
||||
## Function Calling
|
||||
|
||||
|
||||
Reference in New Issue
Block a user