LLM
Stream LLM responses over SSE via API
With stream: true in POST /llm/chat/completions, Connect answers as text/event-stream, one data: line per piece and data: [DONE] at the end, as in OpenAI. The OpenAI SDK stream works without adaptation, with OpenAI, Anthropic and Meta models.
The request
POST /api/llm/chat/completions
Authorization: Bearer <merchant token>
{
"model": "anthropic/claude-sonnet-5-5",
"max_tokens": 1024,
"stream": true,
"stream_options": {
"include_usage": true
},
"messages": [
{
"role": "system",
"content": "You summarize marketing data in English."
},
{
"role": "user",
"content": "Summarize the account's week."
}
]
}What you can do
- Show the answer to the user while it is being generated.
- Ask for usage in the stream with stream_options: {"include_usage": true}.
- Use the same stream code for all three providers.
- Combine stream with fallback (models) and tools on the models that support them.
Rules and limits
- With stream: true the response is text/event-stream: one data: {chunk} line per piece and data: [DONE] at the end.
- The chunk with usage only comes if you send stream_options: {"include_usage": true}.
- Without stream, the response may start with whitespace while the model generates; whitespace before the JSON is still valid JSON and the OpenAI SDK reads it normally.
- An error after the response started does not change the 200 status: in the stream it arrives as a data: {"error": {...}} event and the stream ends without data: [DONE]. Check error before reading choices.
- Timeouts: 30s to the first token, 60s without a new token and 300s in total (llm_timeout, 504).
Main fields
| Field | Type | Required | Description |
|---|---|---|---|
| model | string | Yes | Model id from GET /llm/models, in provider/model form. |
| messages | array | Yes | The conversation, as in OpenAI. role: system, developer, user, assistant or tool. |
| max_tokens | integer | Yes | Required (or max_completion_tokens). Up to the account cap and each model's max_output_tokens in the call. |
| stream | boolean | No | With true, the response is text/event-stream. |
| stream_options.include_usage | boolean | No | With true, the last chunk carries usage (tokens). |
Who it is for
Chat inside your SaaS
Your product's assistant writes the answer in real time, without the user waiting for the whole reply.
Agent interfaces
Show an agent's partial text while it works.
Long report generation
Display a report summary as the text is generated, instead of a loading screen.
Frequently asked questions
How do I turn streaming on?
Send stream: true in the body of POST /llm/chat/completions. The response becomes text/event-stream, with data: [DONE] at the end.
How do I get token usage in the stream?
Send stream_options: {"include_usage": true}. The chunk with usage only comes if you ask for it.
What happens if there is an error mid-stream?
The 200 status has already gone out, so the error arrives as a data: {"error": {...}} event and the stream ends without data: [DONE]. The OpenAI SDK raises the error.
See Connect applied to your product.
Try it for 15 days with no card, or walk our team through your use case.
