LLM
Automatic fallback between LLM models via API
In POST /llm/chat/completions, the optional models field lists up to 3 models to try, in order, when model fails before the first token. The response says which model answered. You get a plan B on another provider without writing retry logic.
The request
POST /api/llm/chat/completions
Authorization: Bearer <merchant token>
{
"model": "openai/gpt-5.4-mini",
"models": [
"anthropic/claude-haiku-5-5"
],
"max_tokens": 512,
"messages": [
{
"role": "user",
"content": "What was yesterday's reach?"
}
]
}What you can do
- Set up to 3 backup models, including from another provider, such as openai/gpt-5.4-mini with anthropic/claude-haiku-5-5 as fallback.
- Know which model answered from the model field of the response.
- Combine fallback with stream and tools, as long as every model in the call supports what you asked for.
- Avoid downtime when a provider fails or is slow.
Rules and limits
- Fallback kicks in when model fails before sending the first token: provider error, limit or timeout. After the first token, the error reaches you.
- There is never a retry on the same model.
- models accepts at most 3 items, all from GET /llm/models; an id outside the list returns 400 llm_model_not_allowed.
- You need the key of every provider in the chain, model and models, otherwise 400 llm_credential_missing.
- tools and images only work if every model in the call has supports_tools and supports_vision; the fallback gets the same max_tokens, which must fit all of them.
Main fields
| Field | Type | Required | Description |
|---|---|---|---|
| model | string | Yes | Model id from GET /llm/models, in provider/model form. |
| models | array | No | Fallback: up to 3 models from GET /llm/models, tried in order if model fails before the first token. |
| messages | array | Yes | The conversation, as in OpenAI. role: system, developer, user, assistant or tool. |
| max_tokens | integer | Yes | Required (or max_completion_tokens). Up to the account cap and each model's max_output_tokens in the call. |
| model | string | No | In the response: the model that answered, model or one of models. |
Who it is for
An AI assistant that stays up
Your product keeps answering when a provider goes down, because the next model in the list takes over.
Cost first, quality as backup
Try a cheap model and keep a larger one as backup for when it fails.
Resilient agents
A long-running agent does not stop because of a one-off provider failure.
Frequently asked questions
When is the fallback triggered?
When model fails before sending the first token: provider error, limit or timeout. After the first token, the error goes to you and the model is not switched.
How do I know which model answered?
From the model field of the response: it carries model or one of models, if there was a fallback.
Can I use fallback with tools?
Yes, if every model in the call (model and each item of models) has supports_tools. If any lacks it, the call is refused with 400 llm_unsupported_field.
See Connect applied to your product.
Try it for 15 days with no card, or walk our team through your use case.
