LLM

Automatic fallback between LLM models via API

In POST /llm/chat/completions, the optional models field lists up to 3 models to try, in order, when model fails before the first token. The response says which model answered. You get a plan B on another provider without writing retry logic.

The request

POST /api/llm/chat/completions
Authorization: Bearer <merchant token>

{
  "model": "openai/gpt-5.4-mini",
  "models": [
    "anthropic/claude-haiku-5-5"
  ],
  "max_tokens": 512,
  "messages": [
    {
      "role": "user",
      "content": "What was yesterday's reach?"
    }
  ]
}

What you can do

  • Set up to 3 backup models, including from another provider, such as openai/gpt-5.4-mini with anthropic/claude-haiku-5-5 as fallback.
  • Know which model answered from the model field of the response.
  • Combine fallback with stream and tools, as long as every model in the call supports what you asked for.
  • Avoid downtime when a provider fails or is slow.

Rules and limits

  • Fallback kicks in when model fails before sending the first token: provider error, limit or timeout. After the first token, the error reaches you.
  • There is never a retry on the same model.
  • models accepts at most 3 items, all from GET /llm/models; an id outside the list returns 400 llm_model_not_allowed.
  • You need the key of every provider in the chain, model and models, otherwise 400 llm_credential_missing.
  • tools and images only work if every model in the call has supports_tools and supports_vision; the fallback gets the same max_tokens, which must fit all of them.

Main fields

FieldTypeRequiredDescription
modelstringYesModel id from GET /llm/models, in provider/model form.
modelsarrayNoFallback: up to 3 models from GET /llm/models, tried in order if model fails before the first token.
messagesarrayYesThe conversation, as in OpenAI. role: system, developer, user, assistant or tool.
max_tokensintegerYesRequired (or max_completion_tokens). Up to the account cap and each model's max_output_tokens in the call.
modelstringNoIn the response: the model that answered, model or one of models.

Who it is for

  • An AI assistant that stays up

    Your product keeps answering when a provider goes down, because the next model in the list takes over.

  • Cost first, quality as backup

    Try a cheap model and keep a larger one as backup for when it fails.

  • Resilient agents

    A long-running agent does not stop because of a one-off provider failure.

Frequently asked questions

When is the fallback triggered?

When model fails before sending the first token: provider error, limit or timeout. After the first token, the error goes to you and the model is not switched.

How do I know which model answered?

From the model field of the response: it carries model or one of models, if there was a fallback.

Can I use fallback with tools?

Yes, if every model in the call (model and each item of models) has supports_tools. If any lacks it, the call is refused with 400 llm_unsupported_field.

Related

See everything you can do with LLM

See Connect applied to your product.

Try it for 15 days with no card, or walk our team through your use case.