Designing for Model Failover: Lessons from Production LLM Traffic

Every team that ships an LLM-backed product eventually learns the same lesson: model providers fail, and they fail in ways that traditional infrastructure playbooks don't fully cover. The team behind Runix has spent years running high-volume LLM traffic through gateways in production, and most of what we know about failover we learned by watching real traffic misbehave. This post is a condensed version of those lessons.

The real failure taxonomy

It helps to be precise about what "the provider is down" actually means, because each failure class needs a different response.

Hard errors. These are the easy ones: 429 rate limits, 500/502/503 from the provider, connection resets. They are unambiguous, cheap to detect, and safe to act on immediately. A 429 means back off or route elsewhere; a 5xx means the request never produced value and can usually be retried somewhere else.

Timeouts. Harder than they look. LLM inference latency is highly variable by design — a long completion legitimately takes longer than a short one. A fixed 30-second timeout will misclassify healthy long generations as failures, and a generous 5-minute timeout will hold user connections open while a provider quietly falls over. In practice you want two separate timeouts: time-to-first-token, which is fairly stable per provider and a strong health signal, and a token-rate floor for streaming (if throughput drops below some tokens-per-second threshold mid-stream, something upstream is degrading even though bytes are still trickling in).

Silent degradation. This is the nastiest class, and the one that gets no attention until it bites you. The provider returns 200 OK, latency looks normal, and the output is wrong: completions truncated at a suspicious power-of-two token count, a model quietly serving degraded quality under load, malformed JSON from a model that reliably produced valid JSON yesterday, or empty tool-call arguments. None of your standard dashboards catch this. Your error rate is 0% while your product is broken.

Detecting silent degradation requires looking at the response body, not just the status code. Useful cheap checks: finish_reason distribution (a spike in length finishes when your traffic mix hasn't changed is a red flag), output-token histograms per model (truncation shows up as a wall in the distribution), JSON-parse success rate for structured-output routes, and refusal-phrase rates. These are aggregate signals — no single response proves anything, but a shifted distribution over ten minutes does.

Why naive retry makes outages worse

The default instinct — wrap the call in a retry loop — is exactly how you turn a provider brownout into a full outage. When a provider starts shedding load with 429s, every client retrying immediately multiplies the offered load right when capacity is lowest. The provider sheds harder, more clients retry, and you get a retry storm. You also burn your own capacity: threads, connections, and user patience are all held hostage by requests that were never going to succeed.

Three rules keep retries safe:

• Exponential backoff with full jitter. Without jitter, all clients that failed together retry together, producing synchronized load spikes. Randomize the entire backoff window, not just a small offset.
• A hard cap on attempts. Two or three, total, across all targets. If three attempts across two providers failed, a fourth attempt is far more likely to add load than to succeed.
• A retry budget. Cap retries as a fraction of overall traffic — for example, retries may not exceed 10% of request volume in any window. When the budget is exhausted, fail fast instead. This is the mechanism that prevents a partial outage from amplifying into a total one, and it is the single highest-leverage change most teams haven't made.

Also decide up front which requests are retryable at all. Idempotent completions are; anything with side effects (a tool call that already executed) is not, and replaying it is a correctness bug, not resilience.

Knowing when a provider is unhealthy

There are two sources of health signal, and you need both.

Passive signals come from live traffic: error rates, time-to-first-token percentiles, token throughput, and the silent-degradation aggregates above. They are free, they reflect your real workload (your prompts, your regions, your API features), and they are the fastest detector when you have volume. Their weakness is low-traffic routes — a model that gets ten requests an hour gives you statistics too thin to act on.

Active probes fill that gap: a small synthetic request per provider+model every 30–60 seconds, with a known-good expected output so you can verify content, not just status. Probes catch failures on cold routes before a user does, and they are the only way to confirm recovery without gambling user traffic. Their weakness is that a tiny canned prompt does not exercise long contexts, streaming, or tool use — so probes can pass while your real workload fails. Treat probes as a floor, passive signals as the truth.

Circuit breakers, per provider and per model

Once you can detect unhealth, you need a mechanism that acts on it faster than a human can. The classic circuit breaker works well here with one crucial adjustment: the breaker key must be provider + model, not just provider. Providers rarely fail uniformly — one model family degrades while others on the same API are fine. A provider-level breaker turns a partial failure into a self-inflicted total one.

# Per (provider, model) breaker — illustrative config
breaker:
  key: "{provider}:{model}"
  window: 60s
  trip_when:
    error_rate: "> 25%"          # hard errors + timeouts
    min_samples: 20              # don't trip on thin data
    or_probe_failures: 3         # consecutive active-probe failures
  open_for: 30s                  # then move to half-open
  half_open:
    trial_requests: 5            # real traffic, capped
    close_when: "5/5 succeed"

retry:
  max_attempts: 3                # total, across all targets
  backoff: exponential_full_jitter(base=250ms, cap=8s)
  budget: "10% of window traffic"

failover_order:
  - primary:   provider_a/model-large
  - secondary: provider_b/model-large-equivalent
  - degraded:  provider_a/model-small
  - fallback:  cached_or_honest_error

Two details matter in production. First, min_samples: tripping a breaker on three requests of thin data causes flapping, which is its own outage. Second, the half-open phase must use a small capped trickle of real traffic — recovery detected only by synthetic probes will happily close the breaker into a provider that still can't handle your actual workload.

Failover is not free

Routing around a failure sounds like a pure win. It isn't; three costs show up immediately.

Semantic drift. Models are not interchangeable. A prompt tuned on one model produces different tone, different formatting, and different tool-calling behavior on another. If your product parses model output, failover can break parsing even though every request "succeeded." The fix is to qualify fallback pairs in advance: run your evaluation suite against the fallback model with your production prompts, and only put pre-qualified pairs in the failover chain.

Prompt-cache warmth. If you rely on provider-side prompt caching for long shared prefixes, a failover means every request pays full input price and full prefill latency on the new target until its cache warms. Model this in your cost projections: a one-hour failover on a cache-heavy workload can cost several times its normal hourly spend. It is also a reason to fail over decisively rather than flapping — every flap re-pays the warm-up twice.

Cost asymmetry. Your fallback may cost 3–5x your primary per token. That is usually the right trade during an incident, but it should be a decision someone made ahead of time, with a spend alert attached, not a surprise on the invoice.

Graceful degradation tiers

Failover to an equivalent model is only the first rung. A complete design defines what happens when that fails too, in order of decreasing capability:

1. Equivalent model, different provider. Full capability, costs the trade-offs above.
2. Smaller model, same family. Most workloads tolerate a capability drop far better than an error page. Summarization, classification, and short-answer routes often degrade imperceptibly.
3. Cached or precomputed answer. For high-repetition queries, serving a slightly stale cached response labeled as such beats serving nothing.
4. An honest error. "The AI service is temporarily degraded, please retry shortly" — returned fast, with a working retry path. A fast honest failure preserves more user trust than a 90-second hang followed by a timeout.

The key property: each tier is a deliberate product decision made calmly in advance, not an improvisation at 3 a.m.

Test it with game days

A failover path that has never run is a failover path that does not work. The failure modes hide in the details: the fallback API key that expired, the secondary provider that rejects a request parameter the primary accepted, the breaker that trips correctly but whose alert routes to a channel nobody reads.

Run game days quarterly. Inject faults at the gateway layer — force 429s from the primary, add artificial time-to-first-token latency, truncate responses to simulate silent degradation — and verify the whole chain: breaker trips within the expected window, traffic shifts to the qualified fallback, output quality on the fallback passes your evals, cost alerts fire, and recovery closes the breaker without flapping. Measure time-to-detect and time-to-mitigate, and track them like the reliability metrics they are. The first game day is always humbling; that is the point of running it before your provider does it for you.

The takeaway

Treat model providers the way you treat any external dependency with a 99.9% ceiling: assume failure, detect it from both live traffic and probes, contain it with per-model breakers and retry budgets, and degrade through pre-qualified tiers instead of falling off a cliff. None of these mechanisms is exotic — the work is in the details, and the details only get validated by real traffic and deliberate practice.