LiteLLM: Multi-Provider Routing, Load Balancing, Failover, and Cost Tracking
- 1 hour ago
- 6 min read
LiteLLM does the same job OpenRouter does — one API for every LLM provider — but hands you the router itself instead of renting you access to one.
........
LiteLLM ships as two things: a Python SDK for routing and cost tracking inside your own code, and a Proxy Server that runs as a standalone gateway with an OpenAI-compatible endpoint.
Five routing strategies are available: simple-shuffle (the default), least-busy, latency-based, usage-based, and cost-based — the last two need Redis to track state across requests.
Failover runs on an order field: lower numbers get tried first, each order level gets its own retries, and the router only escalates to the next order level — or to a separate fallbacks list — once a level is exhausted.
Cost tracking is built in: LiteLLM calculates token usage with provider-specific tokenizers and looks up real-time pricing to compute exact USD spend on every call, across every provider.
Unlike OpenRouter, LiteLLM's core is open source and self-hosted — there's no per-request platform fee, but the team running it owns deployment, security, and uptime instead of a vendor.
··········
TWO WAYS TO RUN IT.
LiteLLM isn't one product — it's a library and a server, aimed at two different points in an application's life.
The Python SDK is a client you import directly into your code. It gives you routing and load balancing across deployments without standing up any separate infrastructure — a Router object you configure with a model_list and call like any other LLM client.
The Proxy Server is a standalone gateway: an OpenAI-compatible HTTP endpoint that centralizes access control, spend tracking, and rate limiting for every application in an organization, not just the one that imported the library.
The practical split: reach for the SDK inside a single application that wants multi-provider resilience without extra infrastructure, and reach for the Proxy when multiple teams or services need to share one governed gateway with its own keys and budgets.
··········
HOW ROUTING DECISIONS GET MADE.
LiteLLM exposes five routing strategies, and which one applies changes the shape of the load-balancing decision entirely.
simple-shuffle is the default: it shuffles requests across deployments and distributes them by requests-per-minute capacity or configured weight.
least-busy sends each request to whichever deployment currently has the fewest ongoing requests — a live concurrency measure rather than a static weight.
latency-based-routing tracks response times and favors the deployment currently answering fastest.
usage-based-routing and cost-based-routing both need Redis, because they depend on caching state across requests: usage-based picks the deployment with the lowest current tokens-per-minute consumption, and cost-based picks whichever deployment is currently cheapest for that model.
··········
LiteLLM routing strategies
Strategy | What it optimizes for | Extra setup |
|---|---|---|
simple-shuffle (default) | Even distribution by RPM/weight | None |
least-busy | Lowest current concurrency | None |
latency-based-routing | Fastest response time | None |
usage-based-routing | Lowest current TPM usage | Redis |
cost-based-routing | Lowest current price | Redis |
··········
FAILOVER: ORDER LEVELS, THEN FALLBACKS.
LiteLLM's failover model is built around an explicit priority field rather than an implicit price-based weighting.
Setting order in litellm_params on each deployment controls priority directly — lower numbers are tried first. When a request to an order=1 deployment fails (connection error, 404, 429, or similar), the router automatically moves to order=2, then order=3, and so on.
Each order level gets its own retry budget before the router escalates to the next level. Only once every order level is exhausted does the router fall through to a separately configured fallbacks list, which points to an entirely different model group rather than another deployment of the same model.
A newer option, enable_weighted_failover, changes that escalation order: instead of jumping straight to a different model group on failure, the router first retries inside the same model group by re-picking among the remaining weighted deployments, and only crosses over to fallbacks once every deployment in that group has been tried. That's specifically useful for regional redundancy — an Azure East US deployment failing over to Azure Sweden Central before the router gives up on Azure altogether.
429 rate-limit errors get special handling: the failing deployment is placed on cooldown immediately, rather than waiting for the normal retry count to be exhausted first.
··········
A REAL FAILOVER CHAIN, END TO END.
A production config typically stacks several fallback tiers rather than relying on one.
A representative chain: Azure OpenAI in one region as the primary, the same model in a second Azure region as the first failover, direct OpenAI as a second failover if both Azure regions are unavailable, and a different provider entirely — Anthropic's Claude, for instance — as the last-resort backup.
That shape — same provider, different region, then different provider, then different model family — reflects a general pattern: exhaust cheaper and more similar failure recovery options before falling back to something that changes behavior or cost meaningfully.
··········
COST TRACKING AND BUDGETS.
Every response LiteLLM handles carries a computed cost, not just token counts — that's the feature most reviewers highlight as its strongest differentiator against a plain SDK wrapper.
LiteLLM calculates token usage using each provider's own tokenizer, then looks up live pricing to compute an exact USD figure per call — so an application can report real spend across OpenAI, Azure, Anthropic, and everything else through one consistent number, not provider-specific estimates.
Budgets can be enforced at the deployment level directly: a deployment that has exhausted its configured spend limit is removed from the eligible routing set until its budget window resets, and rejoins automatically once it does. Combined with cost-based routing, that gives both a soft preference toward cheap deployments and a hard ceiling on what any single one can spend.
Above the deployment level, LiteLLM supports budgets and rate limits scoped to individual API keys, users, teams, or arbitrary tags — the mechanism reviewers point to for internal platform teams that need to attribute AI spend back to the project or customer that generated it.
··········
VIRTUAL KEYS: THE ACCESS-CONTROL LAYER.
Sharing one provider API key across a team is a common failure pattern LiteLLM is specifically built to avoid.
Virtual keys let each team member, project, or customer get their own credential, scoped to its own budget and rate limit, without ever touching the real provider keys underneath.
Because the proxy exposes an OpenAI-compatible endpoint, adopting it doesn't require rewriting application code — pointing an existing OpenAI client at the LiteLLM proxy's base URL, with a virtual key instead of the real one, is usually the entire migration.
Guardrails can sit in front of the routing decision too, so a request can be blocked or modified before it reaches an expensive model, rather than being caught only after the fact in a spend report.
··········
ROUTING GROUPS: MIXING STRATEGIES IN ONE DEPLOYMENT.
A single Router doesn't have to use one strategy for everything — LiteLLM supports assigning different routing strategies to different sets of models within the same configuration.
A routing group assigns a strategy to a named set of model_names — for example, routing premium chat models by latency while routing cheap, high-volume models by cost, inside the same proxy deployment.
Grouping two different models under the same routing group does not let one substitute for the other. It only means both are load-balanced by the same strategy, each within its own separate pool of deployments — a GPT-4o request never gets silently answered by Claude just because the two share a routing group.
That's a distinct concept from a model access group, which is a pure access-control feature — grouping models under one name so a key or team can be granted access to the whole set in a single grant, unrelated to how those models are routed.
··········
HOW LITELLM COMPARES TO A HOSTED ROUTER LIKE OPENROUTER.
The two solve the same category of problem from opposite directions: one is infrastructure you run, the other is a service you call.
LiteLLM's core is open source, and running it costs infrastructure and operations time rather than a per-request platform fee — the tradeoff is that a team has to deploy, secure, monitor, and upgrade the proxy itself unless it pays for LiteLLM's own enterprise or managed offering.
OpenRouter's routing is mostly built-in and configured through request parameters rather than a self-hosted config file — faster to start with, harder to customize at the level LiteLLM's Python SDK allows, since LiteLLM's routing logic can be fully scripted rather than only configured.
The practical decision point: teams that want to own their infrastructure, need deep customization of routing logic, or have strict data-residency requirements tend toward LiteLLM; teams that want to start routing across providers today without operating anything themselves tend toward a hosted router.
··········
PRACTICAL SETUP GUIDANCE.
A production-grade LiteLLM configuration tends to converge on a small set of settings regardless of the specific providers involved.
Start with simple-shuffle unless there's a specific, measured reason to move to latency-, usage-, or cost-based routing — LiteLLM's own documentation recommends it as the default for the best general performance.
Set explicit order values on deployments that have a genuine priority relationship — same-region primary before cross-region backup, cheaper deployment before more expensive one — rather than leaving priority to chance.
Add Redis once the setup needs usage- or cost-based routing, or needs routing state to be shared consistently across more than one running proxy instance.
Scope virtual keys and budgets from day one rather than retrofitting them later — the migration cost of adding access control to a proxy already in production is higher than configuring it at setup.
Treat the fallbacks list as the last resort it's designed to be: order-based retries inside a model group should absorb most transient failures before a request ever needs to cross over to a different model entirely.
··········
·····
FOLLOW US FOR MORE.
·····
·····
DATA STUDIOS
·····
