What for: one OpenAI-compatible proxy in front of 100+ LLM providers — self-hosted, with keys, limits and cost tracking.

What it is and who it's for

LiteLLM is an open-source library and proxy server that puts 100+ LLM providers behind a single OpenAI-style format. It's for platform and backend engineers who need a self-hosted gateway with keys, budgets, limits and per-team cost tracking. What sets it apart: it's infrastructure under your own control (not a SaaS), with an admin UI, virtual keys and logging into your own database.

Key capabilities

One completion() call against OpenAI, Anthropic, Bedrock, Vertex, Azure, Ollama and hundreds more.
LiteLLM Proxy (LLM Gateway): virtual keys, budgets, rate limits, teams and tags.
Spend tracking per key/user/project, logging to Postgres, Langfuse, Datadog.
Fallbacks, retries, load balancing and response caching.
Self-hosted via Docker/Helm — your data and keys stay with you.
OpenAI SDK compatible: clients switch over by changing the base URL.

Get started in 5 minutes

As a library — install it and call any provider through one interface:

pip install litellm

from litellm import completion
r = completion(model="anthropic/claude-3-5-sonnet-20240620", messages=[{"role":"user","content":"Hi"}])

As a proxy — bring up the gateway with a model config:

litellm --model gpt-4o --port 4000

Point your clients at http://localhost:4000 as if it were a regular OpenAI endpoint.

When to use it and when not to

✅ Use it if you need a self-hosted gateway with per-team billing, virtual keys and logs in your own database.
✅ Use it if you're building an internal LLM platform for several teams with quotas and an audit trail.
❌ Skip it → go with "OpenRouter" if you don't want to stand up and maintain infrastructure and just need a managed gateway.
❌ Skip it → go with "Vercel AI Gateway" if you're on Vercel and want a managed option with no DevOps.

The honest price

The open-source core and proxy are free (you pay providers for tokens plus your own hosting). There's a paid LiteLLM Enterprise tier (SSO, SLA, premium features). Exact figures are on the provider's site and change over time.

Pitfalls

This is infrastructure: you need DevOps to deploy, update and monitor the proxy and its Postgres.
Broad compatibility means a common denominator — exotic provider-specific features may need manual configuration.
Virtual keys and budgets have to be configured correctly, otherwise your cost accounting will be off.
Under heavy load the proxy itself becomes the bottleneck — plan for scaling.

🤖 Prompt accelerator

"I'm deploying LiteLLM Proxy on <Docker/Kubernetes> for a team of . Help me write config.yaml: the model list , virtual keys with per-team budgets, fallbacks and spend logging to Postgres. Give me the startup commands and an example of how an OpenAI SDK client switches over to my proxy."