Self-Hosting an LLM Gateway with LiteLLM

At some point I noticed I was pasting the same API key into a fourth tool. Different config file, same key, and no real idea what any of the other tools were spending. One was calling GPT-4o for simple formatting tasks. Another had no caching, so identical prompts were hitting the API fresh every time. A runaway workflow almost wiped out a month's credits in a weekend.

I needed something between my tools and the providers. Not another library to import, but an actual service running on my network that I could point everything at. One endpoint, one set of keys, one place to see what was happening.

LiteLLM is what I ended up with. This is how I set it up and why each piece matters.

The problem

The naive approach is fine for one tool. You paste your API key, you call the provider, it works. Then you add a second tool, a third, maybe an automation pipeline, and things get messy:

• Every service stores its own copy of your API key. Rotating a key means tracking down every config file.
• There's no cache. Two tools sending the same prompt both pay for it.
• There's no spend visibility. Each tool burns credits on its own.
• If your provider rate-limits you, everything breaks until the window resets.
• You can't route cheap tasks to a cheap model and expensive tasks to a smart one without configuring each tool separately.

A gateway fixes this. Tools talk to the gateway. The gateway holds the keys, routes requests, caches when it can, tracks what things cost, and fails over if a provider goes down.

The stack

Three containers:

The gateway exposes a single OpenAI-compatible endpoint. Every consumer points at it. Behind it, you can mix cloud providers and local inference however you want.

Keep it behind your reverse proxy on your internal network. There's no good reason to expose an LLM gateway to the public internet.

Deployment

services:
  litellm:
    image: ghcr.io/berriai/litellm:latest
    container_name: litellm
    restart: always
    security_opt:
      - no-new-privileges:true
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "4000:4000"
    environment:
      - DATABASE_URL=postgresql://litellm:password@litellm-db:5432/litellm
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
      - STORE_MODEL_IN_DB=True
    depends_on:
      litellm-db:
        condition: service_healthy
      litellm-redis:
        condition: service_healthy
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
      interval: 30s
      timeout: 10s
      retries: 3

  litellm-db:
    image: postgres:17-alpine
    container_name: litellm-db
    restart: always
    security_opt:
      - no-new-privileges:true
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${LITELLM_DB_PASSWORD}
    volumes:
      - litellm-db:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"]
      interval: 10s

  litellm-redis:
    image: redis:8-alpine
    container_name: litellm-redis
    restart: always
    security_opt:
      - no-new-privileges:true
    command: ["redis-server", "--maxmemory", "512mb", "--maxmemory-policy", "allkeys-lru", "--save", ""]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s

volumes:
  litellm-db:

I'm using :latest here for readability, but in production I pin every image by SHA digest. A bad upstream release shouldn't be able to break your gateway overnight.

A couple of things in this compose that took me a minute to figure out:

STORE_MODEL_IN_DB=True moves model definitions and API keys into Postgres, so you manage them through the admin UI instead of editing YAML. The config file only holds gateway-level settings like caching and timeouts.

Redis runs without persistence (--save "") on purpose. It's a cache. If it clears on reboot, you get a few cold calls on warmup. No data is lost.

Generate the master key and salt with openssl rand -hex 32. They're the root of the gateway's security, so don't reuse something from another service.

Gateway config

The config file mounted into the container handles behavior that applies to every request:

general_settings:
  store_model_in_db: true
  store_prompts_in_spend_logs: true

litellm_settings:
  drop_params: true
  num_retries: 0
  request_timeout: 600
  cache: true
  cache_params:
    type: redis
    host: litellm-redis
    port: 6379
    ttl: 3600
    mode: default_on

drop_params: true is worth explaining. When you route a request to a provider that doesn't support a parameter you sent (say, temperature to a model that ignores it), LiteLLM drops it instead of erroring. This matters more than you'd think once you start mixing providers.

request_timeout: 600 gives requests 10 minutes. Some LLM calls are long, especially with large context windows. The default timeout will cut those off.

store_prompts_in_spend_logs: true logs the actual prompt text alongside cost data. Indispensable when you're trying to figure out which request burned through budget. Turn it off if you don't want prompts in your logs.

Adding providers

With the DB-based config, you add providers through the admin UI. The structure looks like this if you're doing it in YAML:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: gemini-2.5-pro
    litellm_params:
      model: gemini/gemini-2.5-pro
      api_key: os.environ/GEMINI_API_KEY

  - model_name: deepseek-v3
    litellm_params:
      model: openrouter/deepseek/deepseek-chat
      api_key: os.environ/OPENROUTER_API_KEY

  # Local inference, zero cost
  - model_name: llama-local
    litellm_params:
      model: ollama/llama3.2
      api_base: http://ollama:11434

router_settings:
  routing_strategy: simple-shuffle

You can list multiple deployments under the same model_name. LiteLLM load-balances between them. That's how failover works: two entries for the same model pointing at different providers. If one goes down, traffic shifts to the other.

Once providers are set up, every consumer gets the same two values:

Base URL: http://your-gateway:4000/v1
API Key: sk-your-litellm-master-key

Anything that speaks OpenAI can use it. Chat UIs, automation tools, the OpenAI SDK in a script. They don't know or care which provider actually serves the request.

Caching

Caching was the biggest cost saver for me. With mode: default_on, every request gets checked against Redis before hitting the provider. Same prompt, same model, same parameters, within the TTL window (1 hour default)? Cached response, no provider call, no charge.

You'd be surprised how often this hits. System prompts are identical across requests, so the opening tokens are frequently cacheable. Scheduled workflows that summarize or classify often see identical inputs. And debugging? Running the same prompt five times while iterating on something used to cost five API calls.

The cache lives in Redis with a 512MB cap and LRU eviction. It survives container restarts but not host reboots, which is fine. A cache miss is a cold call, not data loss.

If you need a fresh response, add "cache": {"no-cache": true} to the request body.

Authentication

Two layers to think about.

The LITELLM_MASTER_KEY is the root credential. It can do everything: create keys, set budgets, view all logs. Don't hand it out to tools. Generate virtual keys from the admin UI instead, each scoped to specific models and budgets. A script that only needs the local model shouldn't have access to your most expensive cloud provider.

For the admin UI itself, LiteLLM supports OIDC. If you're running a self-hosted identity provider (Pocket ID, Authentik, Authelia, Keycloak), wire it up:

GENERIC_CLIENT_ID=your-client-id
GENERIC_CLIENT_SECRET=your-client-secret
GENERIC_AUTHORIZATION_ENDPOINT=https://your-idp/authorize
GENERIC_TOKEN_ENDPOINT=https://your-idp/token
GENERIC_USERINFO_ENDPOINT=https://your-idp/userinfo
GENERIC_SCOPE=openid email profile
AUTO_REDIRECT_UI_LOGIN_TO_SSO=true
PROXY_ADMIN_ID=your-oidc-identity-id

With this, the admin UI bounces through your identity provider. PROXY_ADMIN_ID maps your OIDC identity to full admin access.

On the network side, keep the gateway internal. Reverse proxy, gated access, no public exposure without strong auth in front. If something outside your network needs in, route it through a VPN or tunnel.

Budgets and spend tracking

LiteLLM tracks spend per model, per key, per team. You can set daily budgets in USD, and when the gateway hits the limit, it stops accepting requests and sends a notification through whatever channel you've configured (webhook, Slack, Discord, ntfy).

This saved me from a runaway workflow once. An automation got stuck in a loop, making API calls every few seconds. The budget cap kicked in before I even noticed something was wrong. Without it, I would have found out when the credit card charge came through.

The admin dashboard breaks down spend by model and by day. When you can see that a model eating 80% of your budget is being used for tasks a local model could handle, you adjust routing. It's not about being cheap. It's about having enough information to make decisions.

Connecting tools

This part is easy. Every tool just needs the gateway URL and a key:

Base URL: http://your-gateway:4000/v1
API Key: sk-your-litellm-key

For a chat interface, I use OpenWebUI pointed at the gateway. It looks like any other chat app to the user. Every model in the dropdown is served through LiteLLM, some local, some cloud, and the user can't tell the difference.

Automation works the same way. Tools like n8n or Activepieces support custom OpenAI-compatible endpoints, so they plug right in. Workflows that summarize emails, classify tickets, or generate reports all go through the gateway now, with caching and budget protection.

Scripts are the simplest. If you're using the OpenAI Python or JS SDK, swap the base URL and you're done. Same code, same interface.

Wrapping up

The thing I like about this setup is that it gets out of the way. I stopped thinking about which API key goes where. I stopped worrying about a workflow going rogue overnight. I stopped running the same prompt twice and paying for both.

The caching pays for itself. The budget alerts are a safety net. The spend tracking actually changed how I route traffic between models, because for the first time I could see what was costing what.

If you're using LLMs in more than one place, put a gateway in front of them. One compose file, and you don't have to think about any of this stuff again.