Run many local AI models behind one API.

llama-swap starts the model each request asks for and stops the one it replaces. It works with llama.cpp, vLLM and any other OpenAI or Anthropic compatible server. One binary, one config file, no other dependencies.

Install llama-swap Read the docs

1 Install

# macOS and Linux
brew tap mostlygeek/llama-swap
brew install llama-swap

Also available as Docker images, on WinGet, and as release binaries for Linux, macOS, Windows and FreeBSD.

2 Configure

# config.yaml
models:
  qwen:
    cmd: llama-server --port ${PORT} -m qwen.gguf
  gemma:
    cmd: llama-server --port ${PORT} -m gemma.gguf
    ttl: 300

3 Run

llama-swap --config config.yaml

curl localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma","messages":[...]}'

The model field picks which server runs. It listens on port 8080 and serves a web UI at /ui.

Swaps on demand

Each request names a model. llama-swap loads it, and unloads the previous one, or keeps several loaded at once with a matrix.

Any inference server

llama.cpp and its forks, vLLM, stable-diffusion.cpp, whisper.cpp, ComfyUI and more. Upgrade them whenever you like.

Standard APIs

OpenAI chat, completions, responses, embeddings, audio, images and realtime, plus Anthropic messages.

Built-in tools

Unload idle models with ttl, preload with hooks, rewrite requests with filters, and watch it all in the web UI.

Questions about your config?

Chat with the docs: a small fine-tuned model searches llama-swap's documentation and source code to answer. It runs locally in your browser with WebGPU, so nothing you type leaves your computer.

Chat with the docs