Documentation

Configuration overview Start with one small config.yaml, get the Docs agent running, then grow into config.example.yaml as you need more. Reusing configuration with macros Global and per-model macros, environment variable macros, and the ordering rules that trip people up.

Automatic model unloading with ttl How ttl, globalTTL, unloadTimeout and Docker cmdStop interact, plus how to unload on demand or pause inference. Client compatibility and loading feedback Let clients discover models and handle loading responses instead of treating a swap as a failure. Model capabilities and model listings Advertise images, tools and context length in /v1/models, automatically or by hand. Troubleshooting - a model won't load Work through the usual causes when a model fails to start, times out, or hangs on every request. Writing the cmd for a model Minimal model config using cmd and ${PORT}, plus proxy, checkEndpoint loading progress, Docker, and llama-server's --jinja flag.

Profiles and selectors Switch model sets at runtime with profiles, or resolve a virtual model ID per request with selectors. Routing capacity and request queues Configure concurrencyLimit and globalConcurrencyLimit, and understand queued work while a model is loading or busy. Running several models at once with groups, matrix and capacity Choosing between group, matrix and capacity routers, and how each decides what gets unloaded.

API keys, access control, and keeping secrets out of your config apiKeys authentication has no per-key rate limits or user permissions; use env macros for secrets. CORS and browser access How llama-swap decides which Access-Control headers to send, and how to allow a specific dashboard origin. Connecting an MCP client to llama-swap Point any MCP client at /api/mcp to search llama-swap's documentation and source code. Realtime voice chat with /v1/realtime Proxy OpenAI Realtime API websockets to llama-server for speech to speech models like LFM3 Audio, and talk to them from the Playground. Rewriting requests with filters Use stripParams, setParams and setParamsByID while preserving the protected model parameter. Set-if-undefined — set a parameter only if the request didn't Append ? to a setParams or setParamsByID key to apply the value only when the request doesn't carry that parameter.

Proxy and model startup timeouts Distinguish client proxy timeouts from health-check and model startup failures, and fix 502 EOF errors after streams. Request body size limit Raise maxRequestBodyMB when large audio uploads or base64 images fail with HTTP 413 Request Entity Too Large. Route to remote peers Expose models from another llama-swap host through a peer connection. Serving llama-swap under a reverse proxy subpath Run llama-swap at https://host/some/path/ by having the proxy strip the prefix and rewriting it away. Upstream API passthrough Control which /upstream model paths are ignored instead of triggering a model swap.

Building the unified llama-swap image How the unified image is assembled from per-project Dockerfiles, how to build it and pin the source revision (LS_VERSION), add an engine, and verify kubeswap is inside. Choosing storage for model weights in a kubeswap namespace Why the shared model-cache PVC wants ReadWriteMany, which drivers provide it, and the RWO / emptyDir escapes when it does not. Connect llama-swap with Tailcat Privately expose inference over Tailcat and route peers through Tailcat connection tokens. Container security Choose root or non-root llama-swap container images and retain only the host access your deployment needs. Debugging kubeswap backends when a model fails to load The failure chain from kubeswap status reasons to backend pod logs, and what each symptom (Pending, CrashLoopBackOff, never Ready) usually means. Observability, storage and Activity Use logs, metrics, captures and the Activity view to diagnose requests and retain useful history. Startup preloading and hooks Load selected models at startup and run hooks around lifecycle events.

ComfyUI with the /comfyui endpoint Name a model comfyui_auto and use /comfyui/ so ComfyUI's websocket does not block model swapping. audio.cpp UI with the /audiocpp endpoint Name a model audiocpp_auto and use /audiocpp/ to reach audio.cpp's own UI and model management without naming a model in each request.

Fast vLLM swapping with vllm-wrapper and sleep mode vllm-wrapper is used as cmd and cmdStop so llama-swap puts a running vLLM server to sleep instead of stopping it, and wakes it up on the next request. Run vLLM under systemd with vllm-wrapper Start vLLM as a transient systemd user service from vllm-wrapper so it survives llama-swap stopping the model, and forward its journal into llama-swap's logs. Running inference servers in Kubernetes with kubeswap Use the kubeswap wrapper as cmd/cmdStop so llama-swap starts and stops backend servers as Kubernetes workloads in a dedicated namespace. Test llama-swap without a GPU using fake-model fake-model is a tiny OpenAI-compatible server with a configurable load time and token speed, for testing swapping, loading and streaming without a real model. Wake a suspended llama-swap server with wol-proxy wol-proxy sits in front of llama-swap, sends a Wake-on-LAN packet when a request arrives and the server is asleep, and holds the request until the server is back.

Peers - splitting models across machines and providers Route to another llama-swap, a remote OpenAI-compatible server, or a hosted provider under one endpoint. kubeswap: a complete multi-engine Kubernetes configuration A verified llama-swap config that runs llama-server, sd-server, whisper-server and audiocpp_server as Kubernetes workloads through the kubeswap wrapper.

Configuration reference Every configuration option, from the annotated config.example.yaml.