Writing the cmd for a model

cmd is the only required setting on a model. It is a plain command line — whatever you would type in a shell to start the server yourself.

Recent llama-server builds enable --jinja by default. If a model describes a tool call in prose instead of making one, your build predates that default — add --jinja to cmd explicitly. Without it, llama.cpp silently ignores the request's tools array. The separate capabilities.tools setting only advertises support to clients; it does not enable tool calling in llama-server.

Smallest working configuration

A minimal llama-server config must pass its assigned ${PORT} to the server. Omitting --port ${PORT} makes llama-server listen somewhere other than llama-swap's default proxy target.

models:
  model1:
    cmd: llama-server --port ${PORT} --model /path/to/model.gguf

Keep ${PORT} in a minimal llama-server command. It is the host port that llama-swap assigns and the default proxy uses.

Multi-line and comments

Use a YAML block scalar (|) to break it up. Lines starting with # are stripped before the command runs, so you can annotate flags:

cmd: |
  llama-server --port ${PORT}
  --model /models/qwen3-8b.gguf
  --ctx-size 16384
  # only needed on builds older than --jinja's default-on change
  --jinja
  --cache-type-k q8_0

The three built-in macros

macro	expands to	available in
${PORT}	a free port assigned to this model, starting at startPort (default 5800)	cmd, and anywhere else in the model once cmd uses it
${MODEL_ID}	the model's key from the models map	anywhere in the model, and in global macros
${PID}	the upstream process id	cmdStop only

These names are reserved — you cannot define a macro called PORT, PID or MODEL_ID.

proxy: when you need it

proxy tells llama-swap where to send requests. It defaults to http://localhost:${PORT}.

• Used ${PORT} in cmd? Omit proxy. The default is correct.
• Hardcoded a port in cmd? You must set proxy to match, or llama-swap proxies to the wrong place and every request times out.
• Running in a container? Set it explicitly — proxy: "http://127.0.0.1:${PORT}" — because the port inside the container differs from the one outside. In -p ${PORT}:8080, ${PORT} is the host port and 8080 is the container port.

This mismatch is the single most common configuration error. If a model starts successfully but requests hang or return connection errors, check it first.

checkEndpoint: knowing when it's ready

llama-swap polls checkEndpoint (default /health) until it returns HTTP 200, and holds requests until then. Servers that don't expose /health need it changed:

checkEndpoint: /v1/models

Set checkEndpoint: "none" to skip health checking entirely — llama-swap then forwards the first request immediately, which usually means it fails while the server is still loading weights. Only use it for servers with no usable readiness endpoint.

Loading progress

While the server is not ready, it can report how far along it is by answering the health check with a non-200 status and a JSON body like this:

{"progress": 0.42, "message": "loading tensors 12/28"}

• progress is the fraction loaded, from 0 to 1. Values outside that range are clamped, so a percentage such as 42 reads as fully loaded.
• message is a short description of the current step. llama-swap keeps the first 1024 characters.
• Either field can be left out. A body with neither, or one that is not JSON, is treated as a plain "not ready yet". llama-server's own loading response falls in this group.

The Models page and the model's detail page show the percentage, a progress bar and the message while the model starts.

Each time the reported progress value changes, llama-swap restarts the healthCheckTimeout countdown. The timeout then limits how long a load may go without progress, not how long it may take in total, so a large model that keeps reporting progress is never cut off. A server whose progress stops changing fails with health check timed out after ... without loading progress, even if its message keeps changing (an ETA countdown, say).

Without progress, the timeout works as usual: the server must return 200 within healthCheckTimeout. A message on its own is shown in the UI but does not extend the timeout. Set healthCheckTimeout longer than any loading step during which progress does not move.

Environment variables

env injects variables into the command's environment. Each entry is one NAME=value string:

env:
  - "CUDA_VISIBLE_DEVICES=0,1"
  - "HSA_OVERRIDE_GFX_VERSION=11.0.0"

This is the right place for per-model GPU pinning. Note that env values are part of your config file — see guides/api-integration/api-keys-and-auth for keeping secrets out of it.

Model files

llama-swap has no setting that downloads a missing GGUF from Hugging Face or another model registry. The file must already exist, or the configured cmd must fetch it before starting the inference server.

Any OpenAI-compatible server

Nothing here is llama.cpp-specific. vLLM, SGLang, TabbyAPI, Ollama's OpenAI endpoint and anything else that speaks the API work the same way — start it, tell llama-swap the port, point checkEndpoint at something that returns 200.

If the upstream expects a different model name than your llama-swap ID, use useModelName to rewrite it on the way out.

Related

• reference/config/models — every model setting, annotated
• guides/configuration/macros — cutting repetition out of long commands
• guides/model-runtime/troubleshooting-model-wont-load — when it doesn't start