Routing capacity and request queues

There is no models.*.maxConcurrentRequests setting. The real per-model key is models.*.concurrencyLimit, which limits active parallel requests.

The router serializes model swaps and queues requests until the selected model is ready. Avoid using many simultaneous client retries as a capacity control: they create more queued work. Choose a routing policy that fits the models that may coexist, then set client timeouts high enough for the queue and load time.

Inspect the Activity view and logs when latency grows. A queue that never drains usually means the command, proxy, or health check is wrong; see guides/model-runtime/troubleshooting-model-wont-load.

Global concurrency limit

globalConcurrencyLimit is a top-level setting, separate from models.*.concurrencyLimit. It caps the number of inference requests served at once across every model combined, using a single shared semaphore:

globalConcurrencyLimit: 8

The default is 0, meaning no limit — the semaphore is not even added to the request chain, so a default config pays no cost for the feature. Once the limit is reached, further requests are rejected immediately with an HTTP 429 response rather than queued; there is no wait for a slot to free up. Clients should treat a 429 as a signal to back off and retry, the same way they handle a per-model concurrency limit rejection.

Use this to protect shared hardware (CPU, disk, network) from being overwhelmed by traffic spread across many different models, which a per-model concurrencyLimit cannot do since it only counts requests to one model at a time.

Both limits count a websocket, such as a /v1/realtime session, as one request for as long as it stays open; see guides/api-integration/v1-realtime.