AI infrastructure
Inference Routing on Kubernetes: Six Options Benchmarked
We tested six ways to route vLLM requests on Kubernetes, from a plain Service to inference stacks, and explain what changes under load and during failures.
Ivan Porta
Founder & Principal Engineer

A Kubernetes Service spreads connections across model replicas, but it knows neither which replica already holds a prompt's KV cache nor how much work is waiting on each replica. Both gaps matter when requests repeat a system prompt, continue a conversation or compete with long prompts.
In our benchmark, we compared NVIDIA Dynamo, Buoyant Kinference, Volcano Kthena, llm-d and the vLLM production-stack against a plain Kubernetes Service. We measured where cache-aware routing reduced the time to first token and what changed under cache pressure and competing work. We also deleted engine and router pods under load to see how each stack failed and recovered.
Architecture comparison
The six options choose a replica at different points in the request path, using different information. Below are the architectures and routing modes relevant to our tests.
- Kubernetes Service (kube-proxy rules): The client connects to the Service IP. The kube-proxy rules send each new connection to a random ready pod associated with the Service without reading the request.
- Dynamo (Frontend): The client sends the request to the Frontend, which applies the chat template and tokenizes the prompt. In KV mode, the Frontend hashes the tokens into blocks and picks a worker based on cache overlap and load. In round-robin mode, its default, it ignores the cache. Finally, it sends the token IDs to the chosen worker. Optionally, you can run each phase (prefill and decode) on separate workers.
- Kinference (decision service): The client sends the request to Buoyant Gateway, which sends the request body to Kinference. Kinference has the serving engine tokenize the prompt by default, or tokenizes it in-process with the model's tokenizer. Kinference hashes the tokens into blocks and predicts each replica's time to first token (TTFT) based on that replica's cache reuse and queued work. Kinference then returns the endpoint with the lowest prediction, and the gateway sends the request there.
- Kthena (router): The client sends the request to the Kthena router, which finds the pods that serve the model. With the chart's default plugins, the router drops pods with 10 or more queued requests and scores the rest based on prefix match, outstanding requests, KV-cache usage, and latency. Finally, it forwards the request to the highest-scoring pod. Tokenization is also optional and depends on the enabled plugin. For example, the
kvcache-awareplugin has the engine tokenize the prompt and uses KV-cache events from the engine to identify which pods already cache the prompt's prefix. The chart does not enable that plugin, and we did not test it. - llm-d (endpoint picker): The client sends the request to Envoy, which asks the endpoint picker for a pod via ext-proc. The picker has the serving engine tokenize the prompt or estimates the tokens from its bytes, then hashes them into blocks. The picker then keeps the pods that hold at least 80% of those blocks, unless there are none or they are far busier than the rest, picks the one with the fewest tokens in flight, and returns its IP to Envoy, which finally sends the request. Similarly to Kthena, it has multiple plugins that affect the decision, for example, by adding weighted prefix, queue, and KV-cache scores.
- production-stack (router): The client sends the request to the production-stack router, which keeps the endpoints that serve the requested model. The routing logic picks one of them: round-robin by default, or the endpoint with the longest prefix match on the message text in
prefixawaremode. Other logics include routing by session, by LMCache placement (kvaware,loadaware) and by priority. The router then forwards the request to that endpoint.
Kinference Explained and From Ingress to Inference Gateway follow a request through Kinference and through llm-d's endpoint picker step by step.
How we tested
Infrastructure
We wanted to compare routing decisions on a fleet small enough to inspect request by request. To do so, we used HAMi to divide the memory of a 96 GB RTX PRO 6000 between three vLLM replicas, and tested each inference project individually. Both prefill and decode were done by the same workers. The diagram shows the test bed that every stack shared.
| Part | Configuration | Limits and settings |
|---|---|---|
| Node | GKE Spot | One NVIDIA RTX PRO 6000 with 96 GB; Kubernetes v1.36.5-gke.2370000. Routers and the load-generator Job ran on the same node. |
| Replicas | Three engines | Qwen/Qwen2.5-7B-Instruct on vllm/vllm-openai:v0.31.0. Each received 28,000 MiB with no nvidia.com/gpucores limit. Dynamo's own workers used its vllm-runtime:1.6.2 image, which pins the same vLLM version. |
| Engine settings | Same values throughout | Context length 16,384 and gpu_memory_utilization 0.90, plus the three flags that the routers and the measurements need: --enable-prompt-tokens-details, --enable-scale-out and --kv-events-config. We did not tune the engines' own flags. |
| KV cache | Per replica | 166,576 tokens in 10,411 blocks of 16 tokens. The prefill batch (max_num_batched_tokens) was 2,048 tokens per engine step, vLLM's default for a 28,000 MiB slice. For the whole card the default is 8,192. |
| Routers | One stack at a time | Dynamo platform chart and runtime 1.6.2. Kinference 0.1.0-alpha.7 behind Buoyant Gateway 0.1.0-alpha.6. Kthena chart 1.1.0 with only the router. llm-d router 1.0.0 in standalone mode, with its Envoy sidecar and no InferencePool. production-stack chart vllm-stack-0.1.14, using its Python router with existing pods. |
Routing configurations
We used a cache-aware routing mode that each project documents, and we calibrated the values that depend on the model and hardware, like Kinference's profiles. For Kthena and production-stack, that mode is the prefix-based one: we did not test Kthena's kvcache-aware plugin, which needs a sidecar beside each engine, or production-stack's kvaware logic, which needs LMCache. We chose the values in separate tuning runs, before the five repetitions that the results report. The table names each configuration, the method used to select values, and the resulting setup.
| Configuration | Method | Setup |
|---|---|---|
| Kubernetes Service | No tuning | ClusterIP Service selecting the three engines. |
| Dynamo | Weight sweep | KV routing with worker events, DYN_ROUTER_PREFILL_LOAD_SCALE=8 (the better of 4 and 8) and DYN_ROUTER_PREDICTED_TTL_SECS=5 from Dynamo's tuning guidelines. |
| Kinference | Observation script | kvBlockSize: 16. A script on logged picks set base_ms to 55, attention_uncached_block_ms to 1.28, attention_uncached_per_every_block_ms to 0.002 and forward_pass_ms to 17.4. |
| Kthena | Weight sweep | prefix-cache weight 4 instead of the chart's 1. |
| llm-d, approximate | Calibration Job | The optimized-baseline guide with peakPrefillThroughput: 24914, measured at CHUNK_SIZE=2048. |
| llm-d, precise | Calibration Job | The precise prefix cache routing guide with the same throughput and blockSizeTokens: 16. |
| production-stack | Mode selection | routingLogic: prefixaware, which requires no hardware-specific calibration. |
What we tested
| Workload | Tests | Request pattern |
|---|---|---|
| shared-prefix | Reuse | 38 system prompts of about 8,000 tokens, visited six times each. The prefixes occupy 1.8 times one replica's cache. |
| multi-turn | History | 36 conversations of eight growing turns. |
| control | Unique prompts | 228 prompts of about 8,000 tokens with no reuse. |
| pressure | Eviction | 76 system prompts and twelve requests in flight. |
| burst | New prefixes | 24 prefixes, each arriving as six simultaneous requests, followed by one repeat visit. |
| mixed | Competing work | Three load-generator clients sending long prompts and long answers alongside 240 short requests. |
| rate-sweep | Rising arrivals | Warm prefixes at an average of 1, 2, 3, 4 and 6 arrivals per second, for 45 s per rate. |
| replica-restart | Engine pod deletion | One engine pod deleted under steady traffic. |
| router-restart | Router pod deletion | 240 s of traffic, with router pods deleted after 45 s. |
| overhead | Added latency | 300 pairs of one-token requests, one through the router and one straight to an engine (Dynamo's workers have no direct path). |
A repetition runs every stack through these workloads, one stack at a time, and removes each stack before the next one starts, apart from Dynamo's operator. Every stack receives the same prompts within a repetition. The next repetition uses a new seed and starts with a different stack. Each workload starts with recreated engine replicas and empty caches. The first six workloads ran five times. The last four ran three times, and so did the routers' default configurations, on shared-prefix, multi-turn and control only.
Shared-prefix and control keep six requests in flight, and multi-turn runs six conversations at a time. The load generator opened a new connection for every request. A reused connection stays on the replica that kube-proxy picked when the connection opened, so a new connection per request lets the Service choose a replica for every request.
How we measured
The load generator ran inside the cluster and recorded every request. It measured TTFT on the client side, from the moment it opened the connection to the first streamed chunk that carried text. TTFT therefore includes the connection, the router's decision, any wait in the engine's queue and the prefill. The last chunk of each response carries the engine's usage report: the prompt tokens, and how many of them the engine found in its cache. Cached share divides the cached tokens by the prompt tokens, summed over the requests that a figure covers. A request counts as failed when it returns an error or no first token; failed requests stay out of the latency figures.
Shared-prefix and pressure report visits after the first, while multi-turn reports turns after the first. The first visit to a prefix and the first turn of a conversation have nothing to reuse, whichever replica serves them.
We calculated each statistic within a run, then averaged the per-run results. Most per-run percentiles cover 190 to 380 requests, so a p95 rests on the ten to twenty slowest requests of a run.
Where this article says which replica served a request, the placement comes from the routers' logs, matched to the load generator's records by request ID. llm-d's logs and the Service give no placement, so for those two, and for the bursts, we used each engine's count of finished requests.
Results
All three engine replicas shared one GPU, so routing affected both cache reuse and competition for compute. These results apply to the configurations that we tested; separate GPUs or different routing plugins could change the outcome. Latencies below are means of per-run percentiles.
Repeated prefixes
Every tuned cache-aware configuration served about 99% of repeat requests' input tokens from cache, compared with 42% through the Service. Median TTFT fell from 834 ms to between 115 and 132 ms. These figures exclude each prefix's first visit. Total response time improved by a smaller share, from 2.88 s to between 1.85 and 1.87 s, because generating the answer still took most of the time.
Multi-turn conversations showed the same benefit. The replica that served the previous turn was most likely to hold the latest history, and the routers steered subsequent turns toward it. Their p95 TTFT was 146 to 158 ms, compared with the Service's 590 ms.
Unique prompts
Without reusable prefixes, no router improved on the Service's median TTFT of about 1.15 s. Dynamo, Kinference, Kthena and llm-d did improve the tail, showing that load balancing can still help when caching cannot. On our shared GPU it was a trade: their median TTFT was 5 to 29% above the Service's, and their median response took about 5% longer. Kthena's margin at the tail was the smallest and close to its run-to-run spread.
The reason is the shared GPU. A random pick, which is what the Service makes, often leaves one replica idle, and the other two then prefill faster: a prompt that did not have to queue reached its first token in about 0.8 s, against about 1.15 s with all three replicas busy. The routers that balance load almost never left a replica idle. Fewer requests queued, which shortened the tail, but nearly every prefill ran at the slower pace.
Dynamo spread the requests most evenly and kept two on every replica. The three engines therefore prefilled at the same moments and competed for the GPU, and every second request queued behind another prompt on its own replica. Half of Dynamo's requests reached their first token in about 1.22 s and the other half in about 1.75 s. Its median, 1.48 s, is the highest and falls between those two groups. The round-robin configurations showed the same pattern. Separate GPUs could change it.
Cache pressure
The pressure workload exceeded the fleet's cache capacity, forcing engines to evict blocks. Dynamo, Kinference and precise llm-d recorded p95 TTFT of 1.39 to 1.59 s. Kthena, approximate llm-d and production-stack reached 2.42 to 2.79 s.
Dynamo, Kinference and precise llm-d read engine KV events and can learn when a prefix is evicted. Kthena, approximate llm-d and production-stack route on their own history and cannot. In these runs, those three kept a known prefix on the replica that first served it, even after that replica had evicted the prefix. For Kthena and production-stack, whose logs show each placement, those visits make up more than 90% of the slowest 5%.
Dynamo served 33% of input tokens from cache, against 51 to 54% for the other routers, and still had one of the shortest tails. Its load model kept queued prefill work charged until the first token, making a less busy, cold replica more attractive. Cache reuse alone did not guarantee a short wait.
The tail consists of misses that had to queue. A miss with nothing ahead of it on its replica took about 1.2 s with every router. With Kthena and production-stack, every request that took more than 2 s had at least one other prompt ahead of it on the same replica. Dynamo missed more often, but it sent a miss to a replica with little prefill work waiting, and nine in ten of its misses finished within 1.3 s.
Bursts of a new prefix
Each burst sent six requests with the same new prefix. A burst took about 0.4 s when it stayed on one replica, 0.8 s on two and 1.15 s on three. production-stack kept every burst on one replica and recorded a median TTFT of 388 ms. Precise llm-d averaged 1.5 replicas per burst and 492 ms. Approximate llm-d averaged 2.8 replicas, Dynamo, Kinference and Kthena used all three, and all four reached about 1.15 to 1.17 s.
The TTFT follows from how many engines compute the prefix at the same time. On one replica, the engine computes the 8,000-token prefix once, with the whole GPU, and the other five requests reuse it. On two or three replicas, every engine computes the same prefix again while it shares the GPU with the others. production-stack records placement before forwarding, letting later requests follow the first. Precise llm-d also records recent picks, but simultaneous decisions sometimes spread the burst. Separate GPUs could change this trade-off.
Short requests beside long prefills
A short request in this workload either started at once, in about 0.1 s, or reached a replica that was still prefilling an 8,000-token prompt and waited 0.6 to 0.9 s for that prefill to finish.
With the Service, Kthena and production-stack, 11 to 12% of short requests waited in that way. A p95 marks the slowest 5%, so for those three it is one of the waits: 651 ms for the Service, 847 ms for Kthena and 730 ms for production-stack. With Dynamo, Kinference and both llm-d configurations, fewer than 1% waited, so their p95 is an ordinary request, below 180 ms. The medians stayed between 95 and 133 ms.
The difference was how the routers represented outstanding work. With every stack, about four in ten short requests arrived while a long prompt was prefilling on one of the replicas. Through the Service, Kthena and production-stack, about one in four of those requests reached the prefilling replica and waited. Through Dynamo, Kinference and llm-d, fewer than 2% did. The Service and production-stack's tested prefix mode do not read load, so they placed a new short prompt at random. Kthena's least-request plugin counts requests, not tokens, so a replica that is prefilling 8,000 tokens weighs the same as one that is finishing a short answer. The tested Dynamo, Kinference and llm-d configurations account for prompt size, making a long prefill look more expensive than a short request. Kthena's p95 is the highest because its short requests had more of the prefill left to wait for: about 0.8 s, against about 0.56 s with production-stack.
The wait itself happens in the engine: by default, vLLM gives each step's token budget to the long prefill until it is done. Its long_prefill_token_threshold setting caps that; we left it off.
Rising arrival rates
The rate sweep sent requests independently of earlier completions. Across nominal rates of 1 to 6 requests per second, every router's p95 stayed below 190 ms. Sending each visit to the replica that held its prefix avoided much of the repeated prefill work incurred through the Service.
At six arrivals per second, the Service completed 4.3 requests per second and reached a p95 of 10.2 s. Its backlog hit the load generator's limit of 96 in-flight requests, causing an average of 34 dropped arrivals per run. This shows overload under the tested traffic; the 45-second stages do not establish maximum sustainable throughput.
Failures and restarts
Deleting one engine pod failed at most one request per run outright. Up to four more responses per run were cut off after their first token, and the load generator counts those as successful; Dynamo and Kinference had none. This was a clean pod deletion, not a test of node loss, GPU failure or a network partition.
Router restarts exposed both request failures and lost routing state. Failure counts were strongly affected by client behavior: immediate refusals triggered retries every 0.5 s, while some connections to Dynamo, Kthena and production-stack stalled for about 133 s. The span between failed requests' start times therefore does not measure outage duration. Responses that were in flight when the router pod was deleted broke off with Kthena and llm-d and completed with the other three.
Precise llm-d stayed at about 99% cached-token share through the restart: the new picker replayed the engines' buffered KV events when it started. Kinference fell to about 50% and returned above 90% roughly a minute after restart; its tested configuration left event replay disabled. Approximate llm-d dipped to 68% after losing routing history. The shares in the chart are averages over 15-second windows of three runs.
Dynamo returned at 99%, but the stalled clients had given it about two minutes without traffic to rebuild its index. That observation cannot establish its recovery speed. Kthena and production-stack needed new traffic to rebuild their placement history. Test restarts with explicit client connection timeouts before comparing availability.
Resource usage
Averaged across workloads and repetitions, every tested stack used less than 50 millicores of CPU. Average sampled memory peaks ranged from 28 MiB for Kthena and 29 MiB for Kinference to 712 MiB for production-stack.
These figures cover the added routing components, including Dynamo's operator and the data plane of Kinference's gateway, but not that gateway's two control-plane pods, which we did not sample. They are averages, not resource limits. Ten-second sampling can miss brief peaks, and measurements during the replica-restart workload were higher. Size production resources from representative peak usage.
Defaults against tuned settings
The default configurations did not all preserve cache affinity. Dynamo and production-stack started in round-robin mode, while Kthena's default prefix weight competed with three load scores.
Changing Dynamo's KV-routing prefill weight reduced shared-prefix p95 from 1.22 s to 155 ms. Raising Kthena's prefix weight reduced it from 1.18 s to 147 ms. Kinference already reused prefixes; calibration improved its prediction accuracy without materially changing shared-prefix latency. The default figures are three-run means and the tuned ones five-run means.
Tuning also changed trade-offs. Switching production-stack to prefix-aware routing improved reuse, but its unique-prompt p95 rose from 1.76 s to 2.12 s, while its mean TTFT fell from 1.49 s to 1.25 s. Neither mode reads load: round-robin rotates through the replicas, and prefixaware sends a prompt with no known prefix to a random one. Validate the configuration against both reusable and unique prompts.
Which one to pick
Choose around your traffic and existing infrastructure. The findings below apply to the versions and configurations that we tested.
- Kubernetes Service: A reasonable starting point with one replica or little prefix reuse. It balances connections without inspecting prompts, so mixed request sizes or uneven queues may still justify a load-aware router.
- Dynamo: Consider it when you want engine orchestration and the option to separate prefill and decode into different worker pools. Those capabilities extend beyond this benchmark, which used combined workers sharing one GPU.
- Kinference: Consider it when policies must choose between local replicas and hosted providers, a capability that this benchmark did not test. The tested release requires Buoyant Gateway or a client implementing its gRPC API, and has alpha APIs and a single-router limitation.
- Kthena: Fits teams already using Volcano for workload scheduling. We tested its router alone: increasing prefix affinity improved reuse, but the tested configuration struggled with short-request tail latency beside long prefills.
- llm-d: Fits gateways that implement the Gateway API Inference Extension; we tested its standalone mode, in which an Envoy sidecar takes the gateway's place. Its precise configuration performed well under eviction and bursts, but requires engine KV events and compatible tokenization and block sizes.
- production-stack: Fits teams wanting the vLLM project's deployment stack, including routing, monitoring, and optional LMCache integration. Its tested
prefixawaremode favored reuse but ignored load; evaluatekvawareorloadawareseparately if routing around LMCache placement. At this version onlyloadawarereads chatmessages.
Before adopting a router
Start with the workload that misses your latency target. Measure first-token latency, completion time, failures, and cache reuse. For vLLM, calculate cached-token share over a fixed interval:
sum(rate(vllm:prefix_cache_hits_total[1h])) / sum(rate(vllm:prefix_cache_queries_total[1h]))Scope both metrics to the same deployment and model. The ratio includes cold requests, so it is not directly comparable with our repeat-only cache figures.
Before moving production traffic:
- Check the configuration. Calibrate load estimates for your model and hardware. Where routing uses engine KV events, verify event delivery and matching block sizes.
- Replay representative traffic. Include cache pressure, simultaneous requests for a new prefix, and short requests beside long prefills. Set acceptable latency and failure thresholds before testing.
- Test recovery. Use explicit client connection timeouts and restart a router under load. Check both successful responses and recovery of cache-aware routing.
- Verify availability support. Use multiple router replicas only where the release and routing configuration support them. The tested Kinference release and llm-d precise recipe have single-router constraints, and the other tested configurations keep prefix history or in-flight load in each router process.
On our shared GPU, all tested cache-aware configurations improved repeated-prefix latency. The more useful differences appeared under competing workloads and during recovery. Use those cases to validate the configuration that you intend to operate.
Frequently asked
Five questions we keep getting about inference routing.
Will a router help if my prompts are unique?
It can reduce tail latency by balancing queued work, even when prompts have no reusable prefix. In our test on one shared GPU, the routers that shortened the tail also had a higher median TTFT, and their median response took about 5% longer.
Why does a cache-aware router need tuning?
Tuning balances cache reuse against queued work. The right settings depend on your model, hardware, and traffic.
Do I need engine KV events?
Some routers infer cache placement from past requests. KV events let supported routers track actual cache changes and evictions.
Can I put Kinference behind an existing inference gateway?
The tested release supports Buoyant Gateway or clients using its gRPC API. Other gateways need an integration.
Which result should I reproduce before adopting a router?
Test your traffic pattern, then restart the router under load. Check latency, failures, and cache recovery with explicit client timeouts.
If you need help evaluating and operating inference routing, Todea can run the pilot and manage the resulting platform.


