From Ingress to Inference Gateway: Gateway API and the Inference Extension

Traditional load balancing assumes all backends are the same, so it does not matter which one handles a request. LLM traffic changes this. For example, one vLLM replica might have a warm KV cache for an ongoing conversation, while another would need to start from scratch. Some requests are short questions, while others are huge codebases. Round-robin balancers cannot tell the difference, so requests may go to busy replicas while others with cached work sit idle. This leads to higher latency for users and wasted GPU resources.

Kubernetes addresses this with two layers. The Gateway API is replacing Ingress as the main way traffic enters a cluster. On top of that, the Gateway API Inference Extension, developed by the same community (SIG-Network together with WG-Serving), adds a set of APIs and a protocol. This lets a gateway route traffic to model servers based on factors like queue depth, cache state, and loaded adapters, rather than treating them as generic HTTP backends.

From Ingress to Gateway API

Before we get into inference, let’s quickly go over what the Gateway API is and why it matters for Kubernetes networking today. The main difference from Ingress controllers is how it supports collaboration. With Ingress, there’s nothing in the API to prevent multiple teams from using the same hostname. Kubernetes will accept these conflicting objects, and what happens next depends on the controller. For example, ingress-nginx merges all Ingress resources that use the same host, and if there are duplicate paths, the oldest rule takes priority. This means one team could add routes or settings to a hostname that another team thinks they own. There’s also the issue of namespace binding: an Ingress can only use a TLS secret from its own namespace. So, platform engineers either have to copy certificates into every application namespace, which risks exposing private keys, or each team has to manage their own certificate lifecycle. Finally, the Ingress spec doesn’t cover things like timeouts, authentication, rate limits, rewrites, or canaries. These features are usually handled with vendor-specific annotations at different layers, like application, infrastructure, or operations.

The SIG-Network team proposed the Gateway API at KubeCon San Diego 2019 to replace the confusing mix of annotations with a typed, role-aware, and extensible API. It reached general availability in October 2023. However, the real turning point was when the widely used NGINX Ingress Community controller was retired. With no releases, no fixes, no security patches expected, teams worldwide realized that moving to the Gateway API was no longer optional, but an urgent migration they needed to plan and complete.

The Gateway API is designed to break up the overloaded Ingress object by dividing responsibilities based on who owns each decision:

• A GatewayClass declares a kind of load balancer the platform offers (internal L7, external L7, a vendor’s implementation).
• A Gateway instantiates one; creating it is what provisions the listener, the hostname, TLS, and so on.
• HTTPRoute objects, owned by application teams in their own namespaces, attach to that Gateway and declare “requests matching this go to my backend.”

How a gateway becomes an inference gateway

At its core, the Gateway API inference extension is a set of new CRDs and controllers built on top of Envoy’s external processing filter (ext-proc).

Every incoming request passes through Envoy’s chain of HTTP filters. When it reaches the ext-proc filter, Envoy opens a two-way gRPC stream with an external server that acts as the extension’s controller. Envoy sends the request’s headers, and, if the filter’s processing mode requires it, the body as it arrives. The external server then responds with instructions, and Envoy applies them, such as changing headers, modifying the body, adding metadata, or sending an immediate response. Because the processing logic runs on a separate server, it can be deployed, scaled, and updated independently, and it can be written in any language. The ext-proc contract is defined by Envoy, but it is not limited to Envoy. Other gateways that follow the same external-processing protocol include Envoy-based Istio, Agentgateway, and NGINX Gateway Fabric.

When a request reaches Envoy, normal route matching happens first. However, the matched HTTPRoute now points to an InferencePool instead of a regular Service. The InferencePool is a new CRD that lists the model server pods and the ext-proc server managing them. Envoy’s role ends there. The routing logic is handled by the ext-proc server, called the Endpoint Picker (EPP), which collects metrics, scores the candidate pods running the serving engine, and decides which one should handle the request.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llama3-route
  namespace: llm
spec:
  parentRefs:
  - name: inference-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: vllm-llama3-8b
---
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: vllm-llama3-8b
  namespace: llm
spec:
  selector:
    matchLabels:
      app: vllm-llama3-8b
  targetPorts:
  - number: 8000
  endpointPickerRef:
    name: vllm-llama3-8b-epp
    port:
      number: 9002
    failureMode: FailClose

Pods that join a pool are expected to use the project’s model server protocol. This protocol is simple: it requires serving OpenAI-compatible Completions and Chat APIs and exposing Prometheus metrics like queue depth, running requests, and KV-cache utilization. The EPP uses these metrics to make routing decisions. As of now, vLLM, SGLang, and Triton’s TensorRT-LLM backend all support this, each with their own metric names.

How the endpoint picker (llm-d-router) works

As of 2026, the EPP codebase has moved to the llm-d project’s llm-d/llm-d-router (previously called llm-d-inference-scheduler). The version in the Gateway API Inference Extension will soon be archived. For this article, we will focus on the llm-d-router EPP, which has seen significant improvements in its scheduling logic since the move.

EPP and plugins

The EPP is highly configurable and now supports more than 80 plugins across five categories: scheduling, request control, flow control, request handling, and data layer. These plugins can be enabled in the configuration and are loaded at startup. For example:

kind: ConfigMap
apiVersion: v1
metadata:
  name: vllm-llama32-1b-epp
data:
  default-plugins.yaml: |
    apiVersion: llm-d.ai/v1alpha1
    kind: EndpointPickerConfig
    plugins:
    - type: queue-scorer
    - type: kv-cache-utilization-scorer
    - type: prefix-cache-scorer
    - type: metrics-data-source
      parameters:
        scheme: "http"
        path: "/metrics"
        insecureSkipVerify: true
    - type: core-metrics-extractor
    schedulingProfiles:
    - name: default
      plugins:
      - pluginRef: queue-scorer
        weight: 2
      - pluginRef: kv-cache-utilization-scorer
        weight: 2
      - pluginRef: prefix-cache-scorer
        weight: 3

Besides the plugins you define in the configuration, the EPP also loads several default plugins. For example, it adds max-score-picker if a profile does not specify a picker, and it includes the OpenAI, Anthropic, and vLLM-HTTP parsers if no parser is set. It also loads global-strict-fairness-policy, static-usage-limit-policy, and others, including any plugins needed to provide data for those you have configured. For instance, the prefix-cache plugin needs a tokenized prompt, which the EPP creates at startup even if it is not defined in the ConfigMap.

{"level":"info","ts":1785950733.484222,"caller":"loader/configloader.go:164","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"insecureSkipVerify\":true,\"path\":\"/metrics\",\"scheme\":\"http\"}} {Name: core-metrics-extractor, Type: core-metrics-extractor} {Name: single-profile-handler, Type: single-profile-handler} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: <nil>}, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"}

The EPP builds a dependency graph based on the data keys each plugin produces and uses. For every request, it runs the data-producing plugins in the correct order according to this graph.

What the EPP does out of band

Outside of the main request flow, the EPP sets up watches on the Kubernetes API for InferencePool, Pod, InferenceObjective, and InferenceModelRewrite resources. It scrapes metrics from each pod in the pool, emits its own metrics, and, when precise prefix-cache awareness is enabled, subscribes to each pod’s KV-cache event stream.

Precise prefix-cache awareness is especially useful because Prometheus metrics only show how full the KV-cache is, not which blocks are on which pod. If you subscribe to the serving engine notifications via a ZeroMQ PUB socket, you can receive updates about blocks (vLLM's unit of KV-cache by default fixed to 16 tokens and identified by a chained hash of its contents and everything before it) being added or evicted. These updates are decoded by an engine adapter stored in the EPP's memory. Remember, the serving engine must be started with KV-event publishing enabled. vLLM does not send these events by default, so each pod needs a flag like --kv-events-config '{"enable_kv_cache_events": true, "publisher": "zmq", "endpoint": "tcp://*:5557"}'.

When a request arrives

When a new request arrives at the EPP over the ext-proc stream, the EPP waits for the full body to arrive, buffering the chunks until the end-of-stream flag. This way, it has the complete JSON to process. Next, it selects a parser based on the URL path suffix. By default, three parsers are set up, each handling its own suffixes. If none match, the request is rejected:

• openai-parser claims completions, chat/completions, embeddings, responses, and their kind;
• anthropic-parser claims messages and messages/count_tokens;
• vllmhttp-parser claims inference/v1/generate.

The selected parser converts the raw bytes into a typed request, including the model name, messages or prompt, and the stream flag, and stores this in a per-request state object.

{"caller":"handlers/server.go:435","msg":"EPP received request","x-request-id":"a8a70838-..."}
{"caller":"handlers/server.go:448","msg":"Incoming body chunk","EoS":false}
{"caller":"handlers/server.go:448","msg":"Incoming body chunk","EoS":true}
{"caller":"handlers/server.go:454","msg":"decoding"}

One consequence of this design is memory: the EPP holds each request body fully in memory while it processes it, and it buffers non-streaming response bodies the same way, with no size cap of its own, so very large prompts cost gateway memory. Streaming responses are unaffected; the EPP forwards SSE chunks to the client as they arrive.

Next, the EPP looks for an InferenceModelRewrite resource matching the pool and the requested model, and applies its rules. The resource lets you define multiple weighted targets; think of it as HTTPRoute-style traffic splitting, but for models. For example:

apiVersion: llm-d.ai/v1alpha2
kind: InferenceModelRewrite
metadata:
  name: canary-model-split
spec:
  poolRef:
    name: production-llm-pool
  rules:
  - matches:
    - model:
        value: "llama3"  
    targets:
    - modelRewrite: "llama3-stable"
      weight: 90
    - modelRewrite: "llama3-canary"
      weight: 10

In case of match, the EPP changes the model field in the request body to the chosen target and re-serializes the body. On the way back, it replaces the model name in the response body with the original name, so the client never sees the rewrite. In the logs, incomingModelName is what the client requested, and targetModelName is what is actually used after any rule is applied:

{"caller":"requestcontrol/director.go:220","msg":"No associated InferenceObjective found, using default","objectiveKey":""}
{"caller":"requestcontrol/director.go:281","msg":"LLM request assembled","incomingModelName":"meta-llama/Llama-3.2-1B-Instruct","targetModelName":"meta-llama/Llama-3.2-1B-Instruct","priority":0}

Next, before the EPP performs tasks such as tokenization, hashing, or scoring, the default admission controller checks if the pool is saturated, and rejects requests with a priority below 0, set by an InferenceObjective with 429 status code. Requests with priority 0 or higher, which is the default if no InferenceObjective matches, always get through this check.

Note: This accept-or-reject behavior is just the default. With the experimental flow-control feature enabled, requests are queued by priority band and handled fairly within each band. They are dispatched in order of priority as capacity becomes available. The saturation signal is also a plugin, called utilization-detector by default, and its thresholds can be configured.

# PASS
{"level":"trace","caller":"requestcontrol/admission.go:116","msg":"Executing LegacyAdmissionController","objectiveKey":"sheddable-batch","priority":-1,"fairnessID":"default-flow"}
{"level":"trace","caller":"requestcontrol/admission.go:127","msg":"Request admitted","requestID":"4eb39a51-..."}
Rejection, mid-burst. Probes 1–5 got HTTP 429, and each produced this pair:
 
# DROPPED
{"level":"trace","caller":"requestcontrol/admission.go:76","msg":"Request rejected: system saturated and request is sheddable","x-request-id":"98bbfafe-...","objectiveKey":"sheddable-batch","priority":-1}
{"level":"error","caller":"handlers/server.go:478","msg":"Error handling request","error":"inference error: ResourceExhausted - system saturated, sheddable request dropped"}

If the request is admitted, the EPP evaluates the pods in the pool. It first takes a snapshot of all ready pods, then runs each plugin in order based on the configuration and dependency graph. With the previous ConfigMap, the process starts with tokenization: the EPP splits the prompt into 4-byte pseudo-tokens, groups them into blocks, and hashes each block together with all previous blocks. This way, each block’s hash represents the entire prefix up to that point, using the same chaining method as vLLM.

Note: A different configuration can change or skip this phase. If you use the precise prefix-cache plugin, the estimate becomes a lookup in the KV-event index from the out-of-band section. If you remove the prefix scorer entirely, nothing happens in this step.

The EPP then checks these block hashes in order against its index. It stops at the first block no pod is recorded for, counts how many blocks the index attributes to each pod, and records these counts for each candidate pod.

Then the scheduler takes over. The profile handler picks which scheduling profile applies (single-profile-handler always answers default), and the profile's filters prune the snapshot:

{"caller":"scheduling/scheduler.go:69","msg":"Running profile handler, Pick profiles","plugin":"single-profile-handler/..."}
{"caller":"scheduling/scheduler_profile.go:162","msg":"Completed running filter plugins","remainingEndpoints":2}

Each scorer then assigns a score between 0 and 1 to every remaining pod. For example:

• queue-scorer: ranks pods by waiting queue relative to each other, with the shortest queue in the snapshot getting 1, the longest getting 0, and if all queues are equal, everyone gets 1;
• kv-cache-utilization-scorer: rewards free cache space; a pod reporting 83% KV-cache usage scores 0.17, and an idle one scores 1;
• prefix-cache-scorer: rewards cached work with the share of this prompt's blocks the pod already holds, so 40 of 50 blocks scores 0.8, none scores 0.

{"caller":"scheduling/scheduler_profile.go:210","msg":"Calculated score","plugin":"queue-scorer/...","endpoint":"...8p55s...","score":1}
{"caller":"scheduling/scheduler_profile.go:210","msg":"Calculated score","plugin":"queue-scorer/...","endpoint":"...4r286...","score":1}
{"caller":"scheduling/scheduler_profile.go:210","msg":"Calculated score","plugin":"kv-cache-utilization-scorer/...","endpoint":"...8p55s...","score":1}
{"caller":"scheduling/scheduler_profile.go:210","msg":"Calculated score","plugin":"kv-cache-utilization-scorer/...","endpoint":"...4r286...","score":1}
{"caller":"scheduling/scheduler_profile.go:210","msg":"Calculated score","plugin":"prefix-cache-scorer/...","endpoint":"...8p55s...","score":0}
{"caller":"scheduling/scheduler_profile.go:210","msg":"Calculated score","plugin":"prefix-cache-scorer/...","endpoint":"...4r286...","score":0}

Each score is multiplied by its weight from the ConfigMap and then summed. The picker, which is max-score-picker in this example, shuffles the candidates to break ties randomly and selects the one with the highest total score.

{"caller":"scheduling/scheduler_profile.go:303","msg":"Candidate pods for picking","endpoints-weighted-score":[{"...8p55s...","Score":4},{"...4r286...","Score":4}]}
{"caller":"maxscore/picker.go:88","msg":"Selecting endpoints from candidates sorted by max score","max-num-of-endpoints":1,"num-of-candidates":2}
{"caller":"requestcontrol/director.go:488","msg":"Request handled","endpoint":"10.20.0.2:8000"}

The EPP sends the chosen pod’s address to Envoy in the x-gateway-destination-endpoint header and in dynamic metadata with the same key. Envoy then forwards the request body to that pod.

Fine-Tuning , LoRA adapters and the Inference Extension

More companies are now running inference on their own infrastructure, using open-weight models instead of relying only on services like Claude or ChatGPT. This approach offers practical benefits: lower costs at scale, more control over data, and the ability to adapt general-purpose models to specific company needs without having to fully fine-tune and deploy a separate model for each use case. Projects like Meta’s llama-cookbook, Microsoft’s LoRA, and Hugging Face’s PEFT have made this kind of specialization much more accessible, so regular engineering teams can now use techniques that once required a lot of compute, data, and ML expertise.

What is LoRA

The main idea behind LoRA is straightforward. Instead of updating every weight in a large model, you freeze the base model and train only small pairs of low-rank matrices, called “adapters,” which are attached to selected layers for a specific use case. This usually creates artifacts that are tens to hundreds of megabytes, not gigabytes. At each targeted layer, the input goes through both the frozen original weights and the adapter’s low-rank path, and the outputs are added together to produce the specialized result.

This method avoids having to create a separate multi-gigabyte copy of the model for each specialization. During inference, the adapter can be merged into the base weights or kept separate. This lets a LoRA-aware serving engine like vLLM dynamically apply different adapters on top of a single shared base model.

When the serving engine starts, vLLM sets up memory for LoRA adapters using several configuration options. For example, --max-loras sets how many adapters can be used in the same GPU batch, and --max-cpu-loras sets how many adapter weights can stay cached in host memory and be moved to the GPU as needed. The adapters are registered with --lora-modules, so a request can select one through the model field while still sharing the same loaded base model.

spec:
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model
        - Qwen/Qwen2.5-3B-Instruct
        - --port
        - "8000"
        - --max-model-len
        - "2048"
        - --gpu-memory-utilization
        - "0.85"
        - --max-num-seqs
        - "16"
        - --enable-lora
        - --max-loras
        - "2"
        - --max-cpu-loras
        - "3"
        - --max-lora-rank
        - "32"
        - --lora-modules
        - medical=calebking/qwen2.5-3b-instruct-medical-lora
        - korean-law=kim0924/qwen2.5-3b-korean-law-lora
        - reasoning=namanadep/qwen2.5-3b-instruct-reasoning-lora-bf16

EPP and LoRA

When you send a request for a LoRA adapter, the start of the EPP flow stays the same. The request goes to Envoy, is passed to the EPP over the ext-proc stream, and the OpenAI parser extracts the model from the model field and applies any matching InferenceModelRewrite, just as before.

curl -s http://35.216.6.64/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"korean-law","messages":[{"role":"user","content":"전세 계약이 무엇인가요?"}],"max_tokens":64}'

The difference comes when the EPP evaluates the endpoints in the InferencePool. Not every replica can serve a LoRA adapter request equally well. One replica might already have the korean-law adapter active, another might have space to load it, and another might be at its LoRA capacity. The EPP uses the lora_requests_info metric from the serving engine to find out which adapters are active or waiting on each replica, and the maximum number a replica can hold.

{"msg":"Refreshed metrics", "ts":1787719680.300, "endpoint":{"name":"vllm-qwen25-3b-lora-6888d54787-wjcb4-rank-0"}, "metrics":["…", "vllm:lora_requests_info", "…"], "updated":"{ActiveModels:map[korean-law:0] WaitingModels:map[korean-law:0] MaxActiveModels:2 …}"}

To do this, the lora-affinity-scorer plugin has to be added to the EndpointPickerConfig. Once it is set up, the plugin assigns a score to each candidate endpoint:

• 1.0 the adapter is already active on this endpoint
• 0.8 a free adapter slot: active + waiting < max_lora
• 0.6 the adapter is already queued to load on this endpoint
• 0.0 endpoint full: the request would wait for a slot to free

Note: This does not replace the other scheduling signals. The scheduler still considers queue depth, KV-cache utilization, prefix-cache locality, and other configured scorers. LoRA affinity just adds an extra score that shows how suitable each replica is for serving the requested adapter.

After the scorers have run, nothing new happens. Their values are multiplied by the configured weights and summed; the picker selects the winning endpoint, and the EPP returns that pod's address to Envoy in x-gateway-destination-endpoint.

One gateway, many models

According to the State of AI Engineering 2026, using multiple models is now standard. Across thousands of organizations running AI in production, more than 70 percent use three or more models, and the share running more than six has nearly doubled in a year. Teams are adding models instead of replacing them. In the same cluster, you might have one gateway in front of a Llama pool, a Qwen pool, and a Mistral pool, each focused on different tasks.

However, routing is more complicated because the OpenAI API schema puts the model name in the request body, and a body field is not something gateways can normally route on: they usually match only paths and headers. Body-based routing solves this problem. This extension extracts the model name from the JSON body and sets it as the X-Gateway-Model-Name header, which HTTPRoute can match like any other header. You can then use normal HTTPRoutes to route to the right InferencePool. For example:

rules:
- matches:
  - path:
      type: PathPrefix
      value: /
    headers:
    - type: Exact
      name: X-Gateway-Model-Name
      value: meta-llama/Llama-3.1-8B-Instruct
  backendRefs:
  - group: inference.networking.k8s.io
    kind: InferencePool
    name: vllm-llama3-8b

Operational reality

• It is best to use the llm-d router, even though the extension still includes its own. The Gateway API Inference Extension still has a working endpoint picker, and nothing will break if you keep using it. However, the code has moved to llm-d/llm-d-router, which is where new scheduling work is happening, and the old version will be archived. Point new pools to the llm-d EPP from the start, and treat switching an existing pool as a planned migration, not just an image update, since the plugin configuration does not transfer directly.

• You may be running plugins you did not declare. No matter what your ConfigMap lists, the loader adds its own plugins on top, such as a picker for any profile without one, the parsers, the policies, and any plugins needed by the ones you requested. The actual configuration is only known at runtime. The EPP prints the effective configuration once at startup, and logs any data producers it auto-creates as separate "auto-created default producer" lines just after it. Capture both, save them, and compare after every upgrade.

• Default prefix scoring is only an estimate, not an exact lookup. Without KV-event tracking, the EPP cannot see inside a pod’s cache. It fingerprints the prompt in 4-byte pseudo-tokens and scores each pod based on how much of that fingerprint it thinks the pod has. This ranking is helpful but not guaranteed. Check the hit rate from the serving engine, not the scorer, and switch to precise tracking if the two do not match.

A practical recommendation

If you already run several replicas of one model behind a plain Service, move that pool behind an InferencePool this week and let your own traffic guide the rest. You can pilot this in an afternoon: deploy the EPP with just queue-scorer and kv-cache-utilization-scorer, point one HTTPRoute at the pool, and keep the Service route so you can switch back easily. Replay your production prompt-length distribution against both setups and compare p95 TTFT, output tokens per second, and the engine’s prefill cache-hit rate. Add the prefix scorer in a second run, or you will not know which change made the difference. A prefix hit does not just speed up the cached part of prefill; it skips that computation entirely, and only the uncached tail of the prompt still runs prefill. On an H100 node that costs six figures a year, that difference matters. If you do not see an improvement in your traffic, do not adopt it.