AI infrastructure
Kinference Explained: A Control Plane That Knows Your Prompt
How Kinference sends each LLM request to the replica with the lowest predicted time to first token, benchmarked on GKE against a plain Kubernetes Service.
Ivan Porta
Founder & Principal Engineer

When it comes to LLM inference, standard load-balancing methods like round robin miss the mark. The key is which replica handles the request. If an 8,000-token prompt goes to a replica with a warm KV cache, the first token appears in about a tenth of a second. If it lands on a cold replica, it can take over a second.
Inference now surpasses training in AI-optimized IaaS spending. About 66% of the organizations running generative AI models use Kubernetes for some or all of their inference workloads. Datadog’s State of AI Engineering report highlights that 69% of input tokens are system prompts, but only 28% of LLM calls use any cache.
Kubernetes alone can’t solve this problem. A Service doesn’t know what a prompt is, and kube-proxy chooses a backend at random or in order if it’s using IPVS mode. With three replicas, a follow-up request will hit its own cache only about one out of three times.
Routing is just part of the challenge. The same GPUs handle coding assistants, batch jobs, and agents from different teams. Someone needs to decide who waits when queues grow, which requests can use a smaller model, and which should go to a cloud provider. These choices must be made for every request, before dispatch, and without needing a team of experts to manage it. That’s where Kinference helps.
What is Kinference, actually?
Kinference is an open source control plane for LLM inference on Kubernetes, announced by Buoyant on November 4, 2026. It doesn’t handle traffic directly. Instead, a gateway asks Kinference where to send each request, and Kinference replies with the replica that should return the first token fastest. It predicts the time to first token (TTFT) using metrics and KV-cache events from serving engines like vLLM and SGLang, and applies your policies to make the decision.
Kinference resources
Kinference is built around four main CRDs:
- An
InferenceWorkerSetdefines a group of identical engine replicas, where they are located, and how Kinference can monitor them. - An
InferenceProviderspecifies a destination outside the cluster, used when local workers can’t handle all requests. - An
InferencePolicycontains the rules for choosing between destinations for each request. - An
InferencePoolis the target for a route. It lists the possible destinations for requests and the policy that selects among them.
Let’s review each one and focus on the most important fields.
InferenceWorkerSet
An InferenceWorkerSet wraps the Service in front of a group of identical engine pods, called workers. Other fields tell Kinference how to monitor these workers, so it can see how much of a prompt each one already has in its KV cache and predict its TTFT. The main fields are:
serviceRef: the Service in front of the engine pods. Kinference finds the pods through it.kvEvents: where the engine publishes its KV-cache events over ZeroMQ. This block is optional, but without it, Kinference can’t see the caches. As a result, the predicted TTFT becomes much less accurate and Kinference will only route based on load.kvBlockSize: the number of tokens in each block of those events. It must match what the engine publishes.profile: the tuning profile. It sets how Kinference tokenizes the prompt, how it values an uncached block or a queued request in its prediction, and how it makes the final decision.metricsPort: the port where the engine serves its Prometheus metrics, from which Kinference reads each worker's load.modelDiscovery: how Kinference finds out which models the workers serve. In Endpoint mode (the default), it checks each worker’s/v1/modelsendpoint at regular intervals. In Static mode, it relies only on themodelslist.models: the aliases you can use to request a model, such as chat.
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferenceWorkerSet
metadata:
name: qwen-7b
namespace: llm
spec:
profile: default
kvBlockSize: 16
serviceRef:
name: qwen-7b
port: http
kvEvents:
port: kv-pub
metricsPort: http
modelDiscovery:
mode: Endpoint
interval: 30s
models:
- name: Qwen/Qwen2.5-7B-Instruct
aliases:
- chatInferenceProvider
An InferenceProvider defines a destination outside the cluster, like a hosted API such as Claude or DeepSeek. It’s usually used in policies that send overflow requests to the cloud instead of queuing them locally.
This resource only includes what Kinference needs to pick a provider: the wire format (api), the path to call, and the models, including their contextWindow and the alias they share with the local model. The URL and API key are stored in a GatewayBackend, a resource from the Buoyant Gateway, and backendRef points to it. This setup separates responsibilities: Kinference returns the backend name, and the gateway handles resolving the name, opening the connection, and adding credentials. Kinference never contacts the vendor or reads the key. That gateway can be any gateway that implements Kinference’s API. As of now, the Buoyant Gateway is the only one, which is why the backend is a GatewayBackend.
apiVersion: gateway.linkerd.io/v1alpha1
kind: GatewayBackend
metadata:
name: cloud-fallback
namespace: llm
spec:
baseURL: https://api.deepseek.com
auth:
type: ApiKey
apiKey:
secretRef:
name: deepseek-api-key
key: api-key
---
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferenceProvider
metadata:
name: cloud-fallback
namespace: llm
labels:
tier: fallback
spec:
backendRef:
group: gateway.linkerd.io
kind: GatewayBackend
name: cloud-fallback
namespace: llm
api: OpenAI
path: /chat/completions
models:
- name: deepseek-flash
aliases:
- chat
contextWindow: 1000000InferencePolicy
An InferencePolicy decides which destination in a pool will handle a request. It uses an ordered list of rules, and the first matching rule applies. A rule can match the requested model, the tenant (which the gateway reads from a header you choose), and the sizes of the prompt and answer. The route then goes through up to four blocks:
-
consider: This block builds the candidate set, which by default includes the requested model on every destination in the pool. You can narrow the set to local workers or providers (type), to specific destinations (backends), or to certain models. For example, you can keep sensitive prompts inside the cluster, as shown in the first rule for the sandbox tenant. If groups have weights, Kinference selects a group for each request based on those weights, and only that group provides candidates. The second rule acts like a canary: nine out of ten requests use only local workers, while the tenth can also use the hosted model. -
exclude: This block removes candidates based on their state, such as the number of queued prompts (queuedPrompts), the KV cache fill level (kvCacheUsage) averaged over the last 30 seconds, the predicted TTFT (predictedTTFT), and the context window (contextWindow). You can use anifcondition to limit exclusions to certain requests. -
select: This block ranks the remaining candidates in tiers. The first tier with a candidate is chosen. A tier ranks its candidates byPredictedTTFT(for local workers), byEstimatedTTFT(which includes providers using gateway-observed TTFT), or by a fixed order.tieBreakresolves ties:PreferLocalpicks a local worker over a provider, andRandomchooses randomly. If there’s notieBreak, Kinference picks the first candidate in order. -
onEmpty: This block decides what happens if no candidates remain. By default,Rejectrefuses the request.FallThroughmoves to the next matching rule.
In the example, the third rule allows batch jobs and long requests only on workers with enough capacity. The last rule serves all other requests locally and uses the provider only if no local worker is available. If a request doesn’t match any rule, it follows the unmatched policy, which by default sends it to the fastest local worker.
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata:
name: local-first
namespace: llm
spec:
unmatched: ServeLocal
rules:
- name: sandbox-stays-local
matches:
- model: chat
tenant: sandbox
route:
consider:
- name: local-qwen
type: Local
models:
names:
- Qwen/Qwen2.5-7B-Instruct
select:
- type: Local
minimize: PredictedTTFT
tieBreak:
- Random
onEmpty: Reject
- name: beta-compares-local-and-hosted
matches:
- model: chat
tenant:
- beta
- staff
route:
consider:
- name: local-only
weight: 90
type: Local
models:
fromRequest: true
- name: local-or-hosted
weight: 10
models:
fromRequest: true
backends:
names:
- qwen-7b
- cloud-fallback
exclude:
- name: no-hosted-model-for-long-answers
if:
maxTokens:
greaterThan: 4000
drop:
model:
name: deepseek-flash
- name: hosted-is-slow
drop:
backend:
type: Provider
observedTTFTP95:
greaterThan: 5s
select:
- minimize: EstimatedTTFT
tieBreak:
- PreferLocal
- Random
onEmpty: FallThrough
- name: heavy-requests-need-headroom
matches:
- qos:
- batch
- deferrable
- inputTokens:
greaterThan: 16000
- maxTokens:
greaterThan: 4000
route:
exclude:
- name: queue-too-long
drop:
backend:
type: Local
queuedPrompts:
greaterThan: 2
- name: kv-cache-too-full
drop:
backend:
type: Local
kvCacheUsage:
greaterThan: 0.7
- name: cold-prompt-on-a-filling-cache
drop:
backend:
type: Local
kvCacheUsage:
greaterThan: 0.5
prefixCacheHitRatio:
lessThan: 0.2
select:
- type: Local
minimize: PredictedTTFT
- name: everything-else
route:
exclude:
- name: first-token-too-slow
drop:
backend:
type: Local
predictedTTFT:
greaterThan: 30s
- name: context-window-too-small
drop:
contextWindow:
lessThan: 32768
- name: worker-set-down-to-one-pod
drop:
backend:
type: Local
readyEndpoints:
lessThan: 2
- name: sandbox-never-leaves-the-cluster
if:
tenant: sandbox
drop:
backend:
name: cloud-fallback
select:
- type: Local
minimize: PredictedTTFT
- backends:
type: Provider
selector:
matchLabels:
tier: fallback
order:
- cloud-fallbackInferencePool
The InferencePool brings the other three components together. An HTTPRoute uses it as its backend. The InferencePool lists the possible destinations for requests, either by name or label, and specifies the policy that selects among them. If there is no policyRef, each request is sent to the local worker with the lowest predicted TTFT, and requests do not leave the cluster.
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePool
metadata:
name: chat
namespace: llm
spec:
destinations:
- group: kinference.buoyant.io
kind: InferenceWorkerSet
name: qwen-7b
namespace: llm
- group: kinference.buoyant.io
kind: InferenceProvider
namespace: llm
selector:
matchLabels:
tier: fallback
policyRef:
group: kinference.buoyant.io
kind: InferencePolicy
name: local-firstHow Kinference works
Kinference does most of its work before any requests arrive. First, we’ll look at what it does in the background. Then, we’ll follow a request from when it arrives to how Kinference predicts where it should go.
What Kinference does out of band
Outside of the request flow, Kinference monitors its four CRDs along with the Services and EndpointSlices behind them. It keeps an up-to-date view of the fleet in memory.
For each InferenceWorkerSet, Kinference uses the serviceRef to find the ready pods behind the Service. Each pod is a worker. Kinference collects Prometheus metrics from each worker to track how many requests are running, how many are waiting, and how full the KV cache is. It also checks the worker’s /v1/models endpoint to see which models it serves. If the worker set has kvEvents, Kinference subscribes to the worker’s ZeroMQ socket and gets an event whenever a block is stored in or removed from the worker’s KV cache. These events help Kinference build an index of which blocks are on which worker, which later helps it tell a warm replica from a cold one.
Note: As we saw in From Ingress to Inference Gateway, vLLM does not publish these events by default, so the engine must be started with KV-event publishing enabled. kvEvents only tells Kinference where to listen.
At the same time, Kinference turns each InferencePool into a list of destinations and prepares the InferencePolicy that the pool uses. This way, the rules are ready to use as soon as a request comes in.
When a request arrives
It all begins with a route. The gateway matches the request to an HTTPRoute, but this time the backend is an InferencePool instead of a Service. Because of this, the gateway does not choose an endpoint itself and instead asks Kinference to do it.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: chat
namespace: llm
spec:
parentRefs:
- name: kinference-gw
sectionName: http
rules:
- matches:
- path:
type: PathPrefix
value: /v1/chat/completions
backendRefs:
- group: kinference.buoyant.io
kind: InferencePool
name: chatAfter that, a chat completion follows seven steps:
- Take the request. The client streams the request body and specifies the
InferencePoolfrom the route. Kinference stores the body in memory, up to 8 MiB per request and 256 MiB in total by default. If the request is too large, Kinference rejects it. - Find the model. Kinference reads the
modelfield from the request body and figures out which destinations can serve that model. - Measure the cache. Kinference tokenizes the prompt, splits the tokens into blocks of
kvBlockSize, hashes them like the engine does, and checks its index for those blocks. - Apply the policy. If there is no policy, any worker that serves the model is a candidate. If there is a policy, the first matching rule narrows down and ranks the candidates. The chosen candidate might even be a different model or a provider outside the cluster.
- Predict and commit. Kinference then predicts a TTFT for each candidate and chooses the one with the lowest value. It marks the request as in flight on that worker, so the next prediction takes it into account even if the engine has not updated yet.
- Return the request. Kinference responds with the endpoint and the request body. If the model changed, it rewrites the body before sending it back.
- Report the first and the last token. The client reports the TTFT it observed for the first token and, at the end, the total token counts including cached tokens. Kinference uses this information to update its in-flight tracking and to see how accurate its predictions were. For a provider, these reports are the only way Kinference can track its health and speed.
Predicting TTFT
By default, the predicted TTFT for a worker is made up of four parts:
- Base (
base_ms). A fixed overhead that every request pays. - Queue (
queue_ms_per_entry). This is the prefill work Kinference has already sent to the worker that is still ahead of the current request, adjusted for how far along each request should be. Kinference uses the two prefill fields below to price this work and chargesqueue_ms_per_entryfor each waiting request that it did not send itself. - Prefill (
attention_uncached_block_ms,attention_uncached_per_every_block_ms). This is the cost for blocks that the worker does not have cached. The first field is the cost per uncached block. The second is the cost per uncached block for every block in the prompt, since each new token attends to the entire prompt. - Decode contention (
forward_pass_ms). This is a small cost for each request that is already generating on the worker.
Two more penalties exist, and both are set to zero in the chart:
- KV-cache pressure (
cache_pressure_penalty_ms). This cost increases as the worker’s KV cache fills up. - One or two running requests (
q_running_penalty_peak_ms). This is a cost for a worker that is busy enough for a new prefill to compete for the GPU, but not busy enough for the prefill to join the running batch.
Kinference does not automatically learn these coefficients. You need to tune them for your model and hardware, and it’s important to get them right. When the system is busy, Kinference compares the wait time in a warm worker’s queue to the prefill time on a cold, idle worker. If the coefficients are wrong, it may make the wrong choice. Also, any policy rule that uses predictedTTFT with a threshold depends on having the right number. The benchmark below shows how the defaults compared to our hardware.
The coefficients live in a named profile that worker sets reference. This is the default profile from the chart's values:
profiles:
default:
predictor:
base_ms: 400.0
queue_ms_per_entry: 0.2
attention_uncached_block_ms: 2.3
attention_uncached_per_every_block_ms: 0.03
cache_pressure_penalty_ms: 0.0
q_running_penalty_peak_ms: 0
forward_pass_ms: 5
expected_overlap_alpha: 1.0
picker:
top_k_min: 1
top_k_fraction: 0.0
inflight_tracking: true
replication_threshold_pct: 10
replication_warm_target: 2
tie_tolerance_ms: 0
tokenize:
mode: renderTwo refinements cover bursts, where the KV index alone would mislead the prediction:
- A prefix that is still being computed already counts. The engine only publishes a block after it has finished computing it. If ten requests with the same new system prompt arrive in one second, the index will show that no worker has that prefix, even though one worker is already working on it. To handle this, Kinference keeps its own record of what it has just sent to each worker. The part of the prefix that should be computed by now is counted as cached, and the rest is counted as queue. As the prefill progresses, that worker becomes the best choice for the next requests, without waiting for the engine’s events.
- A prefix on a single worker gets a second copy. If only one worker has a prefix, all requests with that prefix go to that worker, making it a hotspot. To avoid this, when a cold worker’s predicted TTFT is within 10% of the warm worker’s (
replication_threshold_pct), Kinference intentionally chooses the cold worker. This usually happens when the warm worker is busy or the prompt is short. The first request covers the prefill cost, and after that, both workers have the prefix (replication_warm_target).
What cache-aware routing is worth
To measure the difference, we sent the same requests through two setups on a GKE cluster: a plain Service and Kinference with the default profile as shipped. Later, we repeat the benchmark using a profile tuned for this model and GPU. The cluster uses one node with an NVIDIA RTX PRO 6000, and HAMi splits its 96 GB of memory among three Qwen2.5-7B-Instruct replicas on vLLM 0.31, each getting 28,000 MiB. This gives each replica about 166,000 tokens of KV cache. Using one card keeps costs low, but since HAMi limits memory, not compute, a cold prefill on one replica also slows down the other two.
The load generator sends three kinds of traffic, which we call workloads. The first two repeat parts of their prompts, so a KV cache can help, and the third repeats nothing:
- Shared prefix. 38 system prompts of 8,000 tokens, each visited six times with a new question. Together they hold 1.8 times the tokens that one replica's KV cache can keep, so no single replica can keep all of them warm, but the fleet can.
- Multi-turn. 36 conversations of eight turns, where every turn carries the whole history.
- Control. The same sizes, with every prompt unique, so there is nothing to reuse.
Each workload starts with a cold fleet and keeps six requests running at once. To ensure a fair comparison, the load generator opens a new connection for every request, so kube-proxy selects a replica each time. If keep-alive were used, each client would stick to one replica, making the Service appear better than it actually is.
We measure TTFT at the client for requests that could use the cache: every visit to a prefix after the first, and every turn after the first. We ran the benchmark five times, each with new prompts and a different front door starting first. The results below show the mean of the five per-run medians, listing the Service first and Kinference second in each row:
| Median TTFT | Kubernetes Service | Kinference |
|---|---|---|
| Shared prefix | 889 ms | 128 ms |
| Multi-turn | 148 ms | 94 ms |
| Control | 1.17 s | 1.23 s |
The following diagram shows where the difference comes from: next to each median, the share of prompt tokens that came from cache, and the spread across the five runs.
How good was the prediction?
To verify the prediction, Kinference logged two times for each pick: the TTFT it predicted for the chosen replica and the TTFT the client actually saw. A pick is considered warm if the replica already had at least 90% of the prompt, and cold if it had none.
Using the default profile, the prediction correctly ranked cold replicas as much slower than warm ones. However, the predicted times were off: all 3,720 predictions were too high, with a median factor of 7. Below are the medians from the five runs, showing observed time first and predicted time second in each row:
| Picks | Observed | Predicted |
|---|---|---|
| 2,209 warm picks | 98 ms | 557 ms |
| 1,511 cold picks | 1.2 s | 9.1 s |
Three single picks show how the number comes together. The observed time is first again, and the prediction second, split into its terms:
| Pick | Observed | Predicted |
|---|---|---|
Warm on vllm-0: 495 of 500 blocks cached | 94 ms | 400 base + 86 prefill + 5 decode = 491 ms |
Cold on vllm-1: 0 of 498 blocks cached | 1,203 ms | 400 base + 8,586 prefill + 10 decode = 8,996 ms |
Cold and queued on vllm-2: 0 of 502 blocks cached, one cold prompt ahead | 1,723 ms | 400 base + 8,047 queue + 8,715 prefill + 15 decode = 17,177 ms |
The issue comes from the profile. The default coefficients are based on another fleet, not a 7B model on an RTX PRO 6000. For example, the base term is 400 ms, which is four times longer than what a warm request actually takes here.
Whether this error matters depends on the load. Kinference only compares candidates to each other. When queues are empty, prefill time is the only difference between workers, so even inaccurate predictions keep the order right and Kinference still makes the correct pick. Under load, Kinference must balance queue time against prefill time for a cold prompt, so the coefficients need to be accurate. For benchmarks under load, start by tuning the profile to match this model and GPU until predicted and observed TTFT are close.
What a tuned profile changes
We tuned the profile by updating four of its coefficients to match this model and GPU. The values were not chosen manually. For each pick, Kinference logs the predicted TTFT, the observed TTFT, and the prediction inputs, like uncached blocks and requests ahead. A script processed the 3,720 logged picks, recalculated predictions for many coefficient sets, and kept the set that best matched the observed TTFT. The default value is listed first and the tuned value second in each row:
| Coefficient | Default | Tuned |
|---|---|---|
base_ms | 400 | 55 |
attention_uncached_block_ms | 2.3 | 1.28 |
attention_uncached_per_every_block_ms | 0.03 | 0.002 |
forward_pass_ms | 5 | 17.4 |
On a new cluster with the same setup, we ran Kinference’s part of the benchmark three times with each profile, using the same prompts. The default profile again predicted a median TTFT 7.2 times higher than observed. The tuned profile predicted 1.02 times the observed TTFT, with a median error of 27 ms compared to 519 ms.
Performance stayed the same because the picks did not change. Out of 2,232 requests, 2,231 received the same type of pick (warm or cold) with both profiles, so the median TTFT for each workload remained unchanged. This also held true when we doubled the load on the shared-prefix workload to 76 prefixes and twelve requests in flight. At that load, the tuned profile made Kinference wait for a busy warm replica more often, 221 times versus 173 out of 1,368 requests, but this was not enough to affect the median.
What the tuned profile changes most are the rules based on the predicted TTFT. For example, a policy might drop candidates whose predicted TTFT is above a certain threshold. With the default profile, a five-second threshold would have dropped every local replica for 36% of requests, even though no request actually waited more than 2.3 seconds. With the tuned profile, no prediction exceeded five seconds.
Operational reality
-
The gateway has to speak its API. Kinference is not limited to one gateway, but currently, the Buoyant Gateway is the one that implements its API and serves only
HTTPRoute. Also, check what the Service exposes: by default, it is typeLoadBalancer, which gives your model a public address on a cloud provider. To keep it inside the cluster, use the annotationgateway.buoyant.io/service-type: ClusterIPon theGateway. -
A misconfigured engine degrades Kinference without telling you. vLLM requires
--kv-events-configwith the ZeroMQ publisher,--enable-prompt-tokens-detailsto report cached tokens, and, for vLLM 0.31,--enable-scale-out, since that version only serves the render endpoint with this flag. In our first test, we missed the last flag. All pods were Ready, the worker set and pool reportedAccepted, and all 744 requests succeeded, but Kinference could not tokenize any prompts and acted like a plain Service. The only signs were thekinference_score_degraded_totalcounter and a single warning in the log. Set up an alert for that counter. -
kvBlockSizehas to match what the engine publishes. Kinference cannot automatically detect the block size, and the correct value is not always the engine’s--block-sizeflag. If they do not match, Kinference will evict the worker after a few mismatches, which can leave the pool with no available workers. After any engine or model change, check the worker set’sstatus.kvEvents,kinference_index_blocks, andkinference_worker_evictions_total. -
The default profile ships with someone else's numbers. With the default profile, every prediction on our cluster was too high by a median factor of 7. The cause was the profile’s coefficients alone: as shown earlier, the same predictor was at 1.02 times the observed TTFT once the profile was tuned. Enable observation events, compare predicted and observed TTFT on your own traffic for a week, and tune the profile before setting thresholds based on
predictedTTFTor relying on picks under load.
A practical recommendation
If you have several replicas of a model behind a plain Service, or want to apply policies to your inference traffic by keeping sensitive queries in your local models or by bursting requests to an external inference service to avoid high latency, try Kinference on one non-critical pool this week. Then, replay your real prompt mix against both the pool and the Service, and compare the median and p95 TTFT along with the engine’s actual prefix-cache hit rate, not just the picker’s estimate.
Before trusting the numbers, check three things. First, kinference_score_degraded_total should stay at zero. If it increases, Kinference cannot tokenize your prompts and is routing based only on load. Second, the worker set’s status.kvEvents should show every pod as subscribed, and kinference_index_blocks should increase with traffic. If not, the engine may not be publishing KV events, or kvBlockSize may be incorrect. Third, make sure your load generator opens a new connection for each request. If you use keep-alive, kube-proxy will pin each client to one replica, making the Service look better than it really is.
After running the pilot, adjust the profile before creating a policy. Enable observation events, then compare the predicted TTFT to the actual TTFT from your traffic. Adjust the coefficients until they match. In our tests, tuning did not reduce TTFT, but it does determine if a predictedTTFT threshold removes a healthy replica.
If your prompts share prefixes, like system prompts, agent loops, or multi-turn chats, you will notice the difference within the first hour. If they do not, you will find that out after a few hours of testing. On GPU nodes that cost hundreds of thousands of dollars per year, recomputing a prefix already held by another replica wastes capacity you have already paid for.
Frequently asked
Four questions to settle before you pilot Kinference.
Do we need Kinference for a single vLLM replica?
No. With one replica there is nothing to choose between. It starts to pay off with several replicas, several models, or several tenants.
Which inference engines does it work with?
vLLM and SGLang. Kinference reads both engines' metrics and KV-cache events, although the documentation leads with vLLM.
Does it work with our existing gateway?
Only if that gateway implements Kinference's gRPC API. The Buoyant Gateway does today.
Can Kinference send requests to a cloud provider?
Yes. An InferenceProvider declares an external API, and a policy can fall back to it. The gateway dials it and holds the credentials.


