KubeCon China 2026: Capacity Recovered From The Same Hardware

KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 in Shanghai ran on two ideas. The first was a workload shift the opening keynote put analyst figures behind: AI compute is now weighted to inference rather than training. Serving it has strained the abstractions Kubernetes was built on, and that complaint ran right through the program: a Deployment does not describe an inference replica, and time to first token is not a request rate. The second idea was enforcement, a layer below the first and largely indifferent to it. HAMi splits the GPU and enforces each workload’s share, while DRA gives Kubernetes a native way to request and account for those slices, whether the workload is training or inference. The two met at the spending line: teams that needed more capacity got it by changing the unit they allocate, not by buying hardware.

What the keynotes framed

The keynotes put that shift in numbers. On the analyst figures cited from the stage, inference now takes 67% of AI compute against 33% for training, and the reason given was frequency rather than size: training happens sporadically, inference happens on every prompt, and agent workloads are inference hungry. The same keynote projected inference power demand reaching 93.3 GW by 2030. It also mapped each cloud-native primitive onto an AI-era counterpart: containers plus sandboxes for isolation, microservices plus agents, service mesh plus agentic protocols such as MCP, declarative APIs plus model-as-a-service, and immutable infrastructure plus SLA-aware infrastructure.

The PyTorch Foundation keynote argued that the frontier is now set by how fast teams iterate. The old linear pipeline, train a model and then serve it, has been replaced by a loop in which production behaviour feeds back into the weights, and the evidence offered was day-zero serving support in vLLM and SGLang for Kimi K3, MiniMax M3 and Qwen 3.8. The keynote's worked example was Shopify, running that loop daily on its commerce agent: production failures become training data, a calibrated judge critiques, repairs and validates the trajectories, PyTorch retrains on them with supervised fine-tuning followed by reinforcement learning, and vLLM serves the updated weights. On their GraphQL evaluation the distilled model passed the frontier reference baseline somewhere between 46,000 and 61,000 training examples. The headline figure, a ~96% serving-cost reduction at up to 2,000 requests per minute, was presented as a modeled comparison of serving cost, not a measured total including training.

HAMi is a clear example of this lower layer taking shape. It allows multiple workloads to share a single GPU card, with each getting its own memory and compute quota. It sits as an open GPU resource layer beneath the schedulers rather than beside them, with Volcano, Koordinator and Kueue above it and a KAI Scheduler integration below. A DRA driver now expresses the same slicing through Kubernetes' own claim API, translating the older device-plugin requests rather than sitting next to them, and the project is explicit that the two modes should not run in one cluster at once. HAMi was named in five sessions on the opening day, and three of the production reports below run it.

What the enterprise talks actually showed

China Merchants Bank faced the problem in its plainest form: one accelerator pool, a training workload that wants stable capacity and an inference workload that wants rapid elasticity, both competing for the same cards. Rather than buy more accelerators, they scheduled both on one shared base. Training arrives through Kueue for admission and quota; inference runs vLLM or SGLang and scales on KEDA against Prometheus metrics. Beneath both sit the Kubernetes scheduler with HAMi for placement, and Fluid for datasets, checkpoints and cached weights. The tidal behaviour is the point: when inference demand rises, KEDA scales out, and training jobs checkpoint themselves and release their cards, resuming when demand falls. They reported average accelerator compute utilization rising from 35% to over 60%, inference cost per one million input plus output tokens down by more than 60%, fivefold multi-tenant training density, and Qwen3-14B loading in under two seconds from Fluid-cached weights, against production targets of TTFT at or below 5 seconds and TPOT at or below 50 milliseconds. Their own summary line was that the value comes from coordination across layers rather than from any single component. They were equally clear about the cost of assembling this rather than buying a platform: ongoing integration and compatibility testing. The work won CNCF's End User Case Study Contest at the event.

Intsig, the company behind CamScanner, met the same problem at fleet scale: a 10,000-card estate, self-built plus multi-cloud, carrying more than 1,000 online inference services and more than 1,000 offline training jobs in parallel, with the training side above 90% utilization over 24 hours. Their constraint was not placement but queue time, people waiting for a card. The practices they credited are slicing first, with whole cards kept for heavy and large-model work, binpack placement to fill partly used GPUs rather than scatter half-card fragments, affinity pairing so two high-load workloads never share a card, a monitoring loop, and elastic scaling. Together those are credited with 50% higher GPU utilization at 30% lower overall cost, and inference performance degradation held under 10%.

Viettel put it most plainly. The Vietnamese operator had about 2,500 GPUs of many generations sitting behind team-by-team assignment, and one flagship card running a single inference service at full load reached roughly 10% utilization. Their diagnosis was that the limit had never been the hardware but the unit of allocation. The four-layer stack they built on that reading provisions bare metal with OpenStack Ironic and Nova, orchestrates with Kubernetes plus HAMi slicing and KEDA, serves with vLLM, SGLang, KServe and llm-d, and meters tokens through Envoy AI Gateway. A team now gets an API key and an OpenAI-compatible endpoint in minutes, instead of waiting weeks for a server. On one production card, same workload and same service level, utilization went from 13% to 59%. They presented that as the cause and the rest as what follows from it: 3.5x more work per card, which is the same result stated as 71% fewer GPUs for the same throughput on the workload they measured, and separately 2.1x work delivered per Watt.

More useful still was where they said not to do it. Sharing returned 3.3x on a forecasting service whose request size was capped upstream and never moved past about 10% of a card. It ran 1.6x worse on a model whose engine already batched continuously, and came out at 0.77x aggregate on compute-bound training, worth doing only for isolation. Their caveat is that the figures are theirs. What transfers is the shape of the workload: where the engine already keeps the card busy, sharing only divides it.

Alibaba made the same trade at the cluster layer rather than the card. Qwen is served across Karmada federations, where the migration plan they showed moves online units one at a time, each round waiting a five-minute stability window, reserving capacity on the target before it moves a unit and holding the old slot so a failed round can roll back. A rebalancer re-splits batch inference as spot capacity moves between clusters. They reported roughly 15% higher allocation rate, roughly 30% higher spot GPU utilization, and cross-cluster migrations that took days finishing in minutes.

The replica became a group

One inference replica is no longer reliably one pod, and that is what makes these workloads hard to schedule. The LeaderWorkerSet (LWS) maintainer session, run by DaoCloud and Huawei Cloud, set out a table of model sizes: Kimi K3 at 2.8 trillion parameters, DeepSeek V4 Pro at 1.6 trillion, GLM 5.3 at around 750 billion. Their point was that model weights of that size now exceed the HBM of a single node, so one replica becomes a group of pods spread across machines. Neither a Deployment nor a StatefulSet expresses a leader plus its workers as that unit of replication, which is what LWS provides: replicas multiplied by group size, injected leader addresses, and a restart policy that by default recreates the whole group when one pod fails. DisaggregatedSet sits above it, one service definition producing slices that each contain every role, each child LWS named by slice, revision and role so a rollout never leaves a new prefill talking to an old decode. The scheduling failure underneath is specific: schedule those pods independently and every GPU slot can fill with leaders while no replica ever becomes ready. Volcano answers it by making an LWS group a PodGroup, so the group is admitted as one unit or not at all, with the scheduler-plugins coscheduler doing the same and YuniKorn reaching it through its own task groups.

Huawei brought the same structure to a consumer assistant. The inference plane behind its Celia assistant is three nested levels: a ModelServing holds serving groups, and each group holds roles, so a multimodal model runs vit, prompt and decode roles with different pod shapes, held at a working ratio. Gang policy compiles to a Volcano PodGroup, topology policy pins prefill and decode to the same rack, and a ratio constraint holds the prefill to decode ratio between 1 and 3 as each role scales on its own metrics. Eviction protection was the detail worth keeping: a validating webhook checks each logical unit ahead of the native disruption budget, and where a single group holds the only instance of a role the eviction is rejected until an operator intervenes or capacity is added. NPU utilization moved from 30.1% to 40.7%, and the automated operations and maintenance rate for NPU resources from 63% to 87%. The Kthena router, a Volcano-community project rather than a Huawei one, was benchmarked against llm-d's KV-cache-aware routing across seven configurations, with the losses shown: two came out worse than the baseline on every measure, two lost only on time to first token, and the best three improved on all six, the strongest reaching 25.1% higher request throughput and 23.8% lower TTFT.

The layer that was not about AI

Samsung SDS took the same question to the physical layer. They offer heterogeneous bare metal and VM infrastructure as a service from 17 cloud data centres, with operations across 40 countries, and every new product had been landing on the same underlay cluster. The cost was never the object count, it was the coupling. One node ended up running Multus, Calico and OVN together, two CSI drivers and two container runtimes, with isolation stopping at the namespace. Kubernetes and driver versions moved together, so one version bump meant a full revalidation, and a data-plane failure in one workload took the others down with it. Their answer was to split the underlay into purpose-built clusters and carry the cost of running many with Cluster API. Each purpose cluster's control plane runs as pods in a shared control cluster, etcd included, with Konnectivity tunneling API server traffic between two overlay networks with no route to each other. Version and CNI are then chosen per cluster. What ties this back to the rest of the event is what it buys them. Because node composition is declarative, one commit moves nodes from an idle cluster to a busy one, and the use case they showed was a daily cycle: GPUs weighted to inference during business hours and to training overnight. The economics they gave for it are blunt, that an idle node is paid-for hardware earning nothing and utilization is the margin.

The themes worth taking home

Four things are worth acting on. First, measure what one card actually delivers under production load before buying another, because three of the teams on stage started somewhere between roughly 10% and 35% utilization, and the one that said so most plainly got past it without buying anything. Second, change the unit of allocation before the budget, whether that means a slice of a card with HAMi, a leader-worker group with LeaderWorkerSet, a role inside a serving group, or a node moved between clusters. Third, test sharing rather than assuming it. Viettel's three results, 3.3x on a workload that could not fill a card, 1.6x worse on one that already could, and 0.77x aggregate on compute-bound training, are the same question asked of three shapes of workload, and what decides it is whether the engine already keeps the card busy. Fourth, if inference already spans more than one node, move to a group-shaped workload API and gang scheduling now rather than retrofitting them later, because separating prefill from decode has become an API surface of its own in DisaggregatedSet and in Huawei's ModelServing, above what LeaderWorkerSet gives you. Allocation was not the only lever in Shanghai, since caching, compression and routing each produced large gains of their own, but it was the one that recurred, and the resource underneath all of them is accelerator time.