HAMi Explained: Why Your GPUs Sit Idle While Your Bill Doesn't

Before the release of Dynamic Resource Allocation and its alpha partitionable-device and consumable-capacity extensions, Kubernetes scheduled GPUs only as whole cards, with no native option to allocate a fraction of a device. For example, if a pod requires only 4 GB of an 80 GB H100, it still reserves the entire GPU for its lifetime. This leads to significant underutilization of expensive hardware in shared clusters. Chatbots designed for peak traffic may remain idle on dedicated cards during off-peak hours; multiple inference services often run on separate GPUs despite low usage, and monitoring dashboards may show all GPUs allocated while tools like nvidia-smi often reveal minimal actual utilization. As a result, new budget requests for additional GPUs arise because the cluster appears to be at capacity. DRA narrows this gap by preventing over-committing a device's declared budget, but it does not natively enforce resource limits once a pod is running, which remains the device driver's responsibility. This is where HAMi provides value.

HAMi (Heterogeneous AI Computing Virtualization Middleware) closes the gap by allowing pods to request an exact portion of a GPU’s memory and compute, with an in-container enforcement layer that holds them to it, with no changes to applications or images. Companies like SNOW and NIO run HAMi in production, and the project reached CNCF Incubating in July 2026.

What is HAMi, actually?

HAMi is an open-source GPU virtualization layer for Kubernetes. It lets a pod request a precise amount of memory and an AI accelerator’s compute, then schedules the pod onto a device with that much free capacity, binding it to that slice at runtime. The same model spans many accelerators, including NVIDIA GPUs, Cambricon MLUs, Hygon DCUs, Ascend NPUs, Moore Threads, MetaX, and others, each via its own device plugin and isolation backend. On NVIDIA, for example, HAMi replaces the stock device plugin and enforces the memory cap by intercepting the CUDA driver API within the container; other vendors provide analogous library- or hardware-level isolation.

The user experience is deliberately simple: the standard device resource request is extended with one or more limits that the scheduler reads at scheduling time to find a card with sufficient free memory and compute resources. On NVIDIA, you keep requesting nvidia.com/gpu and add the two extended resources beside it: nvidia.com/gpumem and nvidia.com/gpucores.

resources:
  limits:
    nvidia.com/gpu: 1       
    nvidia.com/gpumem: 8000 
    nvidia.com/gpucores: 30 

Two comparisons put that isolation model in context. MIG draws a hardware boundary: each instance gets its own SMs and a dedicated route through L2 cache slices, memory controllers, and DRAM channels. HAMi’s boundary is a fence, not a wall. Its limits are enforced in software inside the container, which reliably holds workloads to their budgets but cannot contain a device-level failure: a hung process, a driver reset, or an XID error on the shared card still hits every tenant on it. The two work at different layers, though, and they can be complementary rather than competing; we will come back to how HAMi can drive MIG directly.

Compared with time-slicing, however, HAMi offers significant improvements in stability. Time-slicing simply advertises multiple schedulable replicas of a single card and interleaves their workloads on the compute engine, without per-replica memory caps or isolation. That makes it dangerously easy for one workload to allocate past its notional share and take the others down with it. HAMi refuses the excess allocation instead. The following log shows PyTorch reporting a total capacity matching the 1000 MB limit set by nvidia.com/gpumem; when the pod tried to exceed it, the allocator stopped it with an OOM error, leaving the other vGPU tenants unaffected:

(EngineCore pid=95) ERROR 07-11 21:15:46 [core.py:1231] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 1000.00 MiB of which 12.00 MiB is free. Including non-PyTorch memory, this process has 1020.00 MiB memory in use. Of the allocated memory 864.68 MiB is allocated by PyTorch, and 41.32 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

How a shared-GPU request flows through HAMi

From manifest to enforced slice, a request passes through four hands, shown in the diagram below: the device plugin advertises each card’s capacity, the mutating webhook reroutes the pod to HAMi’s scheduler, the scheduler extender picks a node and a slice, and an in-container library enforces that slice at runtime.

On each node, the device-plugin (which runs as a DaemonSet) polls the local devices every 30 seconds via the vendor's management library (e.g., NVML for NVIDIA, which powers nvidia-smi). It then publishes this telemetry data by patching the node with a hami.io/node-<vendor>-register annotation.

kubectl get nodes gtrekter -o yaml
apiVersion: v1
kind: Node
metadata:
  annotations:
    hami.io/node-handshake: Requesting_2026-07-09 09:03:57
    hami.io/node-nvidia-register: '[{"id":"GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e","count":10,"devmem":6141,"devcore":100,"type":"NVIDIA GeForce RTX 4050 Laptop GPU","mode":"hami-core","health":true,"devicepairscore":{}}]'
    nvidia.com/gpu-driver-upgrade-enabled: "true"
  ...

When a pod is created, the API server calls HAMi's mutating webhook, which checks the pod's manifest against its supported device drivers, including NVIDIA, Ascend, AWS Neuron, and others. Each driver checks the container's resources.limits for recognized resource keys. For example, the NVIDIA driver looks for nvidia.com/gpu, nvidia.com/gpumem, and nvidia.com/gpucores, while the Cambricon MLU driver matches cambricon.com/vmlu. As soon as a compatible request is detected, HAMi updates the pod's schedulerName to hami-scheduler, ensuring the pod is handled by HAMi's device-aware scheduling logic rather than the default Kubernetes scheduler.

Note: Under HAMi, requesting nvidia.com/gpu: 1 no longer means "one whole card," but rather "one shared physical card," with the memory and core limits dictating the exact slice. However, a pod that omits nvidia.com/gpumem falls back to the configured default (100% of the card's memory).

kubectl logs -n hami-system deploy/hami-scheduler -c vgpu-scheduler-extender
I0710 13:51:38.552275  config.go:111] Initializing NVIDIA device
I0710 13:51:38.552355  config.go:248] All devices initialized successfully
...
I0710 14:02:14.161116  devices.go:647] "Resource requirements collected" pod="default/chat-qwen-64c4bd66f9-8qccs" requests=[{"NVIDIA":{"Nums":1,"Type":"NVIDIA","Memreq":4000,"MemPercentagereq":101,"Coresreq":60}}]

The vgpu-scheduler-extender builds a live usage map for every physical GPU in the cluster. It reads device metadata from each node's hami.io/node-nvidia-register annotation, then loops through the pods already assigned to that node to calculate the consumed GPU memory and compute capacity.

kubectl logs -n hami-system deploy/hami-scheduler -c vgpu-scheduler-extender
...
I0710 14:02:14.161182 node_policy.go:82] node node1 used 0, usedCore 0, usedMem 0,
I0710 14:02:14.161191 gpu_policy.go:104] device GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e user 0, userCore 0, userMem 0,
...
I0710 16:54:54.475656 node_policy.go:82] node node1 used 1, usedCore 60, usedMem 4000,
I0710 16:54:54.475667 gpu_policy.go:104] device GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e user 1, userCore 60, userMem 4000,

The scheduler then filters out any nodes lacking sufficient capacity and scores the remaining nodes based on the configured HAMi scheduling policy (nodeSchedulerPolicy and gpuSchedulerPolicy; the chart defaults are binpack at the node level and spread at the GPU level). The selected node and chosen GPU slice are recorded in the pod's annotations (hami.io/vgpu-devices-allocated) as a comma-separated list of UUID,Type,memMB,cores records.

$ kubectl describe pod -n default embed-bge-f84696cc9-7587j
Name:         embed-bge-f84696cc9-7587j
Labels:       hami.io/vgpu-node=node1
...
Annotations:  hami.io/vgpu-devices-allocated: GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e,NVIDIA,2000,30:;
              hami.io/vgpu-devices-to-allocate: ;;
              hami.io/bind-phase: success
              hami.io/bind-time: 1783702494
...

Once the pod is bound to a node, the Kubelet calls Allocate on that node's HAMi device-plugin. The plugin reads the pod's hami.io/vgpu-devices-to-allocate annotation (cleared once the plugin has consumed it, which is why the describe output above shows it empty) to decode the requested slice, identifying the target physical device and its exact compute and memory limits. It then returns a response detailing what the container needs to run against that specific slice: environment variables, host-path mounts (including the enforcement library), and device handles.

kubectl logs -n hami-system ds/hami-device-plugin -c device-plugin | grep -i 'Allocate Response'
...
I0710 16:54:54.499747  232069 server.go:719] Allocate Response [envs:{key:"CUDA_DEVICE_MEMORY_LIMIT_0" value:"2000m"} envs:{key:"CUDA_DEVICE_MEMORY_SHARED_CACHE" value:"/usr/local/vgpu/584ed76a-0856-40b3-8113-1ad696daa252.cache"} envs:{key:"CUDA_DEVICE_SM_LIMIT" value:"30"} envs:{key:"LIBCUDA_LOG_LEVEL" value:"1"} envs:{key:"NVIDIA_VISIBLE_DEVICES" value:"GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e"} mounts:{container_path:"/usr/local/vgpu/libvgpu.so" host_path:"/usr/local/vgpu/libvgpu.so.v2.9.0" read_only:true} mounts:{container_path:"/usr/local/vgpu" host_path:"/usr/local/vgpu/containers/1a5bac44-7888-4ea5-9382-3ef49ae1f1ba_tei"} mounts:{container_path:"/tmp/vgpulock" host_path:"/tmp/vgpulock"} mounts:{container_path:"/etc/ld.so.preload" host_path:"/usr/local/vgpu/ld.so.preload" read_only:true}]

The Kubelet applies this response directly to the container during creation. From that point, HAMi's preloaded control library sits between the application and the vendor driver, enforcing the slice in-process. It intercepts the driver's memory-allocation calls, checks each request against the slice's budget, and returns an OOM error if the workload attempts to exceed its cap. It also intercepts and rewrites driver management queries (like nvidia-smi), ensuring the reported total capacity reflects the assigned limit, and the free capacity reflects that limit minus current usage.

kubectl exec -n default chat-qwen-64c4bd66f9-8qccs -c vllm -- env
CUDA_VERSION=13.0.2
NVIDIA_VISIBLE_DEVICES=GPU-a2fd58d2-edbe-4cfd-4dbd-d4197b35431e
CUDA_DEVICE_MEMORY_LIMIT_0=4000m
CUDA_DEVICE_SM_LIMIT=60
CUDA_DEVICE_MEMORY_SHARED_CACHE=/usr/local/vgpu/5e999bc3-2dbf-46b0-ba81-cc479a1c11dd.cache
...
 
kubectl exec -n default chat-qwen-64c4bd66f9-8qccs -c vllm -- cat /etc/ld.so.preload
/usr/local/vgpu/libvgpu.so
 
kubectl exec -n default chat-qwen-64c4bd66f9-8qccs -c vllm -- nvidia-smi --query-gpu=memory.total --format=csv,noheader
4000 MiB
[HAMI-core Msg(272:133498069993280:multiprocess_memory_limit.c:703)]: Cleanup on exit for PID 272
[HAMI-core Msg(272:133498069993280:multiprocess_memory_limit.c:739)]: Exit cleanup complete for PID 272

Scheduling policies that move the bill

As mentioned earlier, the scheduler first filters out nodes lacking sufficient capacity. For the remaining candidates, HAMi determines the optimal placement using two distinct scheduling tiers: the node level and the GPU level. Currently, HAMi supports two primary strategies for each: binpack (pack tasks as densely as possible) and spread (distribute tasks as widely as possible).

metadata:
  annotations:
    hami.io/node-scheduler-policy: "binpack"
    hami.io/gpu-scheduler-policy: "binpack"

Both policies come with distinct operational and financial trade-offs.

Node-Level Strategies

At the node level, the binpack policy packs workloads densely onto the fewest possible machines. This approach enables cluster autoscalers to quickly drain and spin down empty nodes, directly cutting cloud infrastructure bills. However, this density concentrates tenants, meaning a single node or driver failure has a much larger blast radius, and leads to increased contention with less bandwidth headroom.

Conversely, the spread policy distributes workloads evenly across all available nodes. While this successfully lowers per-node contention to provide headroom for unexpected traffic spikes and securely distributes fault domains, it keeps every node "warm." As a result, the autoscaler cannot scale down the cluster, leaving you paying for idle capacity.

GPU-Level Strategies

At the GPU level, the binpack policy completely fills one physical card before scheduling workloads on the next. This preserves entirely free GPUs for large models that require undivided resources and maximizes the number of small slices sharing a single card. The inherent trade-off is that it creates the densest possible cards, maximizing "noisy-neighbor" pressure and exposing a high number of tenants to single-card XID faults.

Alternatively, the spread policy balances workloads across all available cards. This minimizes compute and memory bandwidth contention, making it the superior choice for latency-sensitive inference workloads. The downside is that it fragments available memory across the cluster, meaning a subsequent large job requiring massive, continuous memory might fail to schedule even if the total free memory across the cluster is technically sufficient.

These are placement-time policies only: HAMi scores new pods as they arrive; it does not actively migrate running workloads. Financial savings therefore materialize as pods churn, meaning binpack pays off aggressively on highly elastic inference fleets, but much less so on long-lived, pinned pods.

HAMi and MIG: complementary layers

The fence and the wall operate at different layers, and you do not have to pick one per cluster: HAMi can manage both from a single pool. With dynamic MIG enabled, HAMi drives nvidia-mig-parted to carve and re-carve MIG geometry to fit incoming jobs, removing manual nvidia-smi mig work and pre-committed slice layouts. Pods keep requesting nvidia.com/gpu and nvidia.com/gpumem exactly as before; by default, the scheduler may satisfy that with either a hami-core slice or a MIG instance, and a workload that needs hard isolation can pin itself to the wall with the nvidia.com/vgpu-mode: "mig" annotation. The trade-off is granularity: MIG placements round up to the nearest allowed geometry (an 8 GB request on an A100-40GB lands on a 2g.10gb instance), where hami-core would have fenced off exactly what you asked for.

Operational reality

• HAMi replaces your device plugin. The HAMi device plugin and the standard nvidia-device-plugin both want to own nvidia.com/gpu; running both on the same node causes them to fight over resource registration, leading to capacity gaps between real and virtualized counts, and GPU allocation is served by whichever plugin registered last, silently bypassing HAMi’s isolation. If you run the GPU Operator, keep it for the driver, container toolkit, DCGM, and disable its device plugin on HAMi-managed nodes. Scope HAMi with the gpu=on node label and roll it pool by pool, not cluster-wide on day one.

• The scheduler image tracks your control plane. hami-scheduler wraps a stock kube-scheduler image, and current charts resolve its tag from the API server’s version at install time, which means a Kubernetes upgrade quietly leaves the old scheduler running until your next Helm upgrade. Add scheduler.kubeScheduler.image.tag to your Kubernetes upgrade runbook, or it will bite you six months in.

• gpucores throttles; it does not reserve. The scheduler books your share at placement time, but at runtime that share is a ceiling, never a floor. The limiter is a token bucket checked at kernel launch, not a hardware partition: expect oscillation around the target, bursting above the share while the card is idle, and a few seconds of overlap before limits re-arm when a neighbor arrives mid-burst (set GPU_CORE_UTILIZATION_POLICY=force to pin the limiter on at all times). Benchmark p99 under a noisy neighbor before you commit to an SLO.

A practical recommendation

If your GPU fleet is A100/H100-class, shared by multiple teams, and your utilization dashboards would embarrass you in the finance review, pilot HAMi on one inference node pool this week. The install fits in an afternoon: label two nodes gpu=on, helm install, convert one over-provisioned inference Deployment to gpumem/gpucores slices, and let it run for a week while you watch allocation against DCGM utilization. The arithmetic you’re testing is blunt: an 8-GPU H100 node runs roughly $290,000 to $480,000 a year on-demand depending on cloud and region, so every node that bin-packing drains is real money back. On owned hardware, at roughly $25–30k per H100, every ten points of sustained utilization on a thousand-GPU fleet is $2.5–3M of capex doing work instead of nothing.