DynamiaDynamia
Case studies

NIO Improves GPU Utilization for Autonomous Driving Workloads with HAMi

DynamiaCase studies
NIO Hami GPU shared caseCustomer practice and delivery stories
Dynamia · Open-source AI infrastructure

NIO adopted a hybrid GPU sharing strategy combining HAMi, NVIDIA MIG, and time-slicing to significantly improve GPU utilization across CI, simulation, and inference workloads — achieving 4× utilization gains and 30% reduction in GPU hours.

On this page

NIO operates large-scale cloud infrastructure to support autonomous driving workloads, including model training, simulation, CI/testing, and online inference. The team focuses on GPU performance optimization and participates in GPU and compute resource planning decisions.

Challenge

NIO's large-scale cloud infrastructure supports diverse autonomous driving workloads, including training, simulation, CI/testing, and online inference. The team is responsible for GPU performance optimization and resource planning. This diversity led to persistent efficiency challenges due to workload–resource mismatch.

Workloads with low GPU utilization

CI and Testing Tasks

Most of the execution time for CI tasks is spent on CPU-intensive operations such as compilation, file fetching, and preprocessing. GPUs are only used intermittently. Under full-GPU allocation, average GPU utilization was typically 5–10%.

Simulation workloads

Simulation pipelines process video streams, radar data, and inference validation. Individual tasks have relatively low compute requirements and are not latency-sensitive, making them suitable for concurrent execution on shared GPUs.

Online inference and small model inference

Many inference services require only ¼ or ½ of a GPU. Allocating an entire GPU is both inefficient and costly.

  • The GPU cluster size is limited, and part of the capacity is sourced from public cloud providers with time-based billing.
  • Low GPU utilization results in wasted GPU hours, directly increasing operational costs.
  • Inefficient allocation also prolongs queue times for high-demand training jobs.

NIO needed a way to improve GPU utilization without sacrificing workload stability, while supporting multiple workload types within the same Kubernetes environment.

Solution

NIO adopted a hybrid GPU sharing strategy, selecting different GPU allocation mechanisms based on workload characteristics rather than enforcing a single approach.

The following diagram illustrates the GPU management architecture after integrating HAMi into Kubernetes.

Resource management chart
Figure 1: Resource management chart

HAMi extends Kubernetes with fine-grained GPU sharing capabilities through scheduler extensions (Layer 2) and device plugin integration (Layer 4), enabling flexible GPU resource allocation while coexisting with NVIDIA MIG and time-slicing strategies.

Evaluated approaches

NVIDIA MIG

Provides strong isolation but supports only predefined partition sizes, making it difficult to match finer-grained requirements (e.g., 1/6 or 1/8 of a GPU).

Time-Slicing

Allows workloads to compete freely for GPU resources with minimal overhead. However, it lacks strict limits on memory and compute usage, making it unsuitable for certain production workloads.

HAMi (a CNCF Sandbox project)

Supports fine-grained control over both GPU memory and compute allocation, enabling proportional allocation based on actual workload requirements. This approach introduces some overhead compared to full GPU allocation.

Production strategy

Rather than replacing existing mechanisms, NIO combined them:

  • MIG: Used for algorithm development and environments requiring strong isolation
  • HAMi: Used for CI tasks and selected inference and simulation workloads
  • Time-slicing: Used for workloads that can tolerate resource contention

This hybrid approach improved GPU utilization while maintaining correctness and operational safety.

Implementation

  • HAMi was deployed on selected GPU clusters, not across all clusters.
  • Large-scale training jobs continue to use full GPUs by default.
  • HAMi primarily covers workloads suitable for fractional GPU allocation.

Deployment scale:

  • Approximately 50–70 active nodes
  • Approximately 400–560 GPUs (assuming 8 GPUs per node)

Resource allocation design

For simulation workloads, NIO treated GPU memory (VRAM) as the primary constraint:

  • Insufficient compute capacity can be mitigated through scheduling delay.
  • Memory exhaustion immediately triggers failure (OOM) and cannot be deferred.

The team also found that finer partitioning is not always better. For certain simulation workloads, allocating approximately 1/6 of a GPU provided optimal efficiency. Smaller fractions (such as 1/8) introduced additional scheduling and virtualization overhead, reducing overall throughput.

Impact

CI Workloads: From Waste to Reuse

Before HAMi:

Because CI tasks were primarily CPU-bound, average GPU utilization was approximately 5% (5–10%).

After HAMi:

By partitioning GPUs into ¼ or smaller fractions, effective GPU utilization in CI pipelines increased by approximately 4×, reaching 30–50%.

Simulation workloads: Higher throughput, fewer GPU hours

  • Fine-grained GPU sharing reduced overall GPU hours by approximately 30%.
  • End-to-end simulation tasks that previously required about 3 days were reduced to about 2 days.

These improvements increased overall system throughput while directly reducing GPU costs.

Lessons learned

GPU partitioning is not "the finer, the better."

Each workload has an optimal partition size. Over-fragmentation can reduce efficiency.

Version upgrades require performance validation.

New GPU virtualization components may introduce performance regressions. NIO adopted a phased upgrade strategy, validating performance benchmarks before allowing different HAMi components to run at different versions.

Operational changes must ensure production safety.

For online inference workloads, device plugin upgrades follow a blue–green deployment–like process: traffic is migrated first, new Pods are deployed, and old instances are gradually decommissioned.

Toolchain compatibility is critical.

Certain compiler or library features (such as pointer or address analysis) may conflict with GPU interception mechanisms. Careful trade-offs between functionality and stability are required.

NIO's deployment of HAMi demonstrates how fine-grained GPU sharing can significantly improve infrastructure efficiency for autonomous driving workloads. By combining HAMi with Kubernetes and existing GPU allocation mechanisms such as MIG and time-slicing, NIO increased GPU utilization, reduced overall GPU hours, and improved workload throughput without compromising stability.

This hybrid resource management strategy enables NIO to support diverse AI workloads, from CI pipelines to simulation and inference — more efficiently within the same Kubernetes environment.

KEEP READING

Keep reading

All articles
DynamiaCommunity & updates
HAMi positioning in CNCF x PyTorch baseBuilding together · discussions · event notes
Dynamia · Open-source AI infrastructure

HAMi Meetup Shanghai Deep Dive (1): Where Does HAMi Fit in the CNCF × PyTorch Cloud Native AI Foundation?

The model has been able to run and the operation is slow to go online. Keith Chan, Director General of the China District, gave an opening speech at the HAMMI Meetup Shanghai Station, giving a structured collaborative idea of how to create a base for the development of claud-native AI: Hami is located in Scheduling with resource management, assuming the role of heterogenouscompute resources sharing and managing.

Read article
DynamiaCommunity & updates
HAMi Community Round Table: From Users to Co-ConstructorsBuilding together · discussions · event notes
Dynamia · Open-source AI infrastructure

HAMi Meetup Shanghai Deep Dive (7): From User to Co-Builder — Why Enterprises Need a Sustainable HAMi Community

Once an enterprise uses open source software in its production environment, the concerns will gradually extend beyond code: can it be responded to when problems are encountered? How do new needs enter public routes? The Hami Meetup Shanghai Station round-table discussion revolves around real use, contribution thresholds, AI ' s liability boundary for assistive development and community knowledge deposition, providing a different set of judgement than technology selection.

Read article
DynamiaCommunity & updates
HAMi 2.10: Soolation, Scheduling and ObservationBuilding together · discussions · event notes
Dynamia · Open-source AI infrastructure

HAMi Meetup Shanghai Deep Dive (2): After GPU Sharing — Isolation, Scheduling, and Observability in HAMi 2.10

The distribution of a GPU to multiple tasks is only the beginning of resource sharing. Dynamia co-founder and CTO, HAMi Mainlainer Mengxuan Li, at the HAMi Meetup Shanghai Station, presented the progress of V2.10 capabilities, DRA ecological appliance, and v2.11 planning (Remote GPU, CPU NUMA alignment). The main thread for these changes is to enable shared compute resources to be distributed on demand and to be subject to monitoring, observation and diagnosis during their operation.

Read article
Image viewer