DynamiaDynamia

How SeresCloud scaled AI inference to 10 billion daily tokens with Kubernetes and HAMi

CNCF case study · Localized editionRead the CNCF original ↗
10B+Daily tokens on SeresCloud

Up from approximately 300 million to over 10 billion

≈90%Small-model inference GPU utilization

HAMi plus platform autoscaling, up from 15–25%

3,000%+Growth in platform daily token volume

SeresCloud platform-wide growth

On this page

About Seres and SeresCloud

Seres is a technology-driven automaker focused on new energy vehicles. As AI adoption expanded across its business, the company built SeresCloud to provide unified model services for internal applications and AI agents, with Kubernetes managing the underlying GPU resources.

During 2026, the number of models in service grew from around 10 to more than 42, while daily token volume increased from approximately 300 million to over 10 billion. This platform-wide growth made GPU efficiency, especially for small-model inference, a priority.

Growth in SeresCloud model count and daily token volume
Figure 1: Growth in SeresCloud model count and daily token volume

Growth in SeresCloud model count and daily token volume

Why whole-GPU allocation fell short

The existing environment used GPU-Operator for device management and MIG on supported A100 and H100 GPUs. As hardware and models diversified, whole-GPU allocation was too coarse for embedding, reranking, voice and OCR workloads. Individual inference workloads often needed only 1–4 GB of GPU memory and followed strong day–night traffic patterns. GPU utilization in these small-model scenarios was typically just 15–25% under exclusive allocation.

Time-slicing enabled sharing, but without clear memory and compute boundaries it introduced contention, out-of-memory failures and latency jitter. Seres needed better utilization with more predictable resource limits for multi-tenant inference.

Comparison of GPU resource allocation approaches
Figure 2: Comparison of GPU resource allocation approaches

Comparison of GPU resource allocation approaches

Combining HAMi with platform autoscaling

SeresCloud already provided a model registry, inference gateway, autoscaling and monitoring. The registry managed images, configurations and versions. The gateway handled routing, load balancing, rate limiting and circuit breaking. Autoscaling adjusted service replicas using QPS, GPU utilization and queue depth.

With HAMi, the platform determines when to scale and how many replicas are needed. HAMi places those replicas on suitable GPUs and applies per-task GPU memory and compute limits. Shared monitoring signals connect service elasticity with fine-grained resource allocation.

How the SeresCloud inference platform integrates HAMi
Figure 3: How the SeresCloud inference platform integrates HAMi

How the SeresCloud inference platform integrates HAMi

Sharing one GPU across inference workloads

With HAMi, a Kubernetes Pod can request a vGPU count, GPU memory and a share of compute. The example below requests one vGPU, 3,000 MiB of GPU memory and 30% compute, allowing a small model to use the resources it needs without reserving a whole GPU.

resources:
  limits:
    nvidia.com/gpu: 1
    nvidia.com/gpumem: 3000
    nvidia.com/gpucores: 30

From exclusive GPUs to sharing with resource boundaries
Figure 4: From exclusive GPUs to sharing with resource boundaries

From exclusive GPUs to sharing with resource boundaries

Scheduling and production configuration

Seres uses both Binpack and Spread strategies in production, selecting placement according to workload requirements. Load testing led to a limit of 10 vGPU tasks per physical GPU, with no memory or compute oversubscription. More task slots do not create additional physical resources: scheduling still checks the resources available on each device.

devicePlugin:
  deviceSplitCount: 10
  deviceMemoryScaling: 1
  deviceCoreScaling: 1
  migStrategy: none
  disablecorelimit: "false"

scheduler:
  defaultMem: 5000
  defaultCores: 0

Small-model inference GPU utilization reaches around 90%

Together, HAMi and SeresCloud autoscaling increased GPU utilization in small-model inference scenarios from 15–25% to around 90%. Across the platform, daily token volume grew by more than 3,000% to over 10 billion, with more than 42 models in service. Better resource efficiency supported the platform as its AI workloads expanded.

GPU utilization in small-model inference scenarios
Figure 5: GPU utilization in small-model inference scenarios

GPU utilization in small-model inference scenarios

Lessons from production

During rollout, the team configured webhook bypass rules for privileged debugging Pods and diagnosed scheduling failures using reasons such as NodeInsufficientDevice, CardTypeMismatch and CardInsufficientMemory. Prometheus and Grafana brought HAMi and platform metrics together so troubleshooting, placement and autoscaling could use consistent signals.

The number of device slices must be validated against real models, concurrency and service quality; a larger number is not automatically better. Clear responsibilities between platform autoscaling and HAMi scheduling, supported by shared monitoring, are central to stable operation.

Next steps: broader hardware and scheduling support

Seres plans to bring Ascend and Hygon DCU devices into its resource pools, evaluate HAMi-DRA and explore priority-based preemption and topology-aware scheduling for larger models. The team also plans to continue sharing production experience with the open-source community.

Original:How SeresCloud Scaled AI Inference to 10 Billion Daily Tokens with Kubernetes and HAMi ↗
Publisher: Cloud Native Computing Foundation (CNCF) · 2026-11-02

This localized edition preserves the source article’s technical information and measurement scope. It is not an official CNCF translation. Architecture diagrams retain their original text. Project status and product information reflect the original publication date.

The original is used under the Linux Foundation content terms and CC BY 3.0. This edition has been localized and reformatted. All brand marks belong to their respective owners.

Image viewer