LLMOps & AI Platform Engineering · Module 14 of 16

Kubernetes for AI Workloads

Kubernetes does not make a model faster by itself. It gives you a control plane for placing, restarting and scaling containers. AI workloads add scarce accelerators, large images/models, warm-up time and unusual capacity constraints.

Engineering focus: Kubernetes scheduling and operating patterns for AI workloads

In plain English

What problem are we solving?

Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.

Animated mental model

Follow a GPU-serving Pod from request to ready endpoint

The scheduler can only place a workload where resource, label and policy constraints all match.

Speed

● Loading interaction engine…

● PAUSEDDescribe the workload

01Model Podresource requestNOW

02Schedulermatch constraintsNOW

03GPU nodedevice + labelsNOW

04Model startupload / warmNOW

05Ready endpointserve trafficNOW

06Scale signalqueue / TTFT / GPUNOW

STEP 1/6Describe the workload

The Pod declares CPU, memory and any accelerator resource plus placement constraints.

nvidia.com/gpu=1

Learning objectives

By the end of this module, you should be able to:

Explain how Kubernetes exposes GPUs through device/resource mechanisms and schedules Pods against them.
Diagnose Pending workloads caused by GPU capacity, labels, taints or memory constraints.
Separate scheduling, readiness and autoscaling signals for AI serving.
Recognize when Kubernetes adds operational value and when it is unnecessary complexity.

🎓 Learning Coach follows this block when it is centered in your browser

Accelerators are schedulable resources, not magic environment variables

Kubernetes device plugins commonly expose vendor resources such as nvidia.com/gpu. Workloads request the accelerator, and the scheduler places the Pod only on nodes advertising available matching resources.

Modern Kubernetes also has Dynamic Resource Allocation for richer device claims, but beginners should first understand the basic resource-and-placement model.

Vendor drivers/device integration must exist on nodes.
GPU request participates in scheduling.
Resource exhaustion leaves the Pod Pending.
Device model/partitioning capabilities depend on cluster integration.

Common beginner misconception

Installing Kubernetes does not automatically install GPU drivers or device integration.

🎓 Learning Coach follows this block when it is centered in your browser

Labels, taints and affinity express where AI workloads may run

GPU nodes are expensive and often isolated with taints so ordinary workloads do not consume them. AI Pods then add tolerations plus node selectors/affinity for the required hardware class.

Every hard placement rule reduces the eligible node set. Debug unschedulable Pods by checking constraints together, not one at a time.

Node labels describe capabilities.
Taints repel workloads unless tolerated.
Node selector/affinity constrains eligible nodes.
Hard constraints can conflict and produce zero candidates.

Common beginner misconception

Adding more labels/affinity does not make placement smarter if the rules conflict.

🎓 Learning Coach follows this block when it is centered in your browser

Running is not the same as ready to serve model traffic

A container can be Running while it is still downloading/loading model weights, allocating GPU memory or warming caches. Readiness should become true only when useful inference can be served.

Keep liveness, readiness and startup semantics distinct so slow initialization does not trigger destructive restart loops.

Startup: give heavy initialization time.
Readiness: route traffic only when serving is usable.
Liveness: restart truly stuck/unhealthy processes.
Warm-up/model load time affects rollout capacity.

Common beginner misconception

Using the same probe for startup, readiness and liveness can create restart loops.

🎓 Learning Coach follows this block when it is centered in your browser

Scale from the bottleneck signal, not merely CPU percentage

AI serving bottlenecks may be queue depth, concurrency slots, TTFT, accelerator utilization or memory headroom. CPU can stay low while the GPU queue is saturated.

Autoscaling also has delay: a new Pod may need scheduling, image pull, model load and warm-up before adding capacity.

Choose workload-relevant scaling signals.
Account for cold-start/warm-up delay.
Protect minimum warm capacity for latency-sensitive services.
Capacity planning still matters even with autoscaling.

Common beginner misconception

Autoscaling cannot instantly create scarce GPU capacity.

Interactive simulator

GPU scheduling simulator

Change GPU request, node capacity, labels and tolerations to see why a Pod runs or stays Pending.

TEACHING MODEL

● Loading simulator engine…

gpuRequest1gpuFree1Hardware labelNode label GPU-node taint is tolerated

Scheduler decision—

This simulator teaches relationships and diagnostic reasoning. Its synthetic values are not vendor benchmarks, hardware-sizing evidence or production SLO measurements.

Worked examples

Basic GPU Pod request

resources:
  limits:
    nvidia.com/gpu: 1

# Scheduler needs a node exposing available nvidia.com/gpu

GPU is part of placement.
Cluster administrators must install/configure the corresponding device support.

Probe separation

startupProbe: allow model load
readinessProbe: inference endpoint ready
livenessProbe: process/runtime stuck detection

Each probe answers a different operational question.
Heavy model initialization makes probe design especially important.

Try it · core lab · account-free

Reason about GPU Pod scheduling without needing a cluster

Goal: Use a deterministic scheduler model to diagnose GPU capacity, taint and label mismatches before running real Kubernetes.

Before you start

Python 3 only. Optional follow-up can be performed on kind/k3s/real GPU clusters later.

STEP 1

Schedule a healthy workload

python3 public/labs/llmops/14_kubernetes_ai_scheduler.py healthy

STEP 2

Exhaust GPU capacity

python3 public/labs/llmops/14_kubernetes_ai_scheduler.py no-gpu

STEP 3

Miss a toleration

python3 public/labs/llmops/14_kubernetes_ai_scheduler.py taint

STEP 4

Request wrong hardware label

python3 public/labs/llmops/14_kubernetes_ai_scheduler.py label

Download this Python lab Download all current labs

Verify

healthy reports phase=Scheduled.
no-gpu reports reason=Insufficient_gpu.
taint reports reason=Untolerated_taint.
label reports reason=Node_selector_mismatch.

Expected outcome

You can reason from Pod constraints to eligible nodes.
You know that restarts do not fix unschedulable resource requests.

Break / fix · diagnose

If every scenario schedules, inspect hard constraint filtering.
If multiple reasons exist, report the first failed constraint and list all exclusions for operator evidence.

Production view

Production GPU clusters need standardized device integration, hardware labels, taints, readiness semantics and capacity telemetry so teams can understand why workloads are Pending or slow to become ready.

Production Engineer checklist

Before you call this concept production-ready:

GPU/device integration and inventory are verified on nodes. Workloads request the correct resource and hardware class. Taints/tolerations and node affinity are standardized/documented. Startup/readiness/liveness probes represent distinct states. Scaling signals reflect queue/accelerator/concurrency bottlenecks. Cold-start, model-load and surge capacity are measured.

This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.

Avoid these traps

Common mistakes

Assuming Kubernetes automatically provides GPU drivers/resources.
Restarting Pending Pods instead of reading scheduling reasons.
Marking Pods ready before model load/warm-up completes.
Scaling only on CPU for GPU-bound inference.

Knowledge check

Reason about the system, not just the vocabulary.

Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.

1What must happen before a Pod can request nvidia.com/gpu?

The GPU node must be configured with drivers/device integration that exposes the schedulable GPU resource.

2Why can a Pod remain Pending even when one GPU is physically free?

Other hard constraints such as taints, labels/affinity or resource naming may make that node ineligible.

3Why separate startup and readiness probes for large models?

Startup can take a long time; readiness should gate traffic without causing premature restarts during initialization.

4What autoscaling signal may be more useful than CPU for inference?

Queue depth, concurrency saturation, TTFT or another signal tied to the actual serving bottleneck.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.

🎓 Learning Coachfloating · safe rail · follows the visible learning block

● Coach interaction engine loading…

14. Kubernetes for AI WorkloadsCURRENT VIEWPORT FOCUSWhat problem are we solving?

Need a different angle?

Speed

Instructor voice ready

The coach follows the learning block nearest the center of the viewport. Scrolling changes context silently; it never interrupts narration already in progress.