LLMOps & AI Platform Engineering · Module 08 of 16

GPUs, Memory & Capacity

GPU memory is shared by more than model weights. A serving process also needs runtime memory and KV cache for active contexts. Longer contexts and more concurrent requests can consume capacity even when the model loads successfully. Quantization can reduce memory footprint, but it introduces compatibility and quality/performance trade-offs that must be tested.

Engineering focus: GPU memory, model weights, KV cache, quantization and capacity planning

In plain English

What problem are we solving?

Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.

Animated mental model

Watch GPU memory fill as the service moves from model load to concurrent inference

Model weights consume a base amount; active request state and runtime headroom determine whether the service can actually handle traffic.

Speed

● Loading interaction engine…

● PAUSEDStart with a finite budget

01GPU VRAMfinite budgetNOW

02Weightsbase allocationNOW

03RuntimeworkspaceNOW

04KV cacheactive contextNOW

05Concurrencymore sequencesNOW

06Headroomsafety marginNOW

STEP 1/6Start with a finite budget

A GPU has finite device memory. Capacity planning is a budget problem, not a yes/no model-load test.

VRAM=24 GB

Learning objectives

By the end of this module, you should be able to:

Explain the major GPU-memory consumers in a serving process.
Reason about why context length and concurrency affect capacity.
Describe quantization as a memory/performance/quality trade-off rather than a free compression switch.
Identify when one GPU, tensor/distributed parallelism or a smaller model is the more appropriate response.

🎓 Learning Coach follows this block when it is centered in your browser

GPU memory is a budget with fixed and variable consumers

The model weights are the most visible memory consumer, but they are not the whole budget. Runtime allocations, kernels/graphs, temporary buffers and request state also need memory.

Serving therefore needs headroom. If weights consume almost all VRAM, the server may load but fail when a long prompt or concurrent requests require additional state.

Fixed-ish: model weights and some runtime allocations.
Variable: KV cache/request state grows with active sequences and context.
Temporary workspace and execution strategy also consume memory.
Capacity planning requires headroom, not only successful startup.

Common beginner misconception

'The weights fit in 12 GB' is not equivalent to '12 GB is enough for production serving.'

🎓 Learning Coach follows this block when it is centered in your browser

KV cache trades memory for avoiding repeated attention work during generation

During autoregressive generation, the runtime keeps attention key/value state from prior tokens so it does not recompute all previous attention state from scratch for each new token. This state is commonly called KV cache.

The teaching implication is simple: active sequence length and the number of active sequences matter for memory. vLLM explicitly manages KV cache and exposes configuration for cache memory; current documentation also describes prefix caching and quantized KV cache options.

Longer contexts generally require more active state.
More concurrent sequences generally require more cache capacity.
Cache management affects achievable concurrency and throughput.
Exact memory per token is architecture/dtype dependent.

Common beginner misconception

KV cache is not the same thing as a prompt-response application cache. It is internal model execution state.

🎓 Learning Coach follows this block when it is centered in your browser

Quantization reduces representation precision to save memory, with trade-offs

Quantization stores/executes some model values at lower numerical precision. vLLM currently supports multiple quantization formats, and its documentation describes the core trade-off as reducing memory footprint so models can run on a wider range of hardware.

There is no universal 'best 4-bit switch'. Hardware support, model format, kernel availability, quality tolerance and throughput goals all matter. The result should be evaluated on your tasks and hardware.

Benefit: lower weight memory can create more headroom or enable a larger model.
Possible cost: quality changes, kernel compatibility limits or different performance.
Quantized weights and quantized KV cache are distinct choices.
Benchmark and evaluate after changing precision.

Common beginner misconception

Smaller memory footprint does not guarantee faster inference on every hardware/runtime combination.

🎓 Learning Coach follows this block when it is centered in your browser

When one GPU is not enough, change the model, precision, parallelism or service shape deliberately

If a model cannot meet memory or throughput requirements on one device, there are several responses: use a smaller model, quantize, reduce context/concurrency limits, use a larger-memory device, or distribute work across devices.

vLLM supports multiple distributed parallel strategies. These can increase capacity, but cross-device communication and operational complexity become part of the performance model.

Smaller model: often the simplest capacity reduction if quality is acceptable.
Quantization: lower memory representation with validation required.
Tensor/distributed parallelism: split model/work across devices with communication overhead.
Admission limits: protect a known capacity envelope instead of allowing OOM.

Common beginner misconception

More GPUs do not guarantee linear speedup; communication and workload shape matter.

Interactive simulator

VRAM simulator: model weights are only the beginning

Change model size, precision, context and concurrency. The memory formula is deliberately simplified; it teaches headroom and pressure, not exact hardware sizing.

TEACHING MODEL

● Loading simulator engine…

GPU VRAMweights14 GBcontextK8k tokconcurrency8 Apply synthetic quantization (weights ×0.55)

Weights Runtime Synthetic KV/request Headroom

Capacity state—

This simulator teaches relationships and diagnostic reasoning. Its synthetic values are not vendor benchmarks, hardware-sizing evidence or production SLO measurements.

Worked examples

A simple VRAM budget

GPU VRAM             24 GB
model weights        14 GB
runtime/workspace     2 GB
KV/request budget     6 GB
headroom              2 GB

The model 'fits' at 14 GB, but production capacity depends on the remaining 10 GB.

Capacity is a budget, not a binary model-load check.
The exact partition is runtime/workload specific; this is a mental model.

Capacity response options

symptom: OOM at long-context concurrency

A reduce max context / concurrency
B choose smaller or quantized model
C increase device memory
D distribute across devices

Choose from SLO + quality + cost evidence

There are multiple levers; the right one depends on the actual constraint.
A retry is not a capacity solution for deterministic OOM pressure.

Try it · core lab · account-free

Explore a synthetic GPU memory budget and trigger controlled OOM pressure

Goal: Reason about weights, runtime headroom, context and concurrency without requiring a GPU.

Before you start

Python 3 only. The cache formula is intentionally simplified and must not be used for hardware procurement.

STEP 1

Run the safe scenario

python3 public/labs/llmops/08_gpu_memory_capacity.py --scenario safe

STEP 2

Increase context

python3 public/labs/llmops/08_gpu_memory_capacity.py --scenario long-context

STEP 3

Increase concurrency

python3 public/labs/llmops/08_gpu_memory_capacity.py --scenario high-concurrency

STEP 4

Apply a mitigation

python3 public/labs/llmops/08_gpu_memory_capacity.py --scenario quantized

Download this Python lab Download all current labs

Verify

Safe scenario retains explicit headroom.
Long-context/high-concurrency scenarios show synthetic capacity pressure.
Quantized scenario reduces the simulated weight allocation but still preserves a quality-validation warning.

Expected outcome

You can explain why successful model load is not a production capacity proof.
You can list several capacity levers and their trade-offs.

Break / fix · diagnose

If you try to translate the synthetic GB directly to a real model, read the warning again: real KV-cache math is architecture/dtype specific.
If your mitigation only retries the OOM, choose a capacity lever instead.

Production view

Use real server profiling, representative context/concurrency and quality evaluation to set memory headroom and admission limits.

Production Engineer checklist

Before you call this concept production-ready:

Memory budget includes weights, runtime/workspace, active request state and headroom. Peak context and concurrency are capacity-tested. OOM is handled by admission/capacity controls rather than blind retry. Quantization changes trigger both quality evaluation and performance tests. Parallelism is justified by a measured constraint. Memory/cache pressure is observable before user-facing failure.

This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.

Avoid these traps

Common mistakes

Sizing GPU only from model-weight file size.
Treating KV cache as an application response cache.
Assuming quantization is always faster and quality-neutral.
Adding GPUs before verifying whether a simpler model/precision/context policy meets the requirement.

Knowledge check

Reason about the system, not just the vocabulary.

Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.

1Why can a model that loads successfully still OOM under traffic?

Active request state/KV cache and runtime workspace consume additional memory as context and concurrency grow.

2What is the main conceptual purpose of KV cache during generation?

Retain prior attention key/value state so the runtime can generate subsequent tokens without recomputing all earlier attention state from scratch.

3What must be revalidated after quantization?

Task quality and hardware-specific serving performance/capacity.

4Why is adding GPUs not automatically a linear performance improvement?

Distributed parallelism adds communication/synchronization and depends on model, topology and workload shape.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.

🎓 Learning Coachfloating · safe rail · follows the visible learning block

● Coach interaction engine loading…

08. GPUs, Memory & CapacityCURRENT VIEWPORT FOCUSWhat problem are we solving?

Need a different angle?

Speed

Instructor voice ready

The coach follows the learning block nearest the center of the viewport. Scrolling changes context silently; it never interrupts narration already in progress.