LLMOps & AI Platform Engineering · Module 07 of 16
Tokens, Latency & Throughput
AI performance is not one number. Prompt tokens affect preparation work, generated tokens affect decode duration, concurrency creates queueing, TTFT tells you how long users wait for the first output, and throughput tells you how much work the service completes over time. These metrics must be read together.
Engineering focus: Token workload shape, queueing, TTFT, decode rate and throughput
In plain English
What problem are we solving?
AI performance is not one number. Prompt tokens affect preparation work, generated tokens affect decode duration, concurrency creates queueing, TTFT tells you how long users wait for the first output, and throughput tells you how much work the service completes over time. These metrics must be read together.
Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.
Animated mental model
See where time accumulates before and after the first token
Separate queue, prefill and decode phases. A change in one phase can hurt users even when the others remain healthy.
● Loading interaction engine…
● PAUSEDTraffic arrives
01Arrivalrequest startsNOW
02QueuewaitingNOW
03Prefillinput tokensNOW
04First tokenTTFTNOW
05Decodeoutput tokensNOW
06Completetotal latencyNOW
STEP 1/6Traffic arrives
The request timer starts. Offered load and concurrency determine how much work competes for service.
request_rate=20/sLearning objectives
By the end of this module, you should be able to:
- Define input/output tokens, TTFT, inter-token/decode rate, total latency and throughput in plain language.
- Diagnose queue saturation separately from slow token generation.
- Explain why concurrency can improve throughput until it creates harmful queueing or memory pressure.
- Design a representative performance test instead of benchmarking one prompt.
🎓 Learning Coach follows this block when it is centered in your browser
Tokens are units of model work, but not all token work happens in the same phase
Language-model APIs operate on tokens rather than characters. A long prompt creates more input work before generation; a long requested response creates more decode work after generation begins.
The exact cost per token is model/runtime dependent, but separating input and output lengths helps explain why two requests with the same HTTP payload size can have very different latency and memory behavior.
- Input tokens: prompt/system/context sent to the model.
- Output tokens: generated response tokens.
- Prefill processes the input context before normal token-by-token decode.
- Decode repeatedly generates subsequent output tokens.
Common beginner misconception
A 'request' is not a uniform unit of work. Ten tiny prompts and ten 32k-context prompts are not equivalent load.
🎓 Learning Coach follows this block when it is centered in your browser
TTFT and decode speed answer different questions
TTFT measures the delay from request issuance/acceptance until the first streamed output becomes visible. It includes waiting and preparation before normal generation. Inter-token latency or output token rate describes the generation phase after output starts.
Total latency combines all phases. When only total latency is available, operators cannot easily tell whether the service is queueing, prefill is slow, or decode is slow.
- Queue/admission delay contributes to TTFT.
- Prompt processing contributes to TTFT.
- Decode speed affects how quickly the remainder of the answer arrives.
- Total latency can worsen for different reasons that require different fixes.
Common beginner misconception
Streaming can make the system feel more responsive without reducing total compute or total completion latency.
🎓 Learning Coach follows this block when it is centered in your browser
Throughput rises with useful concurrency until the system saturates
A GPU often needs enough concurrent work to stay efficiently utilized. Serving runtimes therefore schedule multiple active requests. vLLM documents continuous batching as a core capability for this purpose.
More concurrency is not free. At some point queue depth, memory pressure or contention increases faster than useful throughput. The practical goal is the operating region that satisfies latency objectives while using capacity efficiently.
- Low concurrency can underutilize expensive hardware.
- Moderate concurrency can improve aggregate throughput.
- Excess concurrency grows queues and tail latency.
- Admission control protects the service from accepting more work than it can satisfy.
Common beginner misconception
100% GPU utilization is not automatically success; if users wait 30 seconds in the queue, the service may be failing its objective while the GPU is busy.
🎓 Learning Coach follows this block when it is centered in your browser
A useful benchmark resembles production traffic and reports distributions
Performance tests should represent real model, hardware, server configuration and request shape. Include prompt-length and output-length distributions, concurrency/request rate, streaming mode and warm/cold behavior relevant to your use case.
Report percentiles and error/rejection rates together with throughput. A test that reaches a huge requests-per-second number by allowing unacceptable p99 latency is not a successful capacity result.
- Pin model/runtime/config versions.
- Use representative input/output token distributions.
- Measure warm-up separately from steady state where relevant.
- Record errors, queueing and tail latency as load increases.
Common beginner misconception
Do not compare published throughput numbers from different models/hardware/request shapes as if they were directly interchangeable.
Interactive simulator
Performance simulator: change traffic and watch queue, TTFT and throughput
Adjust concurrency, prompt size and output length. The equations are intentionally synthetic and teach relationships, not hardware benchmark values.
● Loading simulator engine…
Service pressure
Healthy
This simulator teaches relationships and diagnostic reasoning. Its synthetic values are not vendor benchmarks, hardware-sizing evidence or production SLO measurements.
Worked examples
Same total latency, different diagnosis
A: queue=2.5s prefill=0.4s decode=1.1s total=4.0s
B: queue=0.1s prefill=0.4s decode=3.5s total=4.0s- The same total latency can represent different bottlenecks.
- A needs capacity/admission investigation; B points more strongly toward generation/runtime/model behavior.
Find the knee of the curve
load throughput p95_TTFT
5 5.0/s 0.7s
15 14.8/s 0.9s
25 23.0/s 1.6s
35 24.1/s 5.8s
45 24.3/s 12.4s- After about 25–35 units of load, throughput barely grows while TTFT explodes.
- Maximum accepted load is not the same as a healthy operating point.
Try it · core lab · account-free
Diagnose synthetic latency traces and find a safe operating point
Goal: Use phase metrics and load steps to distinguish queue saturation from decode slowdown.
Before you start
- Python 3 only. All performance values are deterministic teaching data.
Diagnose a queue incident
python3 public/labs/llmops/07_latency_throughput_lab.py --scenario queueDiagnose a decode incident
python3 public/labs/llmops/07_latency_throughput_lab.py --scenario decodeRun the load sweep
python3 public/labs/llmops/07_latency_throughput_lab.py --scenario sweepChoose the SLO-safe point
Find the highest synthetic load where p95 TTFT remains below the lab's 2-second objective.
Verify
- Queue scenario reports queue_pressure.
- Decode scenario reports decode_slowdown.
- The load sweep identifies a point where throughput flattens but p95 TTFT rises sharply.
Expected outcome
- You can diagnose performance from phase evidence rather than total latency alone.
- You can explain why capacity targets should include an SLO, not only maximum throughput.
Break / fix · diagnose
- If you choose the highest throughput row, compare its p95 TTFT against the stated objective.
- If both incidents look the same, inspect queue_ms and decode_ms separately.
Production view
Production capacity tests should measure representative token lengths, queueing, TTFT, decode rate, errors and tail latency together.
Production Engineer checklist
Before you call this concept production-ready:
This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.
Avoid these traps
Common mistakes
- Using requests/second without token/request-size context.
- Treating TTFT and decode speed as the same metric.
- Celebrating maximum GPU utilization while user tail latency is unacceptable.
- Benchmarking nonrepresentative short prompts and assuming the result sizes production.
Knowledge check
Reason about the system, not just the vocabulary.
Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.
1TTFT grows rapidly with load while decode rate stays stable. What is the strongest first hypothesis?
Queue/admission or prefill pressure before decode, often a saturation signal.
2Why can requests/second be misleading for LLM capacity?
Requests have very different input/output token sizes, so one request is not a consistent unit of work.
3What is the 'knee' in a throughput/latency curve?
The region where extra offered load yields little throughput gain but causes disproportionate queueing/tail-latency growth.
4Why should benchmarks report p95/p99 and errors, not only average latency?
Tail behavior and rejection/failure rates determine whether real users meet the service objective under load.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.