A model-serving system turns structured requests into model execution and returns generated output. The API contract tells clients how to ask for work; streaming lets output arrive incrementally instead of waiting for the entire response.
Engineering focus: Inference endpoints, streaming, timing and client contracts
A model-serving system turns structured requests into model execution and returns generated output. The API contract tells clients how to ask for work; streaming lets output arrive incrementally instead of waiting for the entire response.
Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.
The first visible token and the final token measure different parts of the user's experience.
The timer starts when the service receives the request.
0.00 sThe inference server sits between clients and the model runtime. It accepts requests, validates parameters, prepares inputs, schedules work, invokes the model and formats results back into the API contract.
This layer is where production systems can add batching, concurrency controls, request cancellation, metrics and hardware-aware scheduling.
The model file itself is not an HTTP server. A serving/runtime layer exposes it to applications.
With a non-streaming response, the client waits until generation is complete and receives the finished payload. With streaming, the server sends incremental pieces as they become available, allowing the user to start reading earlier.
Streaming improves perceived responsiveness but introduces client complexity: partial output, disconnects, cancellation and end-of-stream handling all matter.
The important thing is to understand the boundary and evidence before memorizing implementation details.
Time to first token (TTFT) measures how long the user waits before generation becomes visible. Total latency measures until the response finishes. Token generation rate describes how quickly output proceeds after generation begins.
A service can have slow TTFT but normal generation speed when the queue is overloaded. That is why one latency number is often not enough.
High total latency does not prove the model itself is generating slowly.
An API is not complete when only the happy-path JSON is documented. Clients need to know how long to wait, how failures are represented, whether requests can be cancelled, how authentication works and which retry behavior is safe.
Retries require care because some operations are expensive and user cancellations should not become invisible orphaned generation.
The important thing is to understand the boundary and evidence before memorizing implementation details.
NON-STREAMING
request ───────────────► complete JSON
user waits
STREAMING
request ──► chunk ─► chunk ─► chunk ─► done
user can start readingqueue/prefill 2.8 s
first token ← TTFT ~2.8 s
generation 1.5 s
-----------------------
total 4.3 sGoal: See request validation, queue delay, first output and streamed completion using only Python's standard library.
python3 public/labs/llmops/04_inference_api_streaming.py --self-testpython3 public/labs/llmops/04_inference_api_streaming.py --servecurl -N -s http://127.0.0.1:18081/v1/stream -H 'Content-Type: application/json' -d '{"prompt":"explain batching","queue_ms":700}'Repeat with queue_ms=2500. Observe that the first chunk arrives later even though the chunk-to-chunk interval is unchanged.
Later, the same metrics help distinguish queue saturation, slow prefill and slow token generation on real inference servers such as vLLM.
This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.
Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.
Accepting/validating requests, preparing and scheduling inference work, running the model through its runtime and returning structured output.
Perceived responsiveness because output can be consumed before the entire response is finished.
Delay before generation—often queueing/capacity or prefill—not necessarily slower token decoding.
Generation is expensive; when a client no longer needs the response, cancellation can prevent unnecessary compute and reduce orphaned work.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.