Gnu Group Learning AcademyGnu Group ↗
LLMOps & AI Platform Engineering · Module 05 of 16

From Ollama to vLLM

Ollama is excellent for learning and local workflows. vLLM introduces production-oriented serving ideas such as OpenAI-compatible APIs, continuous batching, prefix caching, quantization options and distributed inference. The important lesson is why those capabilities matter, not that one product always replaces the other.

Engineering focus: Serving runtimes, scheduling and production inference trade-offs

In plain English

What problem are we solving?

Ollama is excellent for learning and local workflows. vLLM introduces production-oriented serving ideas such as OpenAI-compatible APIs, continuous batching, prefix caching, quantization options and distributed inference. The important lesson is why those capabilities matter, not that one product always replaces the other.

Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.

Animated mental model

Watch serving evolve from one local request to a shared inference service

Follow the operational pressure: more clients create queueing, scheduling and capacity problems that a production serving runtime must manage.

● Loading interaction engine…
● PAUSEDOne request
01Clientone request
02Local serversimple pathNOW
03Request queueconcurrency
04Schedulerbatch-aware
05InferenceGPU work
06Capacity evidenceTTFT + throughput
STEP 1/5One request

A single client can use a simple local serving path with very little scheduling pressure.

concurrency=1
Learning objectives

By the end of this module, you should be able to:

  • Explain the difference between a local model service and a throughput-oriented inference server.
  • Describe why continuous batching and prefix caching can improve serving efficiency under suitable workloads.
  • Use an OpenAI-compatible API boundary as a portability mechanism without assuming every implementation behaves identically.
  • Choose a serving approach from workload evidence rather than from product popularity.
🎓 Learning Coach follows this block when it is centered in your browser

One user and many users create different serving problems

A local model service is often optimized for convenience: install it, download a model and call it from one machine. That is an excellent learning environment because the operational surface is small.

A shared production service has a different problem. Requests arrive concurrently, context lengths vary, users expect predictable latency, expensive accelerator memory must be shared, and operators need capacity evidence. The serving runtime therefore becomes a scheduler as well as a model runner.

  • Local success proves the model can run; it does not prove shared-service capacity.
  • Concurrency introduces queueing and fairness questions.
  • Long prompts and long outputs consume different amounts of time and memory.
  • Production selection should start from workload shape and service objectives.
Common beginner misconception

Production does not automatically mean 'replace Ollama with vLLM'. A small internal service may be perfectly adequate with simpler tooling.

🎓 Learning Coach follows this block when it is centered in your browser

vLLM is an inference engine plus a serving and scheduling layer

vLLM exposes HTTP serving interfaces, including OpenAI-compatible APIs, while its engine focuses on efficient request scheduling and model execution. Current vLLM documentation highlights continuous batching, prefix caching, quantization options and multiple forms of distributed parallelism.

These features should be understood as mechanisms for using accelerator memory and compute more efficiently. They do not remove the need for an external gateway, authentication policy, observability, evaluation or application-level reliability.

  • HTTP/API server accepts and validates requests.
  • Scheduler decides which work can progress together.
  • Engine executes model work on available devices.
  • Caches can reuse eligible work; quantization can reduce memory footprint with trade-offs.
Common beginner misconception

An OpenAI-compatible endpoint does not mean operational behavior, limits, defaults and security are identical to OpenAI or to another compatible server.

🎓 Learning Coach follows this block when it is centered in your browser

Compatibility is a contract surface, not identical behavior

An OpenAI-compatible API can reduce application coupling because clients can use a familiar request/response shape against multiple backends. vLLM currently supports several OpenAI-style endpoints, including chat/completions and responses interfaces.

Compatibility should be tested, not assumed. Models differ in chat templates and tool behavior; servers may ignore or extend fields; streaming, errors and authentication boundaries can differ.

  • Keep the application dependent on a documented contract, not a server's internal classes.
  • Test the exact endpoints and parameters your application uses.
  • Version model identifiers and server configuration alongside application releases.
  • Use a gateway when you need uniform policy across heterogeneous backends.
Common beginner misconception

Do not treat 'OpenAI-compatible' as a certification that every OpenAI field or behavior is implemented the same way.

🎓 Learning Coach follows this block when it is centered in your browser

Choose from evidence: simplicity, throughput, portability and control

Serving architecture is a trade-off. Simpler local services minimize setup and cognitive load. Throughput-oriented servers add scheduling and accelerator controls. Managed providers remove much of the infrastructure burden but trade some control for convenience.

A good decision records the workload assumptions: request rate, concurrency, model size, context distribution, latency targets, data sensitivity, hardware, cost and operational skill.

  • Development/local learning: optimize for simplicity and inspectability.
  • Shared GPU service: optimize for predictable capacity and scheduling.
  • Managed provider: optimize for low infrastructure burden and rapid access to models.
  • Hybrid: route different workloads to different backends through a stable policy boundary.
Common beginner misconception

Do not benchmark one prompt and generalize it to all production traffic.

Interactive simulator

Serving workbench: when does the simple path stop being enough?

Change concurrency and workload shape. The numbers are a teaching model, not a benchmark; use them to reason about queueing and scheduling trade-offs.

TEACHING MODEL
● Loading simulator engine…
Simple serving pathLow scheduling pressure
Synthetic queue
0
Synthetic TTFT
0.7 s
Relative throughput
1.0×
Batch-aware serving pathScheduler ready
Synthetic queue
0
Synthetic TTFT
0.7 s
Relative throughput
1.0×

Raise concurrency to see when scheduling becomes visible.

This simulator teaches relationships and diagnostic reasoning. Its synthetic values are not vendor benchmarks, hardware-sizing evidence or production SLO measurements.

Worked examples

Same client contract, different serving backend

POST /v1/chat/completions
{
  "model": "support-model",
  "messages": [{"role":"user","content":"Summarize this ticket"}]
}

Client contract stays stable
Backend can be local, vLLM or provider through a gateway
  • Compatibility can isolate applications from backend changes.
  • Contract tests are still needed because supported fields and behavior vary.

Serving decision record

workload: internal summarization
concurrency_p95: 18
context_p95: 6k tokens
latency_target_ttft: <2 s
data: internal-only
operator_skill: Linux + containers
choice: benchmark local vLLM; keep gateway contract stable
  • The decision starts from workload evidence.
  • The backend choice remains revisitable instead of becoming architectural dogma.
Try it · core lab · account-free

Compare simple and scheduled serving with a deterministic queue model

Goal: See how concurrency turns model serving into a scheduling and queueing problem without requiring a GPU or model download.

Before you start

  • Python 3 only. This is a teaching simulation, not a real Ollama or vLLM benchmark.
STEP 1

Run the baseline

python3 public/labs/llmops/05_serving_runtime_comparison.py --scenario quiet
STEP 2

Increase concurrent demand

python3 public/labs/llmops/05_serving_runtime_comparison.py --scenario busy
STEP 3

Compare scheduler behavior

Observe queue depth, admitted work and synthetic TTFT under the simple and batch-aware policies.

STEP 4

Break it with overload

python3 public/labs/llmops/05_serving_runtime_comparison.py --scenario overload
Download this Python labDownload all current labs

Verify

  • Quiet traffic completes without a meaningful queue in either policy.
  • Busy traffic exposes queueing pressure in the simple policy.
  • The batch-aware policy improves synthetic throughput in the teaching model but does not claim a real benchmark result.

Expected outcome

  • You can explain why serving under concurrency requires scheduling.
  • You can distinguish an architectural teaching model from hardware benchmark evidence.

Break / fix · diagnose

  • If you are tempted to compare the printed numbers to real vLLM performance, stop: the script is intentionally synthetic.
  • If one policy looks universally better, inspect the scenario assumptions; production selection requires real workload measurements.
Production view

Use workload-specific benchmarks and production telemetry to select and tune a real serving runtime; use this lab only to learn the scheduling mental model.

Production Engineer checklist

Before you call this concept production-ready:

This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.

Avoid these traps

Common mistakes

  • Treating vLLM as automatically required for every production deployment.
  • Treating an OpenAI-compatible API as identical behavior across servers.
  • Benchmarking one short prompt and extrapolating to long-context/concurrent traffic.
  • Putting an inference server directly on an untrusted network and assuming its built-in API key secures every endpoint.
Knowledge check

Reason about the system, not just the vocabulary.

Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.

1Your local Ollama setup works for one user but TTFT rises sharply when 25 users arrive together. What new problem has appeared?

Shared serving is now a scheduling/capacity problem: concurrent requests are queueing for the same inference resources.

2Why can OpenAI-compatible APIs improve architecture even if servers are not behaviorally identical?

They provide a common client contract that reduces coupling, while contract tests handle backend-specific differences.

3What does continuous batching try to improve?

How efficiently concurrent active requests share inference compute by continuously scheduling compatible token work.

4What evidence should justify moving from a simple local service to a more complex inference server?

Measured workload needs such as concurrency, queueing/latency, model size, hardware utilization, cost or required serving features.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.