Gnu Group Learning AcademyGnu Group ↗
AI Engineering Fundamentals · Module 02 of 12

Run and connect to an AI model

Learn what happens when an application sends a request to a model, how local serving with Ollama works, and how to choose a model using practical constraints such as memory, speed, context size and task quality.

Engineering focus: Models, inference and local serving

In plain English

What problem are we solving?

Learn what happens when an application sends a request to a model, how local serving with Ollama works, and how to choose a model using practical constraints such as memory, speed, context size and task quality.

The rest of this module introduces the formal terminology and production controls behind that idea. You are not expected to memorize the vocabulary before using it.

Running case · Atlas Assist

Carry the same support assistant forward: approved KB, source citations, explicit insufficient-evidence fallback and one bounded read-only asset lookup. Reuse prior artifacts so each module adds evidence to one production AI system.

Learning objectives

By the end of this module, you should be able to:

  • Describe the inference path from request to generated tokens.
  • Compare model choice using task quality, latency, memory, context and operational constraints.
  • Run a controlled local model preflight and record reproducible model identity.

Model selection is a workload decision

Choose models against the actual task and evaluation set. General benchmark leadership does not guarantee better extraction, retrieval use, tool calling or domain behaviour in your application.

Inference has observable phases

A request loads or reuses model weights, processes the prompt/context, then generates output tokens. Time-to-first-token and generation throughput can fail independently; both matter to user experience.

Quantization trades precision for footprint

Quantized weights can reduce memory and make local inference practical, but quality and speed effects vary by model, hardware and quantization. Treat the quantization tag as part of the deployed artifact identity.

Context windows are limits, not targets

A larger supported context does not mean every request should fill it. Retrieval and state design should minimize irrelevant tokens and reserve space for tool results and response generation.

Local serving changes the trust and cost boundary

Local inference can reduce dependence on external APIs and support offline labs, but it does not eliminate governance. Model files, prompts, logs and retrieved documents still need access control and lifecycle management.

Worked examples

Selection scorecard

candidate = task quality + schema/tool reliability + latency + memory fit + context fit + licence/governance + cost
NOT candidate = benchmark rank alone
  • A model that fits the workload and hardware can outperform a larger model operationally.

Reproducible identity

model name + model digest/version + runtime version + parameters + prompt/template version
  • Record enough identity to explain a regression later.
Try it · core lab · account-free

Lab 02 — Build a model selection scorecard

Goal: Compare model candidates without using a single headline benchmark as the decision.

Before you start

  • No paid API or cloud account is required for the core lab.
  • Core labs use Python 3 and local files only.
  • Running case: you are engineering 'Atlas Assist', an internal support assistant over a small approved knowledge base with one read-only inventory tool.
  • Carry requirements, eval cases, threat assumptions and operational evidence forward across modules instead of starting over.
STEP 1

Create candidate measurements

cat > /tmp/model-candidates.json <<'EOF'
[
 {"name":"small-local","quality":82,"p95_ms":850,"ram_gb":6,"tool_score":90},
 {"name":"large-local","quality":90,"p95_ms":2600,"ram_gb":15,"tool_score":93},
 {"name":"remote-api","quality":94,"p95_ms":1300,"ram_gb":0,"tool_score":97}
]
EOF
STEP 2

Apply a workload constraint

python3 - <<'PY2'
import json
rows=json.load(open('/tmp/model-candidates.json'))
viable=[r for r in rows if r['p95_ms']<=1500 and r['ram_gb']<=12 and r['tool_score']>=90]
print('viable:', [r['name'] for r in viable])
assert {r['name'] for r in viable}=={'small-local','remote-api'}
print('PASS: constraints change the candidate set')
PY2
STEP 3

Record a decision

Choose one candidate for a privacy-sensitive offline environment and one for a globally available SaaS workload. Defend the trade-off rather than naming an absolute winner.

Verify

  • At least two non-quality constraints influence the decision.
  • The decision records what would trigger reconsideration.

Expected outcome

  • A workload-specific scorecard and two defended selections.

Break / fix · diagnose

  • If the highest quality score always wins, add latency, privacy, memory or tool reliability constraints.
  • If local is assumed automatically safer, include model provenance and local log handling.
Production view

Production teams need repeatable model selection criteria because model releases and hardware options change quickly.

Local real-tool practicum · optional but recommended

Optional practicum — Ollama model identity and timing

Run one local model request and record runtime/model identity plus timing counters returned by the API.

Prerequisites

  • Ollama installed.
  • Set MODEL to an already-installed chat model; do not assume a specific model is present.

Resources created

  • Only local disposable files/processes used by the practicum.

Cost / risk

  • No cloud account required; model downloads consume disk and network bandwidth.
TOOL 1
ollama list
TOOL 2
MODEL=${MODEL:-qwen3:8b}; ollama show "$MODEL" | head -80
TOOL 3
MODEL=${MODEL:-qwen3:8b}; curl -s http://127.0.0.1:11434/api/chat -H 'Content-Type: application/json' -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Return exactly: MODEL_OK\"}],\"stream\":false}" | python3 -m json.tool | tail -40

Verify

  • The response includes the model identity and timing/token counters.

Cleanup

  • Remove only the disposable files created for the lab; keep model caches only if you intend to reuse them.

Warnings

  • Model names are examples, not endorsements. Validate against your own task and licence requirements.
Avoid these traps

Common mistakes

  • Selecting by parameter count alone.
  • Ignoring quantization/model identity in regression reports.
  • Benchmarking only warm requests or only one prompt.
  • Assuming local inference removes all privacy and governance obligations.
Knowledge check

Can you connect the AI behaviour to a system control?

Concept and scenario checks. Reveal the answer only after you commit to a decision.

1What should drive model selection?

Representative application evals plus operational constraints such as latency, memory, context, tool/schema reliability, governance and cost.

2Why record quantization/model digest?

They are part of the deployed artifact identity and can change quality, memory use and reproducibility.

3Why should context be budgeted?

Irrelevant context increases cost and distraction and leaves less room for useful evidence, tools and output.

4Scenario: a larger model is 3% better on a benchmark but violates the product latency SLO. What should happen?

It should not automatically be selected; evaluate candidates against the workload's quality and operational acceptance criteria.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.