Gnu Group Learning AcademyGnu Group ↗
LLMOps & AI Platform Engineering · Module 02 of 16

Anatomy of an AI Application

An AI application is usually several cooperating parts. The user talks to the application, the application prepares a model request, an inference service runs the model, and the response travels back. Gateways, retrieval, agents, policy and telemetry can be inserted around that path.

Engineering focus: Request path, runtime boundaries and failure domains

In plain English

What problem are we solving?

An AI application is usually several cooperating parts. The user talks to the application, the application prepares a model request, an inference service runs the model, and the response travels back. Gateways, retrieval, agents, policy and telemetry can be inserted around that path.

Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.

Animated mental model

Follow request REQ-18427 end-to-end

Watch where time is spent and notice that 'the AI is slow' does not identify which component is slow.

● Loading interaction engine…
● PAUSEDRequest created
01Client18 msNOW
02Application12 ms
03Gateway20 ms
04Queue810 ms
05Inference2.1 s
06Stream1.4 s
STEP 1/6Request created

REQ-18427 starts at the client. Give it an identity before it crosses service boundaries.

request_id=REQ-18427
Learning objectives

By the end of this module, you should be able to:

  • Trace a request through client, application, gateway, inference service, model runtime and hardware.
  • Separate application responsibilities from model-serving responsibilities.
  • Identify common failure domains and the evidence each layer should expose.
  • Explain why request IDs and stable API contracts make troubleshooting easier.
🎓 Learning Coach follows this block when it is centered in your browser

Start with one request and follow it

A useful way to understand any AI architecture is to ignore the product names and trace one request. Ask where it enters, where it waits, where it is transformed, where inference happens and where the result returns.

This turns a complicated platform diagram into a sequence of understandable responsibilities.

  • Client: captures user intent.
  • Application/API: validates input and applies domain logic.
  • Gateway/policy layer: controls model access and routing where used.
  • Inference service: turns a request into model execution.
  • Model runtime/hardware: performs computation.
  • Response path: streams or returns generated output to the user.
Common beginner misconception

The important thing is to understand the boundary and evidence before memorizing implementation details.

🎓 Learning Coach follows this block when it is centered in your browser

Interfaces matter more than implementation details

The application should communicate with the serving layer through a documented contract: endpoint, request schema, response schema, errors and streaming behavior. That contract allows the backend implementation to evolve without forcing every application to be rewritten.

An OpenAI-compatible API is one common contract in today's ecosystem, but compatibility does not imply identical behavior, security or performance across servers.

  • Define request and response shapes.
  • Define authentication and authorization separately from payload content.
  • Define timeout and cancellation behavior.
  • Define model/version identifiers.
  • Treat compatibility claims as an interface starting point, not an operational guarantee.
Common beginner misconception

API compatibility does not mean performance compatibility or security equivalence.

🎓 Learning Coach follows this block when it is centered in your browser

Every box is a possible failure domain

A user-visible failure may begin far from the model. DNS can fail, authentication can reject a token, the application can exceed a deadline, a gateway can route incorrectly, a queue can saturate, the inference process can run out of memory or the model can return low-quality output.

Good architecture makes these failures distinguishable instead of collapsing them into a generic 'AI failed' message.

  • Transport failure: connection, DNS or TLS.
  • Policy failure: authentication, authorization or quota.
  • Capacity failure: queue saturation, memory pressure or unavailable accelerators.
  • Runtime failure: process crash, invalid model state or dependency failure.
  • Semantic failure: fluent response that is still incorrect, ungrounded or unsafe.
Common beginner misconception

The important thing is to understand the boundary and evidence before memorizing implementation details.

🎓 Learning Coach follows this block when it is centered in your browser

RAG and agents are additional paths, not the foundation of every request

Some applications retrieve documents before inference. Others allow an agent runtime to call tools. These add useful capabilities but also add latency, policy boundaries and more failure modes.

Keep the base request path clear first. Then add retrieval and tools only where the product needs them.

  • RAG adds retrieval, ranking and source freshness concerns.
  • Agents add tool execution, state, approval and budget concerns.
  • Both should preserve end-to-end correlation and evaluation evidence.
Common beginner misconception

The important thing is to understand the boundary and evidence before memorizing implementation details.

Worked examples

One request, several timings

REQ-18427
client       18 ms
application  12 ms
gateway      20 ms
queue       810 ms
inference  2100 ms
streaming  1400 ms
----------------
total      4360 ms
  • A single total latency number hides where time is spent.
  • Queue time and generation time require different fixes.

Separate infrastructure and semantic failure

HTTP 200 + fluent wrong answer = semantic failure
HTTP 503 + no answer          = service failure
  • Both matter to users, but they need different detectors and remediation paths.
  • Evaluation catches failures that ordinary uptime checks cannot.
Try it · core lab · account-free

Trace a simulated AI request

Goal: Learn to localize latency and failure to one stage rather than blaming 'the model'.

Before you start

  • Python 3 only. The lab is deterministic and does not call an external AI service.
STEP 1

Run the healthy trace

python3 public/labs/llmops/02_trace_ai_request.py healthy
STEP 2

Inject queue pressure

python3 public/labs/llmops/02_trace_ai_request.py queue-spike
STEP 3

Inject a policy failure

python3 public/labs/llmops/02_trace_ai_request.py auth-denied
STEP 4

Compare evidence

For each run, identify the failing or dominant stage and choose the first sensible investigation.

Download this Python labDownload all current labs

Verify

  • Every stage carries the same request ID.
  • Queue-spike shows latency before inference rather than slow token generation.
  • Auth-denied stops before inference and is classified as a policy failure.

Expected outcome

  • You can distinguish queue latency from inference latency.
  • You avoid retrying a permanent authorization failure.

Break / fix · diagnose

  • If total latency surprises you, add the stage durations manually and compare.
  • If you blame the model for every scenario, point to the exact trace stage where evidence first changes.
Production view

Request correlation and stage timing are foundational for debugging gateways, RAG pipelines, agent tools and inference backends.

Production Engineer checklist

Before you call this concept production-ready:

This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.

Avoid these traps

Common mistakes

  • Drawing a large architecture diagram without being able to trace one request through it.
  • Using one generic error for transport, policy, capacity and semantic failures.
  • Assuming an OpenAI-compatible endpoint behaves identically across every backend.
  • Adding RAG or agents before the base serving path is observable.
Knowledge check

Reason about the system, not just the vocabulary.

Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.

1Why is a request ID useful in an AI platform?

It correlates evidence across application, gateway, retrieval/tool and inference layers for one user request.

2A request spends 2.5 seconds waiting and 0.8 seconds generating. Where is the first performance problem?

Queue/capacity before inference; generation is not the dominant delay.

3What is a semantic failure?

The infrastructure successfully returns output, but the output is incorrect, ungrounded, unsafe or otherwise fails the task.

4Why keep the application/inference boundary explicit?

It creates a stable contract and separates domain logic from model-serving implementation, making replacement and troubleshooting easier.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.