Gnu Group Learning AcademyGnu Group ↗
LLMOps & AI Platform Engineering · Module 06 of 16

AI Gateways & Model Routing

An AI gateway sits between applications and model backends. It can authenticate callers, enforce quotas, choose an allowed model/provider, record routing decisions and apply fallback rules. The gateway should make policy explicit rather than letting every application embed its own provider logic.

Engineering focus: Gateway policy, backend selection, quotas and safe fallback

In plain English

What problem are we solving?

An AI gateway sits between applications and model backends. It can authenticate callers, enforce quotas, choose an allowed model/provider, record routing decisions and apply fallback rules. The gateway should make policy explicit rather than letting every application embed its own provider logic.

Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.

Animated mental model

Watch one request pass through policy before it reaches a model

The gateway first decides whether the request is allowed, then chooses an eligible backend from policy and health evidence.

● Loading interaction engine…
● PAUSEDRequest arrives
01Requestidentity + intentNOW
02Policyhard constraints
03Eligible setallowed + healthy
04Rankcost / latency / quality
05Backendselected model
06Decision evidenceroute reason
STEP 1/6Request arrives

The gateway starts with caller identity, requested capability and workload constraints.

sensitivity=internal
Learning objectives

By the end of this module, you should be able to:

  • Explain the gateway as a control boundary rather than merely a reverse proxy.
  • Separate hard routing constraints from soft optimization preferences.
  • Design fallback so it never bypasses security, residency or capability requirements.
  • Use routing evidence to troubleshoot why a request reached a specific backend.
🎓 Learning Coach follows this block when it is centered in your browser

A gateway centralizes decisions that should not be copied into every application

Without a gateway, each application may independently manage provider keys, endpoint URLs, retries, quotas and model names. That creates duplicated policy and makes changes hard to audit.

A gateway can centralize authentication, tenant quotas, model allow-lists, routing and telemetry. It does not replace application authorization or model-serving controls; it provides a shared enforcement point for cross-cutting policy.

  • Authenticate caller identity before expensive inference.
  • Map application model aliases to approved backends.
  • Enforce quotas/rate limits per tenant or workload.
  • Record why a route was chosen and which backend served it.
Common beginner misconception

A gateway does not make downstream model/tool behavior safe by itself. It controls access and routing, not semantic correctness.

🎓 Learning Coach follows this block when it is centered in your browser

Route by hard constraints first, preferences second

A routing policy should distinguish rules that must never be violated from preferences that can be optimized. Data residency, authorization and required model capability are hard constraints. Cost, latency and provider preference are usually soft objectives among eligible choices.

This ordering makes routing explainable: first build the eligible set, then rank it. If no backend is eligible, fail closed or request human/operator intervention rather than quietly weakening policy.

  • Hard: caller authorization, data classification/residency, required capability.
  • Hard: backend must be healthy enough to accept work.
  • Soft: lower cost, lower latency, preferred provider, locality.
  • No eligible backend: explicit rejection/degraded path, not silent policy bypass.
Common beginner misconception

Do not encode a security rule as a low-priority score. Hard constraints should be represented as hard constraints.

🎓 Learning Coach follows this block when it is centered in your browser

Fallback is controlled degradation, not a free second chance

Fallback can improve availability when a provider or model is unavailable, but it changes behavior. A smaller model may be less capable; a remote provider may violate data policy; a second request may double cost; a different tokenizer or tool behavior may alter results.

A safe fallback rule names when it is allowed, which alternatives are eligible, how many attempts are permitted and what evidence is recorded.

  • Fallback only among policy-eligible alternatives.
  • Bound the number of fallback hops.
  • Preserve request deadline across attempts instead of resetting it.
  • Record primary failure and fallback outcome separately.
Common beginner misconception

Successful fallback should not hide the fact that the primary backend is failing.

🎓 Learning Coach follows this block when it is centered in your browser

A routing decision should be explainable after the fact

When users report cost spikes, latency changes or quality regressions, operators need to know which backend actually served each request and why. A routing decision therefore needs structured evidence.

Useful evidence includes caller/tenant identity reference, logical model, selected backend/model version, policy version, decision reason, fallback status, latency and token usage. Sensitive prompts usually should not be copied into every routing log.

  • Use a request/trace ID across gateway and backend.
  • Record policy version and route reason.
  • Measure route distribution and fallback rate.
  • Redact or avoid sensitive content in telemetry by default.
Common beginner misconception

Logging full prompts is not the only way to make routing observable and may create a privacy problem.

Interactive simulator

Routing policy simulator: choose a backend from constraints

Change data sensitivity, task complexity, budget preference and provider health. The simulator explains why a route is accepted, rejected or degraded.

TEACHING MODEL
● Loading simulator engine…
Decision—

Change the policy inputs to see eligibility before ranking.

This simulator teaches relationships and diagnostic reasoning. Its synthetic values are not vendor benchmarks, hardware-sizing evidence or production SLO measurements.

Worked examples

Eligibility before optimization

request: sensitivity=restricted, capability=code

local-code      eligible   cost=2
cloud-fast      REJECTED   residency policy
cloud-strong    REJECTED   residency policy

selected: local-code
  • Cost is considered only after hard policy removes forbidden routes.
  • If local-code is unhealthy, the correct result may be no eligible backend.

Bounded fallback

deadline = 8s
primary attempt = 3s -> provider timeout
remaining budget = 5s
fallback = allowed strong-local
max fallback hops = 1
  • The fallback uses the remaining deadline rather than starting a fresh 8-second timer.
  • One failed request cannot fan out indefinitely across providers.
Try it · core lab · account-free

Route requests through explicit hard constraints and soft preferences

Goal: Build a deterministic routing policy and prove that fallback cannot bypass a hard data rule.

Before you start

  • Python 3 only. No model/provider accounts required.
STEP 1

Run the normal routing cases

python3 public/labs/llmops/06_gateway_routing_policy.py --self-test
STEP 2

Inspect a public low-cost request

python3 public/labs/llmops/06_gateway_routing_policy.py --sensitivity public --complexity low --prefer cost
STEP 3

Inspect a restricted request

python3 public/labs/llmops/06_gateway_routing_policy.py --sensitivity restricted --complexity high --prefer quality
STEP 4

Break the only eligible backend

python3 public/labs/llmops/06_gateway_routing_policy.py --sensitivity restricted --local-down
Download this Python labDownload all current labs

Verify

  • Public requests can choose an eligible lower-cost backend.
  • Restricted requests never route to cloud backends in the lab policy.
  • When the only approved backend is down, the result is no_eligible_backend instead of unsafe fallback.

Expected outcome

  • You can separate eligibility from ranking.
  • You understand that fallback must preserve hard policy constraints.

Break / fix · diagnose

  • If a restricted request reaches cloud in your modification, inspect the order of policy evaluation.
  • If provider health is treated as a cost score, move it into eligibility for this lab's policy.
Production view

Production gateways should make route decisions reproducible from policy version, backend health and request constraints without copying provider logic into every application.

Production Engineer checklist

Before you call this concept production-ready:

This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.

Avoid these traps

Common mistakes

  • Letting every application maintain its own provider credentials and routing table.
  • Using a cost score to represent a security or residency prohibition.
  • Failing over to a forbidden provider during an outage.
  • Logging complete prompt content merely to explain routing decisions.
Knowledge check

Reason about the system, not just the vocabulary.

Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.

1Why should hard constraints be evaluated before cost or latency preferences?

Because forbidden routes must never become eligible just because they are cheap or fast.

2A primary provider times out. What makes a fallback safe?

The alternate backend satisfies the same hard policy/capability constraints, attempts are bounded, and it uses the remaining request budget.

3What telemetry best explains an unexpected provider shift?

Selected backend/model, route reason, policy version, backend health/fallback state and a correlated request/trace ID.

4What should happen when no backend satisfies the request's hard constraints?

Fail closed or enter an explicitly approved degraded path; do not silently weaken policy.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.