Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 03 of 12

Prometheus architecture and scraping

Learn the Prometheus pull model, scrape targets, service discovery concepts, time-series storage and why target health is part of telemetry reliability.

Learning objectives

By the end of this module, you should be able to:

  • Explain scrape targets and intervals.
  • Read a Prometheus scrape configuration.
  • Diagnose a DOWN target before blaming the application.

Prometheus scrapes targets

Prometheus periodically performs HTTP scrapes of configured or discovered targets. Each scrape records timestamped samples for metric names and label sets.

Target health is evidence

A target marked DOWN may mean DNS, routing, TLS, authentication, endpoint, exporter or application failure. `up == 0` describes scrape failure; it does not by itself prove why the target failed.

Collection has failure modes too

Monitoring systems need capacity, retention, HA and their own health checks. A missing metric can be an instrumentation or collection-path failure rather than healthy behavior.

Worked example

See the investigation idea in practice.

Minimal scrape configuration

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["prometheus:9090"]
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 03 — Validate a scrape configuration

Goal: Model targets and detect one intentionally unreachable endpoint.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create a self-contained target probe

python3 - <<'PY'
import socket,threading
server=socket.socket(); server.bind(('127.0.0.1',0)); up_port=server.getsockname()[1]; server.listen()
def accept_once():
 c,_=server.accept(); c.close(); server.close()
threading.Thread(target=accept_once,daemon=True).start()
unused=socket.socket(); unused.bind(('127.0.0.1',0)); down_port=unused.getsockname()[1]; unused.close()
def probe(port):
 s=socket.socket(); s.settimeout(.5)
 try: s.connect(('127.0.0.1',port)); return 'up'
 except OSError: return 'down'
 finally: s.close()
print('reachable target:',up_port,probe(up_port))
print('unreachable target:',down_port,probe(down_port))
PY
STEP 2

Interpret scope

A failed TCP/HTTP probe proves the collection path cannot reach that endpoint. It does not, by itself, identify the application root cause.

STEP 3

Build the diagnostic order

Check target address/path, DNS/routing, TLS/authentication, exporter/application endpoint, then Prometheus target details before changing the application.

Verify

  • The script reports one reachable local target and one deliberately closed local port.
  • You distinguish collection-path failure from confirmed application failure.

Expected outcome

  • A target-health mental model and a concrete diagnostic sequence.

Diagnose / troubleshoot

  • If every target is DOWN, inspect Prometheus networking/config before debugging each application.
  • If only one target is DOWN, compare its address, path, TLS/auth and endpoint behavior with healthy targets.
Real-world connection

Telemetry that cannot be collected is itself an operational failure requiring monitoring.

Optional real-tool lab

Optional tool lab — inspect targets

Use the Prometheus HTTP API to inspect actual target health.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose up -d
TOOL 3
curl -fsS http://127.0.0.1:9090/api/v1/targets

Verify

  • The `prometheus` self-scrape target reports health `up`.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Treating `up=0` as a complete root-cause diagnosis.
  • Assuming missing data means zero activity.
  • Changing scrape intervals without considering load and resolution.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1What does a Prometheus scrape do?

Fetches metric samples from a target endpoint at an interval and stores them as time series.

2What does `up=0` mean?

Prometheus could not successfully scrape that target; further diagnosis is required.

3Why monitor the monitoring system?

Collection, storage and query paths can fail independently of the application.

4Scenario: all application targets suddenly become DOWN but the apps serve traffic. What should you investigate first?

A shared collection-path problem such as Prometheus networking, DNS, credentials or configuration.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.