Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 02 of 12

Metrics and instrumentation

Understand counters, gauges, histograms, labels and cardinality so metrics describe behavior without overwhelming the monitoring system.

Learning objectives

By the end of this module, you should be able to:

  • Choose counter, gauge and histogram types appropriately.
  • Calculate rates and ratios from cumulative counters.
  • Recognize label-cardinality explosions before production.

Metric types encode behavior

Counters increase for cumulative events such as requests or failures. Gauges rise and fall for current state such as queue depth. Histograms bucket observations so distributions and quantiles can be estimated.

Labels make metrics useful—and dangerous

Labels add dimensions such as method, route, status or region. Unbounded values such as user IDs, request IDs or raw URLs can create high cardinality and excessive series counts.

  • Prefer bounded dimensions
  • Normalize dynamic URL paths
  • Keep trace IDs in exemplars/logs/traces rather than metric labels

Golden signals are a starting lens

Latency, traffic, errors and saturation provide a useful service-health vocabulary. They are not a substitute for domain-specific indicators such as queue age, checkout success or replication lag.

Worked example

See the investigation idea in practice.

Counter vs gauge vs histogram

http_requests_total → counter
queue_depth → gauge
http_request_duration_seconds → histogram
BAD label: user_id=834921...
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 02 — Instrumentation design review

Goal: Classify metrics and estimate cardinality before emitting telemetry.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create a metric plan

cat > /tmp/metric-plan.txt <<'EOF'
http_requests_total counter method,route,status
queue_depth gauge queue
http_request_duration_seconds histogram method,route
EOF
cat /tmp/metric-plan.txt
STEP 2

Estimate series risk

python3 - <<'PY'
methods=4; routes=20; statuses=6
print('bounded series upper bound:',methods*routes*statuses)
users=100000
print('if user_id added:',methods*routes*statuses*users)
PY
STEP 3

Review dimensions

Remove identifiers that grow with users, requests or arbitrary values.

Verify

  • Each metric type matches the behavior being measured.
  • No unbounded identifier is proposed as a normal metric label.

Expected outcome

  • A bounded, investigation-oriented metric model with explicit cardinality reasoning.

Diagnose / troubleshoot

  • If a counter appears to decrease, check process restart/reset semantics.
  • If series count grows unexpectedly, inspect new labels and their value counts.
Real-world connection

Cardinality problems can increase cost, memory use and query latency before anyone notices a functional error.

Optional real-tool lab

Optional tool lab — inspect Prometheus self metrics

Use a real Prometheus target to see counters, gauges and labels.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose up -d
TOOL 3
curl -fsS http://127.0.0.1:9090/metrics | head -40

Verify

  • Prometheus exposes its own metrics endpoint.
  • You can identify at least one `_total` counter and one gauge-like current-state metric.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Using a gauge for cumulative request count.
  • Putting request IDs or user IDs in metric labels.
  • Publishing a histogram without thinking about bucket boundaries.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1When is a counter appropriate?

For cumulative events such as requests or errors that normally increase until reset.

2What does a gauge represent?

A current value that can increase or decrease, such as queue depth.

3Why use a histogram?

To capture a distribution of observations such as latency using buckets, count and sum.

4Scenario: a metric adds `user_id` as a label. What is the primary observability risk?

Unbounded cardinality can create an enormous number of time series and degrade cost/performance.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.