Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 04 of 12

PromQL for investigation

Use PromQL to turn raw time series into rates, ratios, grouped views and latency distributions without hiding the failing dimension.

Learning objectives

By the end of this module, you should be able to:

  • Use `rate()` correctly with counters.
  • Preserve useful dimensions with `sum by(...)`.
  • Explain why aggregation order can hide a localized problem.

Rates make counters operational

Cumulative counters become useful for current traffic and error behavior when converted to rates over a window. Choose a window appropriate to scrape interval and operational sensitivity.

Aggregate after preserving the dimension you need

A single global average can hide one slow route, region or instance. Group by the dimension that matches the question before broad aggregation.

Ratios need aligned numerator and denominator

An error ratio compares error request rate with total request rate over the same population and time window. Mismatched labels or filters can produce misleading ratios.

Worked example

See the investigation idea in practice.

PromQL investigation patterns

rate(http_requests_total[5m])
sum by (route) (rate(http_requests_total[5m]))
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 04 — Query reasoning without a server

Goal: Evaluate grouping and ratio logic against a small request-rate dataset.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create route rates

cat > /tmp/rates.csv <<'EOF'
route,total,error,p95_ms
/,100,1,120
/search,80,2,180
/checkout,20,7,2200
EOF
STEP 2

Calculate error ratios

python3 - <<'PY'
import csv
for r in csv.DictReader(open('/tmp/rates.csv')):
 print(r['route'], round(100*float(r['error'])/float(r['total']),1),'%')
PY
STEP 3

Compare aggregation

Explain why a global error ratio is less diagnostic than grouping by route in this incident.

Verify

  • Checkout is identified as the worst route by both error ratio and p95 latency.
  • You explain which grouping dimension preserves the signal.

Expected outcome

  • A query plan that answers a specific operational question without aggregating away the problem.

Diagnose / troubleshoot

  • If `rate()` appears negative after hand-calculation, remember counters can reset; Prometheus `rate()` handles resets.
  • If a ratio is impossible (>1), inspect label alignment and numerator/denominator filters.
Real-world connection

Good PromQL is question-driven: query only after stating what you are trying to distinguish.

Optional real-tool lab

Optional tool lab — run PromQL

Execute basic PromQL against the local Prometheus API.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose up -d
TOOL 3
curl -Gs 'http://127.0.0.1:9090/api/v1/query' --data-urlencode 'query=up'
TOOL 4
curl -Gs 'http://127.0.0.1:9090/api/v1/query' --data-urlencode 'query=rate(prometheus_http_requests_total[5m])'

Verify

  • The API returns successful query results.
  • `up` has a value for the self-scrape target.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Using `rate()` on a gauge.
  • Aggregating all routes before looking for a localized failure.
  • Comparing numerator and denominator from different populations.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1Why use `rate()` with a counter?

To estimate per-second change over a time window while accounting for counter resets.

2What does `sum by (route)` preserve?

A separate aggregate for each route label value.

3Why can a global average mislead?

Healthy high-volume dimensions can mask a failing low-volume dimension.

4Scenario: global latency looks acceptable but checkout is slow. What helps?

Group or filter by route before global aggregation so the failing dimension remains visible.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.