Observability Fundamentals · Module 05 of 12

Grafana dashboards and visualization

Design dashboards that answer operational questions, preserve context and avoid decorative panels that encourage false certainty.

Learning objectives

By the end of this module, you should be able to:

  • Choose visualizations based on the question.
  • Design a service overview around traffic, errors, latency and saturation.
  • Use dashboard variables and drill-downs without hiding scope.

Dashboards should support decisions

Every panel should answer an operational question: Is user experience degrading? Which route or region? Is a resource saturated? What changed?

Overview first, drill down second

A good service dashboard exposes health and scope, then provides route/instance/dependency drill-downs. Avoid forcing users to scan dozens of equally weighted panels.

Visualization can distort

Axis ranges, units, aggregation and time windows change perception. A dashboard should expose units, scope and meaningful thresholds rather than rely on visual drama.

Worked example

See the investigation idea in practice.

Dashboard hierarchy

ROW 1: traffic | error ratio | p95 latency | saturation
ROW 2: by route / region
ROW 3: dependency health + deploy annotations
LINKS: logs | traces | runbook
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 05 — Review a dashboard specification

Goal: Detect panels that are redundant, unscoped or likely to hide an incident.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create dashboard spec

cat > /tmp/dashboard.txt <<'EOF'
requests_per_sec by route
error_ratio by route
p95_latency_ms by route
app_cpu_pct by instance
db_pool_pct
deploy_annotation
EOF
cat /tmp/dashboard.txt
STEP 2

Map each panel to a question

For every line, write the operational question it should answer.

STEP 3

Add drill-down links

Add one logs link and one traces link that carry service/time context.

Verify

  • Each panel has a question and unit/scope.
  • Dashboard includes both user symptom and dependency/resource evidence.

Expected outcome

  • A compact service overview with an investigation path rather than a wall of charts.

Diagnose / troubleshoot

  • If two panels answer the same question, remove or differentiate one.
  • If a panel says “high” without units/baseline, make the threshold and unit explicit.
Real-world connection

Dashboards are most effective when designed around incident decisions and team runbooks.

Optional real-tool lab

Optional tool lab — inspect provisioned Grafana

Verify the local Prometheus data source and Grafana health.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose up -d
TOOL 3
curl -fsS -u admin:academy-lab http://127.0.0.1:3001/api/datasources

Verify

  • A Prometheus data source is provisioned and points at the local Prometheus service.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Maximizing panel count.
  • Using pie charts for time-series investigation.
  • Showing averages when tail latency matters.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1What should determine a dashboard panel?

A concrete operational question or decision.

2Why show p95/p99 latency instead of only average?

Tail latency can reveal poor user experience hidden by a healthy average.

3What is a useful dashboard drill-down?

A context-preserving link from a metric symptom to relevant logs or traces.

4Scenario: a dashboard has 40 panels but responders cannot tell which route is failing. What is the design problem?

The dashboard lacks hierarchy/scope and should preserve actionable dimensions such as route before adding more panels.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.