Building Dashboards

Philosophy

1. Decisions first. Every panel answers a question that leads to an action.
2. Overview → drilldown → evidence. Start broad, narrow on click/filter, end with raw logs.
3. Rates and percentiles over averages. Averages hide problems; p95/p99 expose them.
4. Simple beats dense. One question per panel. No chart junk.
5. Validate with data. Never guess fields, discover schema first.
6. Compute what's asked, or defer. If a panel can't be computed, replace it with a Note documenting the blocker. Never substitute a different quantity, even disclosed. See Compute or Defer.

---

Entry Points

Starting from	Workflow
Vague description	Intake → check dataset kind → design blueprint (APL or MPL) → queries per panel → deploy
Template	Pick template → customize dataset/service/env → deploy
Splunk dashboard	Extract SPL → translate via spl-to-apl → map to chart types → deploy
Grafana dashboard	Project canonical panel spec (expr, legendFormat, unit, title, description) → translate PromQL → map chart types → deploy. See reference/grafana-migration.md.
Exploration	Use axiom-sre to discover schema/signals → productize into panels

---

Intake: What to Ask First

1. Audience & decision

   • Oncall triage? (fast refresh, error-focused)
   • Team health? (daily trends, SLO tracking)
   • Exec reporting? (weekly summaries, high-level)

2. Scope

   • Service, environment, region, cluster, endpoint?
   • Single service or cross-service view?

3. Dataset kind. Run scripts/metrics/datasets <deploy> and check kind.

   • otel:metrics:v1 → metrics dataset, follow the Metrics path.
   • anything else → events/logs dataset, follow the APL path.

   Never run getschema on a metrics dataset. It returns 0 rows without error.