In brief: Observability is useful when it shortens a decision, not when it merely produces more telemetry.
The problem
Infrastructure dashboards can stay green while checkout or publishing fails for one region or tenant tier. Unbounded labels explode cost, sampled traces miss rare errors, and noisy alerts train teams to ignore incidents.
Why the simple answer is not enough
CPU and memory describe components, not whether a customer completed work. SLIs should measure the event the product promises: successful publish, accepted payment or completed import.
Three pillars are not three independent data lakes. Consistent context lets an operator move from a burned SLO to a trace and then to the exact safe log evidence.
System flow
SLI and SLO→Metrics
fast aggregate signal→Traces
request causality→Logs
targeted evidence
Architecture decisions
Measure at the boundary
Count outcomes after authentication and business validation, excluding requests the contract says are ineligible.
Control cardinality
Keep route, region and result as metric dimensions. Put user, tenant and request identifiers in traces or logs with indexed sampling.
Page on symptoms
Use error-budget burn for urgent alerts. Component anomalies become diagnostic alerts unless they directly threaten the SLO.
Going deeper
Sampling must preserve rare failures
Use head sampling for baseline cost control and tail rules that retain errors, slow traces and selected high-value journeys. Record the effective rate.
SLOs govern change
When budget is healthy, teams can release quickly. When it burns, the agreed response shifts capacity toward reliability rather than starting a debate during an incident.
Implementation path
- Define availability and latency SLIs at the service boundary customers experience.
- Propagate trace, tenant and operation context with a controlled cardinality policy.
- Alert on multi-window error-budget burn, then attach runbooks and ownership.
Technical example: Multi-window burn-rate decision
budget = 1 - sloTarget
burnShort = errorRate(5m) / budget
burnLong = errorRate(1h) / budget
if burnShort > 14 and burnLong > 6:
page(owner, runbook, affectedJourney)Two windows catch fast incidents without paging on a brief, low-impact spike. Tune thresholds from historical incidents.
Failure modes to design for
- Trace context is dropped at a queue boundary.
- A tenant ID label creates millions of time series.
- Logs contain tokens or personal payloads during an incident.
What to observe
| Signal | Why it matters |
|---|---|
| Error-budget burn by journey | Connects reliability to release and investment decisions. |
| p50, p95 and p99 by region | Separates normal experience from tail problems. |
| Telemetry drop and sampling rate | Shows whether the evidence itself is incomplete. |
How to validate the design
- Break one user journey while keeping hosts healthy and prove the SLO alerts.
- Send high-cardinality identifiers and verify metric series remain bounded.
- Follow one request through HTTP, queue and worker with a single trace context.
Safe rollout plan
Pick one revenue or operational journey. Run its SLI beside existing alerts for several weeks, compare against real tickets, then replace only the alerts it outperforms. Add another journey after the first has a trusted owner and runbook.
Production checklist
- An SLO has an owner and a decision tied to error-budget consumption.
- Tenant ID is searchable but never an unbounded metrics label.
- Telemetry redacts secrets and applies retention by data class.
Takeaway
Good observability turns customer impact into a fast, evidence-based operational decision.