core practice · coding problem
SLO error budget
99.9% monthly SLO. How many minutes downtime allowed in 30 days? Burn 50% budget in 2 days — ship or freeze?
The mechanism
The three pillars answer different questions: metrics say something is wrong, traces say where, logs say why. A structured log plus a request id is what stitches them together. OpenTelemetry is the reason that stitching survives a service hop — it standardises the signals and, more importantly, propagates trace context across process boundaries, so instrumentation is portable and a trace does not end at the edge of the service that started it.
Expected evidence
~43 min/month; freeze risky releases.
Try this drill in the practice workspace →
Hints when you need them
- Budget = 1 − SLO over window.
- Fast burn → stop deploying.
Watch for these mistakes
- Unstructured logs you cannot query
- Alerting on causes instead of user-facing symptoms (SLOs)
- No correlation id across services
- Instrumenting with a vendor SDK, so switching backends means re-instrumenting everything
- Dropping trace context at an async boundary, which silently truncates every downstream span
- Head sampling at a rate that discards exactly the rare failures you needed
Check the sources
- Site Reliability Engineering — Monitoring Distributed Systems
- Canopy: An End-to-End Performance Tracing And Analysis System (SOSP '17)
- USENIX SREcon22 Asia/Pacific — OpenTelemetry and Observability: What, Why, and Why Now?
- Dapper, a Large-Scale Distributed Systems Tracing Infrastructure
- OpenTelemetry Documentation