Infrastructure & Platforms · core
Observability
Metrics, logs, traces, context propagation, semantic conventions, sampling, collectors, and telemetry pipelines.
Mental model
The three pillars answer different questions: metrics say something is wrong, traces say where, logs say why. A structured log plus a request id is what stitches them together. OpenTelemetry is the reason that stitching survives a service hop — it standardises the signals and, more importantly, propagates trace context across process boundaries, so instrumentation is portable and a trace does not end at the edge of the service that started it.
How to study Observability
Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use Site Reliability Engineering — Monitoring Distributed Systems, Canopy: An End-to-End Performance Tracing And Analysis System (SOSP '17), USENIX SREcon22 Asia/Pacific — OpenTelemetry and Observability: What, Why, and Why Now? to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.
Next, compare Observability with API Design. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete SLO error budget, Design exercise: Observability & OpenTelemetry and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.
Proof of understanding
- Explain the mechanism from first principles and identify the state it reads or changes.
- Give one situation where the concept is the right choice and one where it is not.
- Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
- Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.
Where it matters
OpenTelemetry, Datadog, PostHog, every production service.
Common mistakes
- Unstructured logs you cannot query
- Alerting on causes instead of user-facing symptoms (SLOs)
- No correlation id across services
- Instrumenting with a vendor SDK, so switching backends means re-instrumenting everything
- Dropping trace context at an async boundary, which silently truncates every downstream span
- Head sampling at a rate that discards exactly the rare failures you needed
Learn from primary sources
- Site Reliability Engineering — Monitoring Distributed Systems (doc)
- Canopy: An End-to-End Performance Tracing And Analysis System (SOSP '17) (doc)
- USENIX SREcon22 Asia/Pacific — OpenTelemetry and Observability: What, Why, and Why Now? (video)
- Dapper, a Large-Scale Distributed Systems Tracing Infrastructure (paper)
- OpenTelemetry Documentation (doc)
Practice and explain it back
SLO error budget
99.9% monthly SLO. How many minutes downtime allowed in 30 days? Burn 50% budget in 2 days — ship or freeze?
Expected evidence: ~43 min/month; freeze risky releases.
Open the interactive drill →Design exercise: Observability & OpenTelemetry
Metrics, logs, traces, context propagation, semantic conventions, sampling, collectors, and telemetry pipelines. Implement designOutline() returning non-empty values for: signals, contextPropagation, sampling. Each value must name a concrete mechanism or decision.
Expected evidence: A design outline with signals, contextPropagation, sampling plus an explicit failure mode or trade-off.
Open the interactive drill →Review prompts
- What question does each of metrics, traces, and logs answer?
- You have metrics and logs. What does a trace add that neither gives you?
Build evidence
Structured logging package
A structured logger with a request correlation id.
- JSON-structured log lines
- Correlation id propagated through a request
- Log levels and queryable fields
Synthesize: Infrastructure & Platforms
Design and operate a reproducible, observable, fault-tolerant platform for untrusted workloads. Produce one working system, benchmark, or evidence-backed design that integrates the path.
- Implements or precisely models the core mechanisms from all three milestones
- Includes at least one injected failure or adversarial case and demonstrates recovery
- Reports quality, latency, resource, reliability, or usability measurements relevant to the domain
- Ships a concise architecture note explaining decisions, trade-offs, and remaining risks
Prerequisites
None assigned yet.