12mo roadmap · 9 milestones
Disk-First Databases & RAM
Companion to docs/learning/db-roadmap.md. Phased link-hub roadmap from the disk-first DB PDF: RAM vs SSD, storage engines, LSM, columnar/vectorized execution, ClickHouse internals, cloud warehouses, search/vector, distributed correctness, and a synthesis project.
Outcome: Build a mechanism-first mental model of how disk-based DBs spend RAM, and why.
How to work this roadmap
Treat each milestone as a claim that must be supported by evidence. Before reading, write what you think the mechanism does and where it will fail. After studying the linked concepts, run the drills without copying an answer, preserve the output, and revise the explanation. Move forward when you can connect the milestone goal to a working implementation, benchmark, architecture decision, or reviewable design artifact.
The 12mo horizon is a sequencing aid, not a completion badge. Spend more time where your prediction and the observed behavior disagree. Keep a short decision log containing the mechanism selected, alternatives rejected, expected failure mode, measurement used, and remaining uncertainty. Review that log with FSRS prompts so the roadmap produces durable system judgment rather than a temporary tour of terminology.
At the end, explain Build a mechanism-first mental model of how disk-based DBs spend RAM, and why. from first principles to a reader outside the domain. A strong explanation should survive follow-up questions about correctness, cost, latency, resource use, security, recovery, and operational visibility. If it cannot, return to the milestone that contains the missing mechanism and build a smaller falsifiable example.
Milestones
Milestone 1
Prerequisites — unlock the rest of this path
Concepts the later milestones depend on. Without these the planner cannot serve the rest of this roadmap.
Concepts
Milestone 2
Phase 0-1 — Thesis + hardware
Frame active state vs stored data; understand CPU cache, DRAM, NVMe, page cache, mmap.
Concepts
Milestone 3
Phase 2 — Disk-oriented storage engines
Pages, B+trees, WAL, buffer pools, recovery.
Concepts
Build evidence
- Toy write-ahead log — An append-only WAL with crash recovery.
Milestone 4
Phase 3 — LSM and SSD-first design
Memtables, SSTables, Bloom filters, compaction, read/write/space amplification.
Concepts
Milestone 5
Phase 4-5 — Columnar execution + ClickHouse
Vectorized execution, projection/filter pushdown, external operators, MergeTree internals.
Concepts
Build evidence
- Object-storage-backed index — Store index segments in object storage with a hot in-memory cache.
Milestone 6
Phase 6 — Cloud warehouses
Storage/compute separation, shuffle, slots/warehouses, result caching.
Concepts
Build evidence
- Object-storage-backed index — Store index segments in object storage with a hot in-memory cache.
Milestone 7
Phase 7 — Search + vector DBs
Inverted indexes, BM25, HNSW, IVF, PQ, DiskANN, recall/latency/memory tradeoff.
Concepts
Milestone 8
Phase 8 — Distributed correctness
Replication, Raft/Paxos, MVCC across nodes, 2PC, serializability.
Concepts
Milestone 9
Phase 9 — Synthesis project
Ship one of: low-RAM benchmark, ch-lowram CLI, spill-first OLAP engine, or disk-backed vector search.
Concepts
None assigned yet.