Vector DB & ANN · core

Scalar & Binary Quantization

Shrinking vectors to int8 or single bits, and rescoring to recover the lost precision.

vector-dbcompression

Mental model

Quantisation trades recall for memory, and memory is what decides whether an index fits in RAM. Scalar quantisation maps each float32 dimension to int8 for a 4x reduction and small recall loss; binary quantisation keeps one bit per dimension for 32x, turning distance into a Hamming popcount that is enormously faster but far coarser. The standard pattern is two-stage: retrieve a generous candidate set with the compressed vectors, then rescore the survivors against full-precision copies, which recovers most of the lost accuracy for a fraction of the memory.

How to study Scalar & Binary Quantization

Begin by restating the mental model in your own words, then connect it to a concrete system you have built or operated. Name the mechanism, the constraint it addresses, and the trade-off it introduces. Use the linked roadmap and primary implementation references to check details, but close the source before writing your explanation. Retrieval is the learning step; rereading is only preparation.

Next, compare Scalar & Binary Quantization with Product Quantization, HNSW. Ask what changes in correctness, latency, resource use, operability, and failure recovery. Complete Scalar quantise and rescore and preserve the command, input, output, and one failed attempt as evidence. Finish by explaining the idea without jargon to someone who has not studied the track.

Proof of understanding

Explain the mechanism from first principles and identify the state it reads or changes.
Give one situation where the concept is the right choice and one where it is not.
Predict a realistic failure mode before running the drill, then compare the prediction with evidence.
Connect the result to a roadmap or build artifact instead of treating the concept as isolated trivia.

Where it matters

The reason a billion-vector index fits on one machine, and the first lever when a vector database is memory-bound.

Common mistakes

Quantising without a rescore step and accepting the recall drop as inevitable
Calibrating scalar ranges on unrepresentative data, so outliers clip
Using binary quantisation on low-dimensional embeddings, where there are too few bits to preserve structure
Measuring only the memory win and never re-measuring recall@k afterwards

Learn from primary sources

Use the linked roadmap context and practice prompt.

Practice and explain it back

Scalar quantise and rescore

Implement quantize(vec, min, max) mapping each float to an int in 0..255, dequantize(codes, min, max) mapping back to the bucket centre, and maxAbsError(vec, min, max) returning the largest absolute reconstruction error. Values outside [min,max] clamp.

Expected evidence: Error is bounded by half a bucket width: (max-min)/255/2.

Open the interactive drill →

Review prompts

Binary quantisation cuts memory 32x but loses a lot of precision. Why does the two-stage retrieve-then-rescore pattern get most of the accuracy back?