A useful LLM memory estimate starts with the cache
By Mihai
Separate raw weights, context cache and runtime headroom before choosing a memory budget.
Weight precision is only the first term
An 8-billion-parameter model at an idealized four bits per weight needs 4 billion bytes for raw weights, about 3.73 GiB. Quantized model formats add scales, metadata and mixed-precision tensors. A file that appears small enough for a GPU may still leave too little space for the running application. Treat raw weight storage as a component of memory use rather than the final answer.
Context length changes the problem
For a single sequence, a basic KV-cache model multiplies context tokens by layers, two for keys and values, KV heads, head dimension and bytes per cache value. With 8,192 tokens, 32 layers, eight KV heads, a head dimension of 128 and two-byte values, this gives 1 GiB. Doubling context doubles this term while leaving the raw weight term unchanged. Attention heads and KV heads are not always the same number.
Concurrency needs its own allowance
The simple calculator models one sequence. A serving system handling several conversations can allocate a larger cache, and batching can require additional temporary memory. An advertised context limit describes what a model or runtime supports under certain assumptions; it does not mean that every GPU can use that limit with the same batch size. Test the actual request shape you expect to serve.
Make the budget explicit
Available memory should mean the amount the runtime can use after the display, driver, operating system and other applications. Unified-memory machines share capacity across more processes. Leave headroom rather than planning to consume the final fraction of a GiB. The calculator adds an editable percentage plus a fixed allowance, but neither number is an observed guarantee for a named inference engine.
Measure, then update the model
After loading a real model, measure idle allocation, a representative request and a longest expected request. Record the runtime, quantization format, context setting and concurrency. Use those observations to refine overhead assumptions. If a model only fits by heavy offloading, the memory calculation still does not predict acceptable speed. A useful planning record explains both capacity and the latency tradeoff.