Quick reference
| Setting | Value / behavior |
|---|---|
| 8B weights at 4 bits | 3.73 GiB before metadata |
| 8B weights at 8 bits | 7.45 GiB before metadata |
| 8B weights at 16 bits | 14.90 GiB before metadata |
| KV cache | Grows linearly with context |
Read the model configuration first
Parameter count alone cannot tell you whether a model will fit on a GPU. Precision determines weight storage, while the attention architecture and context length determine much of the KV cache. Find the layer count, number of key/value heads and head dimension in the model’s configuration. KV heads can differ from attention heads in grouped-query attention. The defaults illustrate an 8-billion-parameter configuration; they are not an authoritative specification for every 8B model.
Memory formula
Weight memory is parameters × bits per weight ÷ 8, converted from bytes to GiB by dividing by 2 to the power of 30. KV cache is context tokens × layers × 2 × KV heads × head dimension × bytes per cache value. The factor of two accounts for keys and values. The estimate adds an editable percentage of weight memory plus a fixed 1 GiB runtime allowance. With the defaults, weights are about 3.73 GiB, cache is 1 GiB and the total estimate is about 6.47 GiB.
Why actual usage differs
Quantized file formats include scales, block metadata and tensors at mixed precisions. Runtimes allocate work buffers and may reserve memory. Multiple sequences, parallel requests and larger batches can increase cache and temporary allocations. Mixture-of-experts models often store more parameters than the active parameter count suggests. GPU drivers, displays and other programs also consume memory. Unified memory on a laptop is shared with its operating system; the advertised system capacity is not a safe model budget by itself.
Use the answer as a planning range
Compare the total with memory actually available to the runtime, leaving additional headroom for the application. A green fit result means only that this formula is below the budget you entered; it does not test kernels, context support, speed, offloading or hardware compatibility. If the estimate exceeds the budget, try a smaller context or different weight precision and observe which component changes. Measure the real process once deployed and use that observation to adjust the overhead for the next estimate.
Frequently asked questions
Is this a guarantee that a model runs?
No. This is a planning estimate, not a hardware or runtime compatibility test.
Does 4-bit mean exactly half of 8-bit memory?
Only raw weight storage halves. Metadata, cache and runtime memory do not necessarily halve.
Is batch size included?
The cache formula models one sequence. Account separately for concurrency and batching.
About this tool
Built and maintained by Mihai at Void Fabric. This page documents the behavior of the current implementation. Found an error or an unclear assumption? Send a reproducible example with sensitive values removed.
Reference: Void Fabric field notes · Testing and editorial policy