Skip to main content
FutureStack
All 8 Free Tools
AI & LLM100% Free • Runs Offline in Browser12 min systems guide

GPU VRAM, Quantization & KV-Cache Sizing Calculator

Find out exactly how much GPU memory (VRAM) your AI model needs on 4090, A100, and H100 cards.

Interactive Client-Side Workbench
Model Preset:
Static Model Weights
36.8 GB
Batched KV-Cache
20.0 GB
160 KB / token
CUDA & Activation Buffer
5.6 GB
Total Required VRAM
62.4 GB
62.4 GB / GPU @ TP=1
1x RTX 4090 / L4 (24 GB)OOM
1x RTX 6000 Ada / L40S (48 GB)OOM
1x A100 / H100 (80 GB)78% Used
1x H200 (141 GB)44% Used
2x H100 Tensor Parallel (160 GB)39% Used
4x H100 Tensor Parallel (320 GB)20% Used
8x H200 NVLink Node (1,128 GB)6% Used
Executive Technical Summary (AIEO Reference)

Total LLM serving VRAM equals Static Weights (Parameters * BytesPerParam, e.g., ~2.0 GB/B in FP16 or ~0.56 GB/B in INT4 AWQ) plus Batched KV-Cache (2 * Layers * KV_Heads * Head_Dim * KV_Bytes * ContextLen * BatchSize) plus 12%–18% CUDA activation and NCCL buffers. Using Grouped-Query Attention (GQA) with FP8 KV-cache slashes dynamic concurrency memory by 50% without accuracy collapse.

How to Use GPU Memory & VRAM Sizer

  1. 1
    Select a Model Preset or Enter Exact Transformer Architecture Dimensions

    Click Llama-3.3 70B, Qwen-2.5 32B, or enter custom Parameter Billions, Transformer Layers, GQA KV Heads, and Head Dimension from the model's config.json.

  2. 2
    Choose Weight Quantization, KV-Cache Precision, and Tensor Parallelism

    Select FP16, FP8, or INT4 weight precision alongside FP16 vs FP8 KV-cache and set your target context length, concurrent batch size, and TP GPU count.

  3. 3
    Verify Hardware Topology Headroom

    Check Per-GPU VRAM requirements against the hardware compatibility matrix—maintaining under 96% utilization to reserve safety headroom for PyTorch CUDA allocator fragmentation.

Core Capabilities & Specifications

  • Supports 8B, 14B, 32B, 70B, 405B presets plus custom Layers, GQA KV Heads & Head Dim inputs
  • Computes Grouped-Query Attention (GQA) KV-cache growth per token and per concurrent batch
  • Models FP16/BF16, FP8/INT8, and 4-bit (AWQ/GPTQ/Q4_K_M) weight quantization tiers
  • Simulates Tensor Parallelism (TP=1/2/4/8) per-GPU VRAM pressure
  • Maps total cluster memory directly onto RTX 4090, L40S, A100 80GB, H100, and H200 hardware topologies

The Exact Two-Part Equation for LLM Inference Memory

Serving an open-weights Transformer requires allocating VRAM across three distinct memory pools: static quantized model weights (stored across tensor-parallel ranks), the dynamic per-token Key-Value (KV) cache retained across every sequence inside the continuous-batching scheduler, and CUDA kernel activation buffers / PyTorch NCCL communication arenas (typically 10% to 18% overhead).

In modern Grouped-Query Attention (GQA) models—including Llama 3, Qwen 2.5, Mistral, and Gemma 2—sharing Key/Value projection heads across groups of 4 to 8 Query heads dramatically compresses KV-cache growth compared to legacy Multi-Head Attention (MHA). Without GQA, serving 32K–128K context windows at moderate batch concurrency would exhaust single-node HBM capacity almost immediately.

Exact GQA KV-Cache Footprint Formula (Bytes per Token)
Python
def kv_cache_bytes_per_token(
    num_layers: int,
    num_kv_heads: int,
    head_dim: int,
    kv_bytes: int = 2  # 2 for FP16/BF16, 1 for FP8 E4M3
) -> int:
    # 2 tensors per layer (Key + Value) * kv_heads * head_dim * bytes_per_elem
    return 2 * num_layers * num_kv_heads * head_dim * kv_bytes

# Llama-3.3 70B: 80 layers, 8 KV heads (GQA), 128 head_dim, FP16
bytes_tok = kv_cache_bytes_per_token(80, 8, 128, kv_bytes=2)
print(f"Per-token KV cache: {bytes_tok / 1024:.1f} KB")  # 320.0 KB / token
print(f"At 32k context, batch=8: {(bytes_tok * 32768 * 8) / (1024**3):.2f} GB")

Why Concurrency Fails with OOM Under Long Contexts

Engineering teams frequently purchase hardware based solely on static weight size (`Params * BytesPerParam`), forgetting that a single 70B user request at 64K context reserves 20 GB of FP16 KV-cache alone. Under a modest continuous batch of 16 concurrent users, KV-cache footprint exceeds 320 GB—dwarfing the 4-bit quantized weights (~39 GB) by more than 8x.

Enabling FP8 E4M3 KV-cache quantization alongside vLLM PagedAttention virtual block allocation (16-token blocks) cuts dynamic memory pressure in half while eliminating nearly all internal slot pre-allocation waste.

  • Static Weights: FP16 takes ~2.0 bytes/param, FP8 takes ~1.02 bytes/param (with scales), INT4 AWQ/GPTQ takes ~0.56 bytes/param.
  • Activation Scratchpad: Prefilling long prompts simultaneously triggers large temporary MLP and attention softmax buffers.
  • Continuous Batching Swap Pressure: When KV-cache saturates available HBM blocks, engines stall or evict/recompute in-flight sequences, spiking p99 tail latency.

Tensor Parallelism vs Pipeline Parallelism Across NVLink and PCIe

When weights exceed a single GPU's HBM (for example FP8 70B ~72 GB weights + KV-cache across 2x or 4x H100), Tensor Parallelism (`TP=2` or `TP=4`) shards individual attention heads and MLP rows evenly across GPUs. Because every single transformer layer executes two `all-reduce` collectives per forward pass (after Self-Attention and after the SwiGLU MLP), TP requires ultra-low-latency NVLink interconnects (450–900 GB/s bidirectional on H100/H200).

Deploying `TP=4` across PCIe Gen4 consumer cards introduces synchronization overhead per token step that often limits throughput gains unless using PCIe peer-to-peer P2P kernels or fitting quantized weights cleanly into fewer GPUs.

Production vLLM Launch Configuration Aligned with Memory Sizing
Bash
# Serve Llama-3.3 70B INT4 AWQ with FP8 KV-cache across 2x L40S/A100 (TP=2)
python3 -m vllm.entrypoints.openai.api_server \
  --model casperhansen/llama-3.3-70b-instruct-awq \
  --quantization awq_marlin \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

KV-Cache Bandwidth Ceiling and Time-Between-Tokens (TBT)

Unlike compute-bound prefill, autoregressive token generation hits an HBM memory-bandwidth wall: on a single H100 SXM (3.35 TB/s memory bandwidth), reading ~39 GB of INT4 70B weights plus batched KV-cache blocks caps single-stream decode speed around ~85–110 tokens/sec. Increasing batch concurrency amortizes reading those shared weight matrices across all active requests—boosting cluster tokens-per-dollar linearly until KV-cache blocks fill remaining VRAM.

Engineering Comparison Matrix

Model TierFP16 WeightsFP8 / INT8 WeightsINT4 (AWQ) WeightsRecommended Production Node
Llama 3.1 / 3.3 8B~16 GB~8.5 GB~5.0 GB1x L4 (24GB) or RTX 4090 (24GB)
Qwen 2.5 14B~28 GB~15 GB~8.5 GB1x RTX 4090 (24GB) or L40S (48GB)
Qwen 2.5 32B~64 GB~33 GB~18 GB1x A100/H100 (80GB) or 2x L40S
Llama 3.3 70B (GQA-8)~140 GB~72 GB~39 GB1x H200 (141GB) or 2x–4x H100 (TP)
Llama 3.1 405B~810 GB~415 GB~210 GB8x H100 / H200 NVLink Node

Common Production Pitfalls to Avoid

  • •Sizing GPUs strictly on checkpoint `.safetensors` disk size without reserving 25%–50% additional VRAM for batched KV-cache and CUDA graphs.
  • •Assuming 100% of VRAM is allocatable—CUDA runtime context, cuBLAS workspaces, and PyTorch allocator fragmentation consume 4–10 GB per GPU.
  • •Using Tensor Parallelism (`TP > 1`) across slow PCIe links during low-batch decode workloads.
  • •Leaving `gpu-memory-utilization` at 0.98 alongside multi-tenant image generation or embeddings on the same card.
  • •Ignoring Multi-Head Attention (MHA) KV-cache explosion on older checkpoints without GQA (`kv_heads == query_heads`).

Frequently Asked Questions

How much VRAM does a 70B model need at 4-bit quantization?

Model weights for a 70B 4-bit (AWQ/GPTQ/Q4_K_M) checkpoint consume ~38–42 GB, fitting on a single 48GB GPU at modest context—while production batched traffic (16K–32K context, batch 8–16) generally needs an 80GB A100/H100 or 2x 48GB GPUs for KV-cache capacity.

Does FP8 KV-cache hurt reasoning accuracy?

In modern vLLM and TensorRT-LLM kernels using per-head dynamic scaling factors, FP8 E4M3 KV-cache cuts dynamic KV memory by ~50% while retaining >99% accuracy across coding and long-context evaluations.

What is the exact formula for LLM KV-cache memory per token?

Multiply `2 * num_layers * num_kv_heads * head_dim * bytes_per_kv_element`. Multiply that byte cost per token by sequence context length and active batch size to obtain total dynamic KV-cache bytes.

How does PagedAttention prevent GPU out-of-memory crashes?

Traditional engines contiguously pre-allocated maximum context buffers (`max_seq_len`) per request even if a prompt only used 300 tokens. PagedAttention allocates non-contiguous 16-token physical pages on demand, recovering up to 60%–80% wasted VRAM.

Should I pick 4-bit weights + FP16 KV-cache or FP8 weights?

On memory-bandwidth-bound single-GPU setups, INT4 AWQ (Marlin kernel) + FP8 KV-cache yields higher tok/s throughput and larger batch capacity. Where hardware allows (like 2x–4x H100), FP8 weights maximize reasoning fidelity while keeping GEMM FLOP efficiency high.

Why does batch size matter so much for VRAM budgeting?

Static model weights are loaded once regardless of batch size, whereas every additional concurrent request multiplies its active KV-cache allocation linearly with context length.