How to Save VRAM When Running AI Models Locally Without Upgrading Your GPU

Running an artificial intelligence model on your own computer has become much more accessible, but VRAM is still what sets most of the limits. Quantization, FlashAttention, shorter context windows, an optimized KV cache, and CPU offload all help you get more out of a GPU, though it’s worth avoiding universal rules: a 13B model doesn’t always need the same amount of memory, and cutting layers doesn’t guarantee it keeps its quality.

The keys to cutting an LLM’s VRAM footprint in 30 seconds

• INT8 or INT4 quantization can drastically reduce the memory used by the weights.
• The VRAM required also depends on the context length, the KV cache, the batch size, and the inference engine.
• FlashAttention reduces memory during attention, but its savings aren’t a fixed percentage.
• CPU offload lets you exceed physical VRAM by using RAM, with a variable performance penalty.
• Tools like llama.cpp and vLLM offer several of these optimizations right out of the box.

The growth of open models and tools like llama.cpp, Ollama, vLLM, or Transformers has made it possible to run language models without sending every query to an external API. The problem shows up when you download a model and the GPU runs out of memory before it even generates the first token.

The solution doesn’t necessarily mean buying another graphics card. There’s plenty of room to adapt the model and how it runs to the hardware you already have.

An LLM’s VRAM usage doesn’t depend on parameter count alone

A first approximation makes the problem easy to understand. If a model has 7 billion parameters and each parameter takes up 16 bits, its weights need roughly 14 GB.

The basic formula is:

memory = number of parameters × bits per parameter / 8

Applying it to different precisions:

These figures are a guide for the weights alone, not the total memory requirement.

A model quantized to four bits takes up more than that simple multiplication suggests, because of the format used, scaling factors, metadata, and other overhead. During inference you also have to add execution buffers and, especially, the KV cache.

That’s why two users can run the exact same model and end up with quite different VRAM usage.

INT4 and INT8: quantize before you buy more memory

Quantization remains one of the most effective techniques for running large models on limited hardware.

Instead of storing the weights in FP16 or BF16, they’re represented with fewer bits. Going from 16 to 8 bits can bring the weights’ footprint close to half, while using four bits shrinks it even further.

The ecosystem currently offers plenty of alternatives.

llama.cpp typically uses models in GGUF format and supports several quantization levels. GPTQ and AWQ are two other widely used techniques, while Hugging Face integrates quantization through bitsandbytes. vLLM also supports different formats and techniques to reduce the memory needed during inference.

The trade-off is that quantizing isn’t entirely free.

Lowering precision can introduce quality loss, and that loss doesn’t have to be the same across every model or task. A model used for general conversation may tolerate aggressive quantization better than one intended for code, reasoning, or a specialized application.

The practical question is finding the point where the memory savings outweigh the possible degradation.

Context length can end up using more memory than you’d expect

There’s another factor that’s often overlooked when estimating VRAM: context length.

Transformer models keep information about previously processed tokens through what’s called the KV cache (Key-Value cache). Thanks to it, they don’t need to recompute the entire sequence every time they generate a new token.

The mechanism speeds up generation considerably, but it consumes memory.

And the larger the context, the more memory it can use.

That’s one of the reasons why automatically setting 64K, 128K, or more tokens simply because the model supports them can be a poor decision. Just because a model supports a given context window doesn’t mean an application needs to use it at all times.

On a limited GPU, lowering the maximum context length can free up a significant amount of memory without touching the weights.

Engines like vLLM also let you work with a quantized KV cache and control how much memory is reserved for it, which matters especially when serving multiple requests at once.

FlashAttention reduces memory, but it isn’t magic

FlashAttention is another optimization that has changed how transformers run.

Conventional attention can require large amounts of intermediate memory, especially as the sequence grows. FlashAttention reorganizes the calculations to reduce memory access and movement and avoid materializing certain full matrices.

The result can be lower memory use and higher speed.

However, crediting it with a universal saving — say, 30% of VRAM — is misleading.

The benefit depends on sequence length, architecture, batch size, GPU, implementation, and how much memory the weights already occupy.

If a model uses 20 GB for its weights on a 24 GB GPU, FlashAttention won’t turn that 20 GB into 14 GB. Its effect mainly applies to the memory used during the attention calculations.

PyTorch currently includes optimized implementations of scaled dot product attention, while inference-focused engines also take advantage of memory-efficient attention techniques.

Small batches for desktops, efficient batching for servers

Reducing the batch size is another simple option.

If there’s only one active request, there’s not much point reserving memory to process many sequences at once. A small batch size can reduce the activations and buffers needed during execution.

On a PC used by a single person, that can be a good strategy.

The situation changes in production.

A server serving dozens or hundreds of users needs to make the most of the GPU’s parallel capacity. Indiscriminately shrinking the batch could lower memory usage, but it could also waste a GPU that costs thousands or tens of thousands of euros.

vLLM specifically uses continuous batching to combine requests and improve how well the accelerator is used.

The right optimization, then, depends on whether the goal is to make a model fit on a home GPU or to maximize the tokens processed per second on a server.

CPU offload: using RAM when VRAM runs out

When a model is still too large, there’s a particularly practical solution: putting part of its weights in RAM.

llama.cpp lets you split the work between CPU and GPU. vLLM also offers CPU offload mechanisms.

That opens up scenarios that would otherwise be impossible. A machine with 24 GB of VRAM and plenty of RAM can run models whose weights physically exceed the GPU’s memory.

But RAM doesn’t magically turn into VRAM.

When the GPU needs data stored in main memory, that data has to move between the two components. On machines with a dedicated GPU, PCI Express then becomes part of the inference performance equation.

The larger the portion offloaded to the CPU, the bigger the impact can be.

That’s also why there’s no reliable rule saying CPU offload makes a model run only 20% or 30% slower. It can be less, but it can also be much more.

For personal use, that trade-off can work out just fine. For serving with high concurrency, maybe not.

Pruning layers isn’t the same as optimizing memory

Removing layers from a model deserves separate treatment.

It’s technically possible to shrink a transformer by removing certain layers, and there is research on layer pruning and other forms of structural compression.

But this changes the architecture.

It isn’t equivalent to running the same weights at four bits.

Removing the last two or four layers of a 13B model doesn’t automatically mean it will keep nearly all of its capability. The outcome depends heavily on the model and should be checked with evaluations on the tasks it will actually be used for.

For someone simply trying to run an LLM on a small GPU, quantization, context length, KV cache, and offload are usually more conservative options before turning to pruning.

An 8 GB GPU still has plenty of life left

Advances in quantization have given a second life to GPUs that once seemed too small for generative AI.

With 8 GB of VRAM you can run plenty of small and mid-sized quantized models. Models around 7B or 8B are especially well suited to this tier.

With 12 or 16 GB, the room to use larger models, wider contexts, or less aggressive quantization grows considerably.

24 GB GPUs open up another tier, letting you work with considerably larger models or split memory between weights and long contexts more comfortably.

Beyond that come multi-GPU setups and professional accelerators with much larger amounts of memory.

But having more VRAM doesn’t automatically make a setup better.

Optimization changes completely once the LLM reaches production

The obsession with getting a model to “fit” makes sense on a personal computer. In a company, the problem is considerably broader.

Inference infrastructure has to account for VRAM, memory bandwidth, compute power, RAM, CPU, storage, network, and power consumption all at once.

There are also metrics that don’t show up in a simple nvidia-smi snapshot: time to first token, tokens per second, concurrency, aggregate throughput, and cost per million tokens.

A 70B model running with a huge amount of CPU offload might technically work, but be a terrible platform for serving users.

And a correctly chosen, quantized, and specialized 8B model can handle certain tasks with far less infrastructure.

That’s where one of the most important decisions in any enterprise AI project comes in: don’t choose the model first and figure out where to run it afterward.

It’s worth starting with the process.

What input it receives, what output it needs to produce, what accuracy it requires, how many requests it will have to handle, what latency is acceptable, and what data is allowed to leave the organization.

Only then does it make sense to decide whether you need 8, 24, 80, or hundreds of GB of GPU memory.

Local inference engineering is moving precisely in that direction: getting more performance out of the same resources. Quantizing weights is part of that work, but the real optimization happens when the model, context, runtime, and hardware are designed as a single system.

Frequently asked questions

How much VRAM does a 13B model actually need?

Its weights need about 26 GB in FP16 and roughly 6.5 GB in theory at four bits. Real-world usage will be higher because you also need to add the quantization format overhead, KV cache, buffers, and runtime memory.

Is INT8 or INT4 better for running an LLM?

INT4 saves more memory, while INT8 preserves more numerical precision. The choice depends on the model, the hardware, and the required quality, so it’s worth evaluating the result on the actual task.

What can you do when a model doesn’t fit on the GPU?

The first options are using a smaller quantization, reducing the context length, and optimizing the KV cache. You can also turn to CPU offload or split the model across multiple GPUs if the inference engine supports it.

Does FlashAttention always reduce VRAM usage?

It mainly reduces the memory tied to the attention calculation, but there’s no fixed percentage of savings. Its effect depends on the model, the context length, the batch size, the GPU, and the implementation used.