Ollama or vLLM? Choosing a serving stack for your own GPU

Both tools will serve a model behind /v1/chat/completions, both are open source, and both will happily run on a single GPU. Choosing between them is not a matter of quality. It is a question of how many requests you expect at once, and how much context you expect each one to hold.

What each tool actually is

Ollama is a friendly wrapper around llama.cpp with a model registry. Weights are GGUF files, usually quantised, and the runtime is optimised for getting a capable model running on modest hardware with one command. It has a REST API, an OpenAI-compatible endpoint, and it handles model downloads, swapping and prompt templates for you.

vLLM is a serving engine built for throughput. Two design choices define it: PagedAttention, which stores the KV cache in fixed-size blocks instead of one contiguous allocation per sequence, and continuous batching, which lets new requests join a batch that is already generating. The result is much higher tokens per second per GPU when requests overlap, at the cost of more moving parts.

The difference is the KV cache

Model weights are the easy part of the VRAM budget. Context is the part that surprises people.

Every token in a conversation needs a key and a value vector, for every layer. A useful approximation for a 8B-class model with 32 layers, 8 KV heads and a head dimension of 128:

• per token, per layer: 2 × 8 × 128 = 2,048 values
• across 32 layers: 65,536 values
• at two bytes each (fp16): about 128 KB per token

So a single 8,000-token conversation costs roughly 1 GB of KV cache. Four concurrent conversations of that length cost four times as much, on top of the weights.

Ollama allocates cache per sequence and is generally conservative about how many sequences it runs at once. vLLM packs the same memory into blocks and pages them, which is why it can hold far more concurrent sequences in the same VRAM, and why it degrades more gracefully as context grows.

Rough decision guide

If you are unsure, start with Ollama. Migrating later is mostly a matter of pointing your client at a different base URL, because both speak the same API shape.

Sizing the GPU honestly

Work backwards from weights plus cache plus overhead:

1. Weights. A 7-8B model at fp16 is about 14-16 GB. A 4-bit quantisation brings that to roughly 4-5 GB. A 70B model at 4-bit is about 40 GB, which means a single 48 GB card or two 24 GB cards.
2. KV cache. Use the arithmetic above. Context length times concurrent sequences is the number that actually determines whether you fit.
3. Overhead. CUDA context, activations and fragmentation. Leave 10-15% headroom.

If the sum does not fit, the lever that costs least quality is a shorter max_model_len, not a smaller model. Serving an 8k context when your prompts average 800 tokens is wasted VRAM.

Operational notes that matter after week one

• Keep weights on local NVMe. Loading a 40 GB model from network storage turns a restart into a coffee break. Loading from local NVMe is a matter of seconds.
• Pin versions. A model tag that silently updates underneath you will change your outputs, your latency and occasionally your licensing story.
• Cap concurrency deliberately. Both stacks let you limit batch size or sequence count. Uncapped concurrency turns a memory problem into an OOM kill at the worst moment.
• Watch VRAM, not just GPU utilisation. Utilisation near 100% is normal for inference. Memory climbing steadily across requests is a leak or a cache that is not being freed.
• Measure tokens per second at your real concurrency. Single-stream numbers favour llama.cpp builds; batch throughput tells a different story.

What this means for the hardware you rent

Inference is memory-bandwidth-bound as much as compute-bound. The card matters, but VRAM capacity and the bandwidth feeding it matter more than the headline FLOPS number. An instance with the right VRAM and a fast local disk will outperform a nominally faster card that spends its time loading weights and swapping cache.