What Hardware Do You Need to Run an LLM Locally

The most important spec for running an LLM locally is memory: RAM, or GPU memory (VRAM) if you use a graphics card. The model must fit, with some room left for the context. After memory, memory bandwidth decides how fast text appears. A modern laptop with 16 GB can run small models; larger models need more memory or a GPU.

The one rule: the model has to fit

A quantized model needs roughly its file size in memory, plus extra for the KV cache and the engine. A 4-bit 7B or 8B model is about 4 to 5 GB. A 4-bit 14B model is about 8 to 9 GB. A 4-bit 70B model is about 40 GB. Add a gigabyte or two of headroom. Post 21 shows the exact calculation.

Starting points by machine

Machine	Sensible model sizes (4-bit)
8 GB laptop, no GPU	1B to 3B, patience needed
16 GB laptop or Apple Silicon Mac	Up to about 7B or 8B comfortably
32 GB Mac or PC	Up to about 14B to 32B depending on context
GPU with 8 GB VRAM	7B or 8B fully on GPU
GPU with 24 GB VRAM	Up to about 30B-class models
64 GB or more unified memory / multi-GPU	70B-class models

These are guidelines, not guarantees. Context length, the exact quantization, and what else is using memory all move the numbers.

CPU, GPU or Apple Silicon

• CPU only. Works everywhere and is fine for small models and batch work. Generation is slower because system RAM has lower bandwidth than GPU memory.
• NVIDIA GPU. The best-supported option. Fast generation if the model fits in VRAM. Engines like llama.cpp, Ollama and vLLM all support CUDA.
• Apple Silicon. CPU and GPU share one pool of unified memory, so a Mac with lots of memory can hold models that would need an expensive GPU on a PC. Speed is good, though usually below a high-end desktop GPU.
• AMD and others. Supported to varying degrees through ROCm or Vulkan. Check your engine's documentation for your exact card.

Storage and other resources

Models are large files. Keep an SSD with tens of gigabytes free, since a handful of models adds up quickly. Loading from an SSD is much faster than from a spinning disk. Power and cooling matter too: sustained generation pushes a GPU or laptop CPU hard, which can cause fans to spin and clocks to drop.

Check your own machine

These commands show what you have:

# Linux: RAM and CPU
free -h
lscpu | head -20

# NVIDIA GPU memory
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

# macOS: total memory in bytes and chip name
sysctl hw.memsize
sysctl -n machdep.cpu.brand_string

nvidia-smi prints the GPU name and its total and used memory as CSV. On macOS, hw.memsize is the installed memory in bytes; divide by 1,073,741,824 for gigabytes.

After you install Ollama: see where the model landed

Run a model, then ask Ollama where it loaded:

ollama run llama3.2 "hi" > /dev/null
ollama ps

The PROCESSOR column shows how much of the model is on GPU versus CPU, for example 100% GPU. Anything partly on CPU will be slower, which is the first clue that the model is too big for your VRAM.

Buying advice in brief

• For GPUs, VRAM capacity usually matters more than raw compute for LLMs.
• For Macs, buy more memory than you think you need; it cannot be upgraded later.
• Do not buy anything before trying small models on what you own.
• Leave 20 percent or more of memory free for the operating system and context.