What Is Running LLMs Locally and Why Do It

Running an LLM locally means the model file sits on your own computer or server and the text generation happens on your own CPU or GPU. No prompt leaves your network, and you do not pay per token. You trade the convenience and top quality of hosted APIs for control, privacy and predictable cost.

What actually happens when a model runs locally

A large language model is a big file of numbers called weights. To use it you need three things: the weights, a program that can read them (an inference engine), and hardware with enough memory to hold them. The engine loads the weights into RAM or GPU memory, turns your prompt into tokens, and then predicts one token at a time until the answer is finished.

Popular engines include Ollama, llama.cpp, LM Studio and vLLM. They all do the same core job. They differ in how easy they are to install, which hardware they favour, and how well they serve many users at once. This course covers all four.

The local stack in four layers

1. Model weights. Files you download, usually from Hugging Face or a registry such as the Ollama library. Examples of families are Llama, Qwen, Gemma, Mistral and Phi.
2. Inference engine. The program that runs the weights: Ollama, llama.cpp, LM Studio, vLLM and others.
3. API or interface. A terminal chat, a desktop app, or an HTTP server. Most engines expose an OpenAI-compatible endpoint so existing client code works with a small change.
4. Your application. A Python script, a Node.js service, a chat UI, an editor plugin.

Why people run models locally

• Privacy. Source code, contracts, medical notes and customer data stay on hardware you control.
• Cost at volume. After you own the hardware, extra tokens are almost free. This helps for batch jobs such as classifying a million support tickets.
• Offline use. A local model works on a plane, in a secure facility, or when your provider has an outage.
• Control. You choose the exact model file, so behaviour does not change under you when a provider updates a hosted model.
• Learning. Seeing memory use, token speed and sampling settings first-hand teaches you how LLMs really work.

Why people do not

The largest open-weight models need serious hardware, and the best hosted models are usually still stronger on hard reasoning and long, complex tasks. Local setups also make you responsible for updates, security and uptime. The next post compares both sides in more detail.

Your first taste in two commands

Ollama is the quickest way to try this. After installing it from ollama.com, you pull a small model and chat with it:

ollama pull llama3.2
ollama run llama3.2 "Explain what a hash map is in two sentences."

The first command downloads the model, which can be a couple of gigabytes. The second loads it into memory and prints an answer. On a modern laptop a small model like this answers in seconds. If the command works, you already have a local LLM and an HTTP API on port 11434.

You can prove the API is live with curl:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Say hello in French.",
  "stream": false
}'

The reply is a JSON object whose response field holds the generated text. Later posts build on exactly this pattern.

What you will learn in this course

The course starts with the vocabulary and first installs, then covers file formats and quantization, hardware sizing, using local models from Python and JavaScript, embeddings and retrieval, performance tuning, and finally serving a whole team securely. Fine-tuning is a separate topic and is not covered here.

Quick checklist before you start

• Check how much RAM you have and whether you have a GPU. Memory is the main limit.
• Start with a small model (a few billion parameters) so the first run is fast.
• Keep at least 10 to 20 GB of free disk space for model files.
• Treat the first model as an experiment. You will probably swap it several times.