Parameters, Tokens and Context Windows: Vocabulary for Local LLMs

Every local LLM guide uses the same handful of terms. Parameters set the model's size, tokens are the unit of text it reads and writes, and the context window limits how much it can see at once. Knowing these makes hardware, speed and cost much easier to reason about.

Parameters

Parameters are the learned numbers inside the model. A "7B" model has roughly seven billion, an "70B" model seventy billion. More parameters usually mean more knowledge and better reasoning, but also more memory and slower generation. Parameter count is the single biggest factor in whether a model fits on your machine.

Tokens

Models do not read characters or words. A tokenizer splits text into pieces called tokens, which can be a whole short word, part of a long word, or punctuation. As a rough rule for English, one token is about three quarters of a word, so 100 words is around 130 tokens. Code and non-English text often use more tokens per word. Each model family has its own tokenizer, so the same text can have different token counts in different models.

Context window

The context window is the maximum number of tokens the model can handle in one request, counting your prompt, the conversation history and the answer together. If a model has an 8,192-token window and your prompt is 8,000 tokens, only about 192 tokens remain for the reply. Larger windows allow longer documents but need more memory, as post 22 explains. Engines also let you set a smaller working window than the model's maximum, and the defaults can be small, which causes surprising behaviour covered in post 37.

Prompt processing and generation

A request has two phases. First the model reads the whole prompt (called prefill or prompt processing). Then it produces the answer one token at a time (decoding or generation). Speed is usually quoted as tokens per second, and prefill and decode rates are different numbers. Prefill is fast because tokens are processed in parallel. Decode is slower because each new token depends on the previous one.

Sampling

At each step the model produces a probability for every possible next token. Sampling settings decide how to pick one. temperature controls randomness: low values are focused and repeatable, high values are more varied. top_p and top_k limit the candidate pool. Post 36 covers these in detail.

Quantization

Quantization stores weights with fewer bits, for example 4 bits instead of 16. This shrinks the file and memory use by a large factor with a modest quality loss, and it is why a 7B model can run on a laptop. Posts 13 and 14 go deeper.

See tokens for yourself

Ollama reports token counts and timings in its non-streaming response. This script prints them so you can connect the vocabulary to real numbers:

import json
import urllib.request

payload = {
    "model": "llama3.2",
    "prompt": "Write one sentence about the ocean.",
    "stream": False,
}
req = urllib.request.Request(
    "http://localhost:11434/api/generate",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req) as resp:
    data = json.load(resp)

print(data["response"])
print("prompt tokens:", data["prompt_eval_count"])
print("generated tokens:", data["eval_count"])
seconds = data["eval_duration"] / 1e9
print("generation speed: %.1f tokens/s" % (data["eval_count"] / seconds))

The fields prompt_eval_count, eval_count and eval_duration (in nanoseconds) come from Ollama's response. The script divides generated tokens by generation time to give tokens per second. Expect single digits on a weak CPU and dozens or more on a decent GPU or Apple Silicon.

Quick glossary

• Weights: the saved parameters.
• Inference: running a trained model to get output.
• Prompt: the input text, including any system message.
• Completion: the text the model generates.
• Embedding: a vector of numbers representing text meaning, used for search.

Keep these terms handy. The next post turns them into concrete hardware requirements.