Tokens and Tokenizers for Fine-Tuners: What Your Data Becomes

A language model never sees your text. It sees a sequence of integer token IDs produced by a tokenizer. Fine-tuning cost, memory use, truncation and even bugs all depend on tokens, so you should inspect how your data tokenizes before training.

What a token is

Most modern tokenizers use subword methods such as byte-pair encoding. Frequent words become one token, rarer words split into pieces, and unusual strings such as IDs or code can split into many tokens. A rough rule for English prose is that one token is a bit less than one word on average, but this varies by tokenizer and language. Never rely on the rule for budgeting. Count with the real tokenizer.

Inspect tokenization

import os
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained(os.environ["BASE_MODEL"])

text = "Invoice INV-2024-00871 totals 1,204.50 EUR."
ids = tok(text)["input_ids"]
print(len(ids), "tokens")
print(tok.convert_ids_to_tokens(ids))
print(tok.decode(ids))

You will see the list of pieces. Identifiers and numbers tend to split into several short tokens. Decoding returns the original text, perhaps with a special start token at the front depending on the tokenizer.

Special tokens

Tokenizers define special tokens such as beginning of sequence, end of sequence and padding. Chat models add more, for example markers for role boundaries. You can list them:

print(tok.special_tokens_map)
print("eos:", tok.eos_token, tok.eos_token_id)
print("pad:", tok.pad_token, tok.pad_token_id)

Two details trip people up. First, many tokenizers have no pad token. Trainers need one for batching, so you often set tok.pad_token = tok.eos_token. Second, the end-of-sequence token must appear at the end of each training answer. If it is missing, the model never learns to stop and keeps generating after it finishes.

Measure your dataset

Before training, compute token length statistics. They decide your maximum sequence length, your batch size and your cost.

import os
import numpy as np
from datasets import load_dataset
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained(os.environ["BASE_MODEL"])
ds = load_dataset("json", data_files="train.jsonl", split="train")

lengths = [len(tok(ex["text"])["input_ids"]) for ex in ds]
print("examples:", len(lengths))
print("mean:", np.mean(lengths))
print("p95:", np.percentile(lengths, 95))
print("max:", max(lengths))
print("total tokens:", sum(lengths))

This assumes each line has a text field. The p95 value is a good choice for the maximum length: it keeps almost all examples intact without padding every batch to a rare extreme. Total tokens times epochs gives the number of tokens processed, the basic unit of training cost.

Why tokens matter for fine-tuning

• Memory: activation memory grows with sequence length and batch size.
• Truncation: examples longer than the maximum are cut, which may remove the answer. Check how many are affected.
• Language: non-English text and code often use more tokens per character, so the same content costs more.
• Format tokens: chat templates add tokens for roles and separators. Count after templating, not before.

Never change the tokenizer casually

The model's embeddings are tied to its tokenizer. Use the tokenizer that ships with the checkpoint. If you add new special tokens, you must resize the embedding matrix with model.resize_token_embeddings(len(tok)) and train those new embeddings, which makes the job harder. Prefer existing tokens or plain-text markers when possible.

Checklist

• Load the tokenizer from the same checkpoint as the model.
• Confirm a pad token and an EOS token exist.
• Count tokens after applying the chat template.
• Report mean, p95 and max lengths, and the share of truncated examples.
• Keep padding side consistent: right for training, left for batched generation in most decoder models.

A worked example of a hidden bug

A team trains on chat transcripts and sees good loss, but at inference the model never stops talking. The cause is nearly always the end token. The training text ended with the assistant's last word and no end-of-sequence marker, so the model had no signal to stop. Printing one tokenized example and reading the last few token IDs would have caught it in a minute. Make this inspection a habit: decode one full training example back to text and read it, including the special tokens, before every run.

A second common bug is double beginning-of-sequence tokens. If the chat template already inserts a BOS token and you tokenize the templated text with default settings, the tokenizer adds another one. Check for repeated starts, and pass add_special_tokens=False when the template text already contains them.