What Are Embeddings? Turning Text into Vectors for Search

An embedding is a list of numbers that represents the meaning of a piece of text. An embedding model reads a sentence and returns a vector, for example 384 numbers. Texts with similar meaning get vectors that point in similar directions, so you can search by meaning by comparing vectors instead of comparing words.

Why numbers instead of words

Keyword search matches exact terms. It fails when users and documents use different words for the same idea. "Car", "automobile" and "vehicle" are three different strings to a keyword index. An embedding model has learned from huge amounts of text that these words appear in similar contexts, so it places them close together in vector space.

You can imagine a map. Each piece of text is a point. Sentences about cooking cluster in one region, sentences about databases in another. A question becomes a point on the same map, and the nearest points are the most relevant passages. Real embedding spaces have hundreds or thousands of dimensions rather than two, but the intuition holds.

Create embeddings in Python

The sentence-transformers library runs small embedding models locally, with no API key. Install it with pip install sentence-transformers. The model all-MiniLM-L6-v2 is small and returns 384 numbers per text.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

sentences = [
    "How do I reset my password?",
    "Steps to recover a forgotten login",
    "The cat sat on the mat",
]

vectors = model.encode(sentences)
print(vectors.shape)   # (3, 384)
print(vectors[0][:5])  # first five numbers of the first vector

The shape (3, 384) means three texts, each with 384 numbers. The individual numbers have no human meaning on their own. Only the relationships between vectors matter. The first two sentences ask about the same thing, so their vectors will be closer to each other than either is to the cat sentence. The next post shows how to measure that closeness.

Key properties to remember

• Fixed length. Every input, short or long, becomes a vector of the same size. The size is set by the model.
• Model specific. Vectors from different models are not comparable. You must use one model for both documents and queries.
• Input limit. Each model has a maximum input length. Text beyond it is cut off, which is one reason we split documents into chunks.
• Meaning, not exact words. Embeddings capture topic and intent well. They are weaker at exact identifiers such as error codes, SKUs or names, which is why keyword search is often added later.
• One vector per text. A long document compressed into one vector loses detail, so we embed chunks rather than whole files.

Queries and documents

Some models are trained so that a short question and the passage that answers it end up close together, even though they look different. These are called retrieval or asymmetric models. A few of them expect a prefix on the input such as "query:" or "passage:", or provide separate methods for queries and documents. Always read the model card and follow its instructions. Skipping a required prefix is a quiet way to lose accuracy.

What embeddings are not

Embeddings are not a database of facts. A vector does not "contain" the sentence in a way you can read back, and similarity is not truth. Two sentences that say opposite things about the same topic, such as "the service is free" and "the service is not free", can land very close together. Retrieval finds text that is about the right topic. The language model, or a reranker, must still read it carefully.

Where embeddings fit in RAG

1. At ingestion time you embed every chunk and store the vectors.
2. At query time you embed the question with the same model.
3. You ask the index for the stored vectors nearest to the question vector.

That is the whole mechanism. Everything else in vector search, including indexes such as HNSW, is about doing step three quickly when you have millions of vectors.

Common mistakes

• Embedding documents with one model and queries with another.
• Ignoring the model's maximum input length, so the end of long chunks is silently dropped.
• Changing the model later without re-embedding the whole corpus.
• Expecting exact-match behavior for codes and names from a semantic model.