AI & Machine Learning for Developers
Embeddings, Explained
An embedding is a list of numbers (a vector) produced by a model to represent a piece of data — most commonly text — in a way that captures something about its meaning rather than its exact wording. Two sentences that mean similar things end up with similar vectors, even if they don't share many words. This is the mechanism underneath semantic search, recommendation systems, and retrieval-augmented generation, covered elsewhere in this section.
Why it matters
- It's how "search by meaning" actually works
- A keyword search for "cheap flights" misses a document that says "budget airfare"; an embedding-based search can still find it, because the two phrases end up with similar vectors even though they share no words.
- It turns unstructured data into something a computer can compare
- There is no built-in way to ask "how similar are these two paragraphs?" without first converting both into numbers in a comparable form — embeddings are one standard way to do that conversion.
- It's the retrieval half of retrieval-augmented generation
- RAG systems, covered in this section's dedicated guide, use embeddings to find the most relevant pieces of text before handing them to a language model.
- The choice of embedding model has real, visible consequences
- Two embedding models can disagree meaningfully on which documents count as "similar," so swapping the model can change search results even when nothing else in the system changes.
What a vector actually captures
An embedding model converts a piece of text into a fixed-length list of numbers — for instance, a few hundred or a few thousand numbers per piece of text, the exact length depending on the model. That list is a point in a high-dimensional space, and the model is trained so that inputs with similar meaning end up as nearby points, while unrelated inputs end up far apart. "Nearby" and "far apart" are usually measured with a similarity calculation such as cosine similarity, which looks at the angle between two vectors rather than their raw values. Nothing in this process understands the text the way a person does — it is a learned statistical mapping from patterns in training data, so it captures the kind of similarity present in that data, not an objective notion of meaning.
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# Illustrative vectors only — a real embedding model
# would produce hundreds or thousands of numbers per input
vec_dog = np.array([0.9, 0.1, 0.4])
vec_puppy = np.array([0.85, 0.15, 0.42])
vec_stock_market = np.array([0.05, 0.95, 0.2])
print(cosine_similarity(vec_dog, vec_puppy)) # high: related concepts
print(cosine_similarity(vec_dog, vec_stock_market)) # low: unrelated conceptsWhere embeddings come from and what they're used for
Embedding models are trained on large amounts of text (or images, audio, and other data, depending on the model) using tasks that force the model to represent meaning to do well — predicting a missing word, or telling whether two pieces of text appeared near each other in the source data. Once trained, the model is used purely to produce vectors, not to generate new text. Common uses include semantic search (finding documents related to a query in meaning, not just keywords), recommendation (finding items similar to ones a user liked), clustering related content together, and, most relevantly for developers building on top of language models, retrieval for RAG systems. Different embedding models are trained differently and are not generally interchangeable — vectors from two different models are not directly comparable to each other.
Mistakes people make here
- Treating embedding similarity as a measure of truth or correctness
- A high similarity score means two pieces of text are related in meaning, not that either one is factually accurate — embeddings capture semantic closeness, nothing more.
- Comparing vectors produced by two different embedding models
- Each model learns its own internal geometry during training; a vector from one model has no defined relationship to a vector from a different model, so similarity comparisons across models are meaningless.
- Assuming embeddings understand negation or nuance perfectly
- "I love this product" and "I don't love this product" can end up closer together in embedding space than intuition suggests, because much of the surrounding wording is shared — embeddings capture broad topical similarity better than fine-grained logical distinctions.
- Not re-embedding content after it changes
- An embedding is a snapshot of the text at the time it was generated; if the underlying document is edited, the old vector becomes stale and has to be regenerated to stay accurate.
Strengths and trade-offs
Where it is strong
- Captures semantic similarity that exact keyword matching simply cannot, which is why it powers most modern search-by-meaning features.
- Works across formats — text, images, audio — as long as a suitable embedding model exists for that data type.
- Once computed, comparing vectors is a cheap, well-understood mathematical operation, which makes it practical to search over large collections.
The trade-offs
- Embeddings capture the kind of similarity present in the model's training data, not an objective or universal notion of meaning — biases and blind spots in that data carry through.
- They measure relatedness, not truth: a very similar piece of text can still be wrong, outdated, or contradictory.
- Quality depends heavily on the embedding model chosen, and there is no single model that's best for every domain or language.
- Storing and searching large numbers of vectors efficiently is its own problem, covered in this section's vector-databases guide, rather than something embeddings solve by themselves.
Who needs this
Any developer building search, recommendations, deduplication, or retrieval-augmented generation needs to understand what an embedding actually represents, since most of these systems are built directly on top of vector similarity. If you're only ever calling a language model directly with no search or retrieval step, you can treat this as useful background rather than a daily concern.
Questions about embeddings, explained
- Are embeddings the same as the vectors inside a neural network?
- They're related but not identical. An embedding is specifically the output vector meant to represent a whole input (a sentence, an image) for comparison purposes; a neural network also has many internal, intermediate vectors that aren't meant to be used this way.
- Can two very different sentences have similar embeddings?
- Yes, if they're about the same topic in the training data's sense of "topic," even with very different wording. This is usually the intended behavior, but it can also produce surprising matches.
- Do I need to train my own embedding model?
- Almost never for typical application work. Pre-trained embedding models are widely available and are trained on far more data than most teams could gather themselves; training your own is a specialized undertaking reserved for unusual requirements.
- How is this different from a language model generating text?
- A language model generates new text one piece at a time; an embedding model does not generate anything — it maps an entire input to one fixed vector meant purely for comparison. Some systems use models that can do both, but they are conceptually distinct tasks.