What are Embeddings?
Embeddings are dense vector representations that capture semantic meaning. Similar concepts have similar embeddings, enabling machines to understand relationships between words, sentences, and documents.
How Embeddings Work
Embedding models map discrete tokens to continuous vectors in a high-dimensional space (often 768-4096 dimensions). The position of each vector encodes its semantic meaning.
Semantic Similarity
Similar meanings cluster together in embedding space. "King" and "Queen" are closer than "King" and "Banana".
Vector Dimensions
Each dimension captures some aspect of meaning—though these dimensions aren't human-interpretable.
Vector Operations
Classic Word2Vec-style intuition: King - Man + Woman ≈ Queen. It is a useful mental model, but modern contextual LLM representations are more dynamic and less clean.
How Embeddings Are Created
Embeddings come from the embedding layer—a learned lookup table that sits at the very beginning of a neural network.
The Embedding Layer
When a token enters the model, its ID is used to look up a row in a large matrix. This row IS the embedding—a dense vector of learned weights.
Learning Through Training
During training, the embedding weights are adjusted via backpropagation. Words that appear in similar contexts develop similar embeddings.
In LLMs
The embedding layer converts each token ID to a vector. Position information is added separately, and transformer layers turn these static token vectors into contextual representations.
Dedicated Models
Models like text-embedding-3-small or all-MiniLM are trained specifically to produce embeddings useful for similarity search, with contrastive learning objectives.
Embedding Dimensions
Embedding size varies: GPT-2 uses 768d, some large models use much wider hidden vectors, and dedicated embedding models often use 384-1536d. Larger is not always better—it depends on the task.
Common Use Cases
Semantic Search
Find documents by meaning, not just keyword matching.
Clustering
Group similar documents, detect topics automatically.
Classification
Categorize text based on embedding similarity to examples.
RAG Systems
Retrieve relevant context for LLM prompts.
Interactive Visualization
Explore how embeddings cluster by meaning
A geometry exercise with manually chosen 3D vectors, not embeddings from a trained model or a projection of high-dimensional data. The clusters are drawn by design. Cosine values below are calculated from exactly these coordinates.
Select words to visualize
Click words to add/remove from visualization
Vector Space (3D)
Drag to rotate, scroll to zoom. Use the selection below to compare a word without the 3D controls.
Key Takeaways
- 1Embeddings convert text to vectors that capture semantic meaning
- 2Similar concepts have similar embeddings (cosine similarity)
- 3Embeddings enable semantic search, clustering, and RAG
- 4Different embedding models have different strengths and dimensions