What are Embeddings?
Embeddings are dense vector representations that capture semantic meaning. Similar concepts have similar embeddings, enabling machines to understand relationships between words, sentences, and documents.
How Embeddings Work
Embedding models map discrete tokens to continuous vectors in a high-dimensional space (often 768-4096 dimensions). The position of each vector encodes its semantic meaning.
Semantic Similarity
Similar meanings cluster together in embedding space. "King" and "Queen" are closer than "King" and "Banana".
Vector Dimensions
Each dimension captures some aspect of meaning—though these dimensions aren't human-interpretable.
Vector Operations
Famous example: King - Man + Woman ≈ Queen. Relationships are encoded as directions in the space.
How Embeddings Are Created
Embeddings come from the embedding layer—a learned lookup table that sits at the very beginning of a neural network.
The Embedding Layer
When a token enters the model, its ID is used to look up a row in a large matrix. This row IS the embedding—a dense vector of learned weights.
Learning Through Training
During training, the embedding weights are adjusted via backpropagation. Words that appear in similar contexts develop similar embeddings.
In LLMs
The embedding layer converts each token ID to a vector. These vectors are then processed through transformer layers, combined with positional encodings to understand word order.
Dedicated Models
Models like text-embedding-3-small or all-MiniLM are trained specifically to produce embeddings useful for similarity search, with contrastive learning objectives.
Embedding Dimensions
Embedding size varies: GPT-2 uses 768d, GPT-4 uses 12,288d, dedicated embedding models often use 384-1536d. Larger isn't always better—it depends on the task.
Common Use Cases
Semantic Search
Find documents by meaning, not just keyword matching.
Clustering
Group similar documents, detect topics automatically.
Classification
Categorize text based on embedding similarity to examples.
RAG Systems
Retrieve relevant context for LLM prompts.
Interactive Visualization
Explore how embeddings cluster by meaning
Select words to visualize
Click words to add/remove from visualization
Vector Space (2D Projection)
2D projection of embedding space
Key Takeaways
- 1Embeddings convert text to vectors that capture semantic meaning
- 2Similar concepts have similar embeddings (cosine similarity)
- 3Embeddings enable semantic search, clustering, and RAG
- 4Different embedding models have different strengths and dimensions