31 Topics

Large Language Models

Understand the inner workings of large language models, from tokenization to attention mechanisms.

Help Make This Better

This guide is open source. Got an idea for a new topic? Found a bug? Want to improve an explanation? Every contribution helps.

Request a Topic Report a Bug Star on GitHub

Fundamentals

TokenizationHow text gets broken into pieces the model understandsSep 13, 2026

B

Subtoken BlindnessWhy models miss letters, digits, and exact counts hidden inside tokensSep 13, 2026

I

EmbeddingsTurning words into numbers that capture meaningSep 13, 2026

B

Positional Encoding & RoPEHow LLMs represent token order with positional encodings, RoPE, and long-context tradeoffs.Jul 12, 2026

E

Attention MechanismHow transformers decide which tokens matter for each predictionSep 13, 2026

I

Multi-Head Attention, MQA & GQAHow attention heads run learned projections in parallel, and how MQA/GQA share key-value heads.Jun 12, 2026

Behavior

TemperatureHow logits become sampling probabilities.Sep 13, 2026

B

Reasoning Models & Inference-Time ComputeHow models spend extra runtime compute on search, candidate generation, and verification — and when it pays off.Jul 12, 2026

E

Jagged FrontierWhy frontier models solve hard benchmarks and still fail simple-looking tasksSep 13, 2026

I

Context RotEvidence on context length, position and distractors, with clear experimental limits.Sep 13, 2026

Capabilities

RAGAugment LLM answers with retrieved external knowledgeSep 13, 2026

I

Vision & ImagesHow LLMs process and understand images alongside textSep 13, 2026

B

Visual ChallengesWhere vision models fail: counting, spatial reasoning, OCRSep 13, 2026

I

Agentic VisionActive visual investigation through zoom, crop, and code executionSep 13, 2026

E

MultimodalityProcessing text, images, audio, and video in one modelSep 13, 2026

Architecture

Transformer ArchitectureThe foundational architecture behind GPT, BERT, and all modern LLMsSep 13, 2026

I

Looped TransformersExplore how recurrent depth updates a hidden state, reuses model weights, and changes the tradeoffs between memory, compute, latency, and monitorability.Sep 7, 2026

E

Feed-Forward NetworksHow the same MLP transforms each token position; routing is covered in Mixture of Experts.Sep 13, 2026

I

Residual Stream & LayerNormHow residual streams and normalization keep deep transformer stacks trainable.Sep 13, 2026

I

Next-Token PredictionHow logits, softmax, and decoding turn model outputs into text one token at a time.Sep 13, 2026

B

LLM TrainingFrom pretraining on raw text to fine-tuning with human feedbackSep 13, 2026

I

Training DataWhere AI models get their knowledge: legitimate sources, controversies, and synthetic dataSep 13, 2026

I

Mixture of ExpertsRun only a fraction of model parameters per tokenSep 13, 2026

E

QuantizationShrink model size by reducing numerical precisionSep 13, 2026

E

Nested LearningLearning algorithms that operate at multiple levelsSep 13, 2026

E

Multi-Token Prediction (MTP)Train models to predict several future tokens at once instead of only the next tokenSep 13, 2026

E

N-gram EmbeddingsLearned lookup tables for short local token patternsAug 27, 2026

E

DistillationTransfer knowledge from a large model to a smaller oneSep 13, 2026

I

Fine-Tuning & LoRAAdapt large models efficiently by training only tiny low-rank matricesSep 13, 2026

I

AbliterationRemove a model's refusal behavior by ablating a single direction in its residual streamSep 13, 2026

E

Speculative DecodingSpeed up inference by drafting tokens with a smaller modelSep 13, 2026

Back to all topics