What is Training?
Training is the process of teaching a neural network to perform a task by exposing it to examples and adjusting its parameters based on errors.
Training Phases
The stages of training a neural network.
Initialization
Set random starting weights. Good initialization helps training.
Forward Pass
Input flows through the network to produce predictions.
Loss Calculation
Compare predictions to ground truth with a loss function.
Backpropagation
Compute gradients of loss with respect to each weight.
Optimization
Update weights using gradient descent.
Key Concepts
Epoch
One complete pass through the entire training dataset.
Batch Size
Number of examples processed before updating weights.
Overfitting
Model memorizes training data but fails on new data.
Regularization
Techniques to prevent overfitting (dropout, weight decay).
Training Progress Visualizer
Train one linear neuron, inspect its predictions and compare training and validation error.
Train a linear neuron
This browser really updates w and b by gradient descent on mean squared error: prediction = w·x + b. Blue points are training data; orange points are held-out validation data. Their targets and the initial weights are fixed, so runs are comparable.
A single linear neuron cannot fit every noisy point. Validation error may change differently from training error; there is no universal epoch at which overfitting begins. Try a strong L2 penalty to see underfitting.
Key Takeaways
- 1Training iteratively reduces prediction errors
- 2Evaluate on held-out data to distinguish fitting the training set from generalizing.
- 3Batch size and learning rate significantly affect training
- 4Modern LLMs require massive compute for training