Language Models From First Principles
Understand how large language models work by building the pieces yourself: backpropagation, n-gram and neural language models, embeddings, a BPE tokenizer, attention and a small GPT, with guided notes, timestamped lesson outlines and checks for every lesson.
Created and taught by Bhim Prasad Adhikari
8
Lessons
11h
Learning
30
Checks
Why this path
Understanding before implementation
Most people use LLMs through an API and never see what happens inside. This path takes you inside, one working piece at a time, without skipping the maths or the code. The course is built around the best free lectures on the subject: Andrej Karpathy's "Neural Networks: Zero to Hero" series, 3Blue1Brown's deep-learning chapters on transformers, and StatQuest's explanation of word embeddings. For each video you get a timestamped outline you can click to jump to any section, a study guide (mental model, key ideas, common misconceptions, recall prompts and an explain-it-back exercise) and a quiz that checks you understood the ideas, not just the vocabulary. How the path is ordered: 1. Start with the whole machine: what a transformer computes and how it generates text. 2. Build backpropagation from scratch, so training is never a black box. 3. Build language models: a bigram count table, the same model as a neural network, then learned embeddings (Bengio et al., 2003) and word2vec. 4. Build a GPT-style byte-level BPE tokenizer and see why tokenization explains many LLM quirks. 5. Understand attention visually, then implement a complete small GPT in PyTorch. Total watch time is about 11 hours. Plan for roughly twice that if you code along, which we strongly recommend: every Karpathy lecture links to its notebook or repository. Companion reading on this site: the "From Shannon to Transformers" blog series covers the history behind lessons 3 and 4, starting with Shannon's 1948 n-gram experiments and his 1951 guessing game. All videos are embedded from their creators' official YouTube channels. Credit and copyright for the lectures belong to Andrej Karpathy, Grant Sanderson (3Blue1Brown) and Josh Starmer (StatQuest).
What you will understand
- Explain, step by step, how a transformer turns a prompt into a probability distribution over the next token and generates text from it.
- Implement backpropagation from scratch and explain what PyTorch's autograd is doing for you.
- Build and evaluate n-gram and neural language models, and measure them with cross-entropy (negative log-likelihood).
- Explain learned embeddings and why they let models generalize beyond exact contexts seen in training.
- Implement byte-level Byte-Pair Encoding and predict how text will be split into tokens.
- Compute scaled dot-product attention with causal masking, and explain multi-head attention.
- Build and train a small decoder-only transformer (GPT) in PyTorch.
- Diagnose common problems: overfitting, bad learning rates, forgotten zero_grad, broadcasting bugs, tokenization quirks.
Before you begin
- • Comfortable writing Python: functions, classes, lists and dictionaries.
- • High-school algebra. Knowing what a derivative is helps, and lesson 2 rebuilds it from scratch.
- • Basic familiarity with vectors and matrices (what a matrix multiplication does).
- • A laptop with Python, or a free Google Colab account to run the notebooks.
Curriculum
A guided sequence of ideas
Module 1
Start Here: The Whole Machine
Get the full picture before taking it apart. What a language model computes, how it generates text, and where each later lesson fits.
Module 2
Neural Network Building Blocks
Derivatives, the chain rule and backpropagation, built by hand in a tiny autograd engine, so nothing in later lessons is a black box.
Module 3
From Counting to Neural Language Models
Go from a count table (Shannon's n-grams) to a neural network that learns the same table, then to learned embeddings that let a model generalize to contexts it has never seen.
Module 4
Tokenization: How Text Becomes Numbers
Build the byte-level BPE tokenizer used by GPT models and see why many LLM quirks are really tokenizer quirks.
Module 5
Attention and the Transformer
Understand self-attention visually, then implement a complete decoder-only transformer (a small GPT) from scratch in PyTorch.