From Attention Is All You Need (2017) to the frontier — black and white, one step at a time.

Roadmap

  1. 01 · The bottleneck — why RNNs had to go, and the input: tokens → embeddings
  2. 02 · Positional encoding — giving the network a sense of order
  3. 03 · Attention — Q, K, V and the dot product
  4. 04 · Multi-head attention and the full encoder block
  5. 05 · Feed-forward layers, residuals, layer norm
  6. 06 · The decoder — masking and cross-attention
  7. 07 · The complete 2017 architecture, wired end to end
  8. 08 · Training — next-token prediction, loss, BPE
  9. 09 · 2018: GPT-1, BERT, and the transfer-learning takeover
  10. 10 · Scaling: GPT-2 → GPT-3, scaling laws, emergent abilities
  11. 11 · InstructGPT and RLHF — aligning models to instructions
  12. 12 · The frontier — MoE, long context, reasoning, multimodal

Steps are published one at a time. Newest posts appear below.