This repository serves as a structured, hands-on roadmap to understanding how Large Language Models (LLMs) work, from their mathematical foundations to a working application.
This project is directly inspired by and built around the curriculum of Andrej Karpathy's LLM101 ("Let's build a Storyteller") course. The goal is to build a Storyteller AI — an end-to-end language model that writes (and eventually illustrates) short stories — completely from scratch until we have a working, ChatGPT-style web application.
This is a build-it-yourself guide. We start with simple models that can be simulated on paper and work up to a small but fully functional chat application. Every line of code is written and understood from first principles, with no hidden abstractions or framework magic.
- From Scratch Implementation: We build the autograd engine, the tokenizer, the Transformer architecture, training loops, and the web application interface. We start in Python for clarity and dive into C and CUDA for performance.
- Hands-On Learning: Each chapter includes a detailed lesson, interactive Jupyter notebooks for exploration, targeted exercises, and a structured mini-project with starter code and reference solutions.
- Cohesive Storyteller Theme: The chapters are tied together by a single narrative theme. We build a model that starts by generating character names, progresses to sentences and paragraphs, and finally outputs fully illustrated stories.
This course is designed for individuals who have a solid grasp of basic Python (variables, loops, functions, lists) and want to understand how ChatGPT-like systems operate under the hood. No prior machine learning or advanced calculus experience is required; we build all mathematical intuition step-by-step as we go. This material matches the complexity level of Karpathy's Neural Networks: Zero to Hero series.
Each chapter is contained within its own self-sufficient directory:
chapters/NN-topic/
├── README.md <- Detailed lesson and core theory
├── code/ <- Runnable, heavily commented reference scripts
│ └── explore.ipynb <- Interactive Jupyter notebook to explore concepts
├── exercises/ <- Brief exercises to reinforce the material
│ └── solutions/ <- Worked solutions for reference
└── project/ <- Mini-project to implement yourself
├── starter/ <- Scaffold code with TODOs to complete
└── solution/ <- Reference implementation
Recommended study flow:
- Read the chapter lesson.
- Run and experiment with the interactive Jupyter notebook.
- Complete the exercises.
- Build the project using the starter template, then compare your work to the reference solution.
- Refer to the corresponding Karpathy resources as needed.
Below is the roadmap of chapters. Each chapter is designed to build directly on the concepts established in the previous ones.
| Chapter | Title | Focus Areas | Status |
|---|---|---|---|
| 01 | Bigram Language Model | Language modeling, sampling, the loss function, and the count-neural net equivalence | Ready |
| 02 | Micrograd | Machine learning, backpropagation, and building an autograd engine from scratch | Ready |
| 03 | N-gram Model | Multi-layer perceptrons, token embeddings, matrix multiplications, and GELU activation | Ready |
| 04 | Attention | Attention mechanisms, softmax functions, and positional encoding | Ready |
| 05 | Transformer | The Transformer architecture, residual connections, LayerNorm, and GPT-2 | Ready |
| 06 | Tokenization | minBPE implementation and byte-pair encoding (BPE) | Ready |
| 07 | Optimization | Parameter initialization, optimization techniques, and the AdamW optimizer | Ready |
| 08 | Need for Speed I: Device | Hardware acceleration across CPU, GPU, Apple Silicon, and CUDA devices | Ready |
| 09 | Need for Speed II: Precision | Numerical precision formats, including mixed precision, FP16, BF16, and FP8 | Ready |
| 10 | Need for Speed III: Distributed | Distributed training strategies, including DDP and ZeRO | Ready |
| 11 | Datasets | Data loading pipelines, synthetic data, and the TinyStories dataset | Ready |
| 12 | Inference I: KV-cache | Implementing the Key-Value (KV) cache for fast autoregressive inference | Ready |
| 13 | Inference II: Quantization | Model quantization methods, including INT8, GPTQ, and AWQ | Ready |
| 14 | Finetuning I: SFT | Supervised Finetuning (SFT), PEFT, LoRA, and chat-based models | Ready |
| 15 | Finetuning II: RL | Reinforcement Learning from Human Feedback (RLHF), PPO, and DPO | Ready |
| 16 | Deployment | Building REST APIs and deploying a functional web application | Ready |
| 17 | Multimodal | Vision-Language models, VQVAE, Diffusion Transformers, and story illustration | Ready |
| — | Appendix | Tensor operations, data types, C and assembly primers, and math refreshers | Ready |
Our ultimate goal mirrors the architecture of Karpathy's nanochat: building a complete pipeline from tokenization, pretraining, finetuning, and RL, to KV-cache inference and a web interface. The final model is trained on the TinyStories dataset. This dataset allows sub-10M parameter models to produce coherent, grammatically correct stories, enabling full training and experimentation on modest consumer hardware.
To get started, you will need Python 3.10+ (Python 3.13 is recommended) and git installed on your system.
# 1. Clone the repository
git clone https://github.com/ImAlno/Understanding-LLMs.git
cd Understanding-LLMs
# 2. Create a virtual environment and install dependencies
# Option A: Using uv (fast package manager)
uv venv --python 3.13 .venv
uv pip install -r requirements.txt
# Option B: Using standard python venv and pip
# python3 -m venv .venv
# source .venv/bin/activate # On Windows: .venv\Scripts\activate
# pip install -r requirements.txt
# 3. Verify the installation by training a simple bigram model
.venv/bin/python chapters/01-bigram/code/bigram_counts.pyIf the verification script successfully runs, prints sampled name outputs, and outputs a loss value of approximately 2.45, your environment is correctly configured.
Note: If you do not have a dedicated GPU, you can still run chapters 1 through 7 on a standard laptop CPU. A GPU is only required starting from the hardware acceleration and speed optimization chapters, where instructions for using free cloud GPU instances (like Google Colab) are also provided.
- resources/videos.md: Mappings from each chapter to the relevant video lectures and repositories in Karpathy's Zero to Hero series.
- resources/papers.md: A collection of foundational papers and friendly explanations recommended for each topic.
This repository is deeply indebted to Andrej Karpathy's open-source educational work, including Neural Networks: Zero to Hero, makemore, micrograd, nanoGPT, minbpe, and nanochat. Specifically, this project is a fully-realized, end-to-end implementation and expansion of Karpathy's archived LLM101n "Let's build a Storyteller" syllabus.
We also acknowledge the authors of the foundational research papers referenced throughout the chapters, whose work makes this educational content possible.
The written lessons and custom code implementations in this repository are provided for educational purposes. Please consult the individual resource links (such as external datasets or papers) for their respective licenses.
Ready to start? Begin with Chapter 1: The Bigram Language Model