HackerTube.runGood videos, with context.
Makemore tutorial: your first language-model baseline
HackerTube viewing guide · Source-checked November 2, 2026
Before you press Play
Choose this before attention because the count-table model makes training data, sampling and likelihood visible without a Transformer hiding them. It gives you a baseline against which a more complex model has to earn its place.
Micrograd opens the differentiation engine; makemore opens the modeling loop. This first lecture is a bigram model, not the whole makemore series or a Transformer tutorial.
Work toward these outcomes
- Count adjacent characters and turn those counts into a distribution over the next character.
- Sample names and evaluate average negative log likelihood instead of judging samples alone.
- Compare a count-table baseline with a one-layer neural version of the same bigram task.
Bring these foundations
- Python and basic derivatives; micrograd is a useful previous step.
- Willingness to work through tensor shapes, indexing and broadcasting. The author introduces torch.Tensor here.
A useful readiness check
Try the author’s trigram exercise and evaluate it on held-out data. Keep the data split fixed so that a different sample of generated names does not masquerade as a better model.
Makemore tutorial chapters
Selected landmarks from the creator’s chapter list (1:57:45 total). Choose a timestamp to load the YouTube player at that point; without JavaScript, the link opens YouTube.
- 12:45counting bigrams in a 2D torch tensor ("training the model")
- 24:02sampling from the model
- 36:17efficiency! vectorized normalization of the rows, tensor broadcasting
- 50:14loss function (the negative log likelihood of the data under our model)
- 1:02:57PART 2: the neural network approach: intro
- 1:18:46transforming neural net outputs into probabilities: the softmax
- 1:38:36backward and update, in PyTorch
See every chapter in the creator’s description
Original course files and resources
- Neural Networks: Zero to Hero syllabus
- Lecture notebook: bigrams
- makemore repository
- Author-linked Python / NumPy practice
Between bigrams and GPT
The author recommends the earlier makemore lessons before GPT. These four intermediate lessons are on the original course, rather than separate pages in this archive.
Next in your learning path
About this video
Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.
Why it appears in the archive
HN recommends it as a long but worthwhile way to demystify how language models work technically.
AI-generated summary and labels from the source dataset; verify important details against the video. Source & credits.
From the Hacker News discussion
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. [VIDEO]
Read original comment ↗You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [LINK] - Highly recommend this one [VIDEO] - [LINK] - [LINK] 1: ex: [LINK]
Read original comment ↗This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] [VIDEO]
Read original comment ↗For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [LINK] It appears to be a generalized version of the bi-gram project, which he links to here: * [VIDEO] It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.
Read original comment ↗