Transformers, the tech behind LLMs | Deep Learning Chapter 5

3Blue1Brown is really a godsend. I had been trying and failed to get an intuition for the self-attention mechanisms, and also why you want to stack you transformer layers. I went through a lot of material (Key-Query-Value interpretations, SVD of these, etc.) but his comment in chapter 5 video [1] made it click for me: "But you should think of the primary goal of this network that it (the word embedding) flows through as being to enable each one of those vectors to soak up a meaning that's much more rich and specific than what mere individual words could represent". Here is my intuition: Transformers are merely a Convnet for token sequences. In the sense that information is first local and each data means something wrt its immediate neighbors. You extract data locally, and then you dezoom to and redo exactly the same at a higher level. Convnet have been shown to reproduce edge detecting kernels first (vertical edge), then assemble them into basic shapes (vertical+oblique: corner), then bigger shapes (something round), then link shapes together (round+round separated by a horizontal shape with rectangles on top), and then it's a car! [2] Transformers do the same. They assemble meaning among tokens ("ju" + "mps" : a conjugated verb), then assemble these at a higher level ("cat jumps" -> noun+verb), then it's a sentence "the cat jumps over the wall", then a paragraph, then a chapter, then a book. All differentiable so any deviation from something coherent can tweak its knobs high

3Blue1Brown has a fantastic video on GPT that I'd highly recommend called "But what is a GPT? Visual intro to transformers" [1]. You'll need to have some background in mathematical reasoning, but not too much. He does a great job of drawing pictures to show the mathematical transformations that happen under the hood. [1] [VIDEO]

This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [VIDEO] 1. [LINK]

This video explains the direction-encodes-trait topic very well IMO: [VIDEO] It’s the first in a series of three that I can very highly recommend. > there's no way that there's a 1-to-1 correspondence between concepts and dimensions. I don’t know about that! Once you go very high dimensional, there is a lot of direction vectors that are almost perfectly perpendicular to each other (meaning they can cleanly encode a trait). Maybe they don’t even need to be perfectly perpendicular, the dot product just needs to be very close to zero.

I really recommend watching this section of this video. Embeddings do encode plenty of "human knowledge" into the vector values and their relations to each other. [VIDEO] This even opens up a more data-based approach to linguistics, where it is also heavily used.

3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs. It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop. [VIDEO]

To quote GPT-4, "AI's 'creativity' is more about remixing and reinterpreting the information it has been fed". I think the degree to which a person perceives human-like creativity from AI/LLMs is inversely proportional to their understanding of how they work. Here's a great video that can help demystify GPTs: [VIDEO]

There are many others that are better. 1/ The Annotated Transformer Attention is All You Need [LINK] 2/ Transformers from Scratch [LINK] 3/ Andrej Karpathy has really good series of intros: [LINK] Let's build GPT: from scratch, in code, spelled out. [LINK] GPT with Andrej Karpathy: Part 1 [LINK] 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning [VIDEO] Attention in transformers, visually explained | Chapter 6, Deep Learning [LINK] Full 3Blue1Brown Neural Networks playlist [LINK]