Learning Dump:
Worked a lot on Trendspot this weekend. The long break made the infra simpler and a sane CI/CD with a better logging experience. I’m currently interested into spending more time reading on AI and distributed systems with my free time for the rest of the year (I need to kill my ADHD lol).
Good resource found: https://docs.google.com/document/u/1/d/e/2PACX-1vQD8IlBotGdBxp3BnXkSjk8bNZlPV_0EH9ZA6wHd5dNf-BLSiwXUinvgv8ZoBEnNyTCF-chWO30NRw0/pub - A lot of SOTA implementation and guide for AI and visual explanation for distributed systems: https://codexsims.com/
Why we use Softmax ?
Generally when a model gets a logit after initial training or fast forward. There needs to be a probability distribution for each incoming element:
This is what we are talking about, this gave us a clean probability distribution that we can use for further runs going forward. See how the -3 was nicely mapped to a 0.0005, that’s why the softmax is useful in this scenario.
This actually one training pipeline:
We clearly see why we are going toward the loss to make the next training or pass better.
Everything that we need to know for the moment is just that for a next prediction token, the cross entropy of a prediction is defined as:
H = - log Pcorrect
Parameters, Back Propagation and Gradient Descent:
The question is now, how can we minimize the loss ? Which parameter should we tune to make the loss smaller ? What is a parameter even ? Is it a scalar ? Is it a song ? Is it an image ? What are the algorithms ? What is the literature saying ?
OK so let me summarize my full understanding and tell me what you think about it. So when we get a sentence like: The animal was sick so he started walking. Technically we can represent it in an 8(9 is the dot is also a token) * 4 matrix where the information about each token is an embedding of dim4. So if we label them from 0 to 7 and take the embedding from 5, we would like to somehow keep it related to animal. It shouldn't be only independent. For that, we add the concept of QKV(Question, Key, Value) . Which are three additional vectors that allow to bring additional information to a token. The question is the information that we are looking for (This is my guess, but I believe we define a set of question like what is the opposite term of this token, or is there a token related to this one, what is the synonym of that token and those questions are just big learned matrices(Now I don't know if they are trained for each neural network or it's a known thing or constant for neural nets)), Using the Question we compute QKt for each token in the sentence (t going from 0 to 7). This gives us a new one dimensional matrix and the bigger the dot product is, the stronger the token answer our question, for the model to answer our question then, we apply a softmax and this kinda closes the loop with the retrieval of the appropriate value using the appropriate key.
Chat is Goated lol
To implement in pytorch and later CUDA:
single-head attention
↓
scaled dot-product attention
↓
causal masking
↓
multi-head attention
↓
residual connections
↓
MLP
↓
one complete transformer block
↓
stack blocks
↓
train our tiny language model