A tokenizer-free transformer architecture proposed by Meta makes a lot of sense. More efficient compute allocation and increased flexibility for complex use cases
ai.meta.com/research/pub...

TIL: weight tying improves performance by sharing(tying) the embedding and softmax matrixes as they both can be represented by vectors with low cosine distance between each other.
A comparison:

TIL that this site exists: paperswithcode.com/method/layer...
Concise explanations and both paper and code available on a single place for deeper understanding. Super useful!

TIL: GELU activation function is usually preferred over a simpler ReLU because it's smoother, allowing for small, yet non-zero output for negative values in the input tensor. It means that neurons with negative values will contribute to the learning process, albeit at a lesser extent than positives.

TIL: ReLU is a simple activation function that swaps every negative value in the input tensor with zeroes. Activation functions like this introduce non-linearity to neural networks, forcing them to learn more complex patterns.

TIL: Multi-headed attention (MHA) is a technique to make the model perform self-attention over a longer dataset context in parallel and then combine results. You can either 1) stack multiple single-head attention layers or ->

What if an LLM receives a word out of its vocabulary?
GPT-like models use a byte-pair encoder that breaks words into subwords and even single characters to handle unknown words without introducing special tokens.

TIL: padding is a technique (a trick?) to align text lengths when training LLMs with batch sizes > 1. Shorter texts are extended ("padded") up to the length of the longest text in the batch.