Long way ahead 😅

Even better - (I view this as a duality) that when we multiply a matrix by a vector, we're essentially translating that vector into the coordinate system defined by our matrix's basis vectors!

🧵I’m still not 100% confident about this but this is where I currently stand.

🧵(2) Even though simply Transformers is made to predict the next token as it scales it seems possible that it will continue to reliably predict the next proper token but this time tokens form the correct proofs or code.

🧵There are two reasons that made me switch: (1) The current performance improvement to Maths (AlphaGeometry) and Coding (SWE-bench) just through scaling and improving the current paradigm alone.

🧵But through my quick search before tweeting I come to the opposite conclusion.

🧵My thinking is it wasn’t natural for the Transformers to learn to reason. Unlike CNN architecture to learn classification tasks. And transformers to translate or generate more words.