xjdr
@xjdr.bsky.social
hot takes, linear Algebra, JAX apologist, Raconteur
I have become radicalized
November 26, 2024 at 5:27 PM
I have become radicalized
now that people are paying attention again, here is your periodic reminder. Always run in bf16. always apply ROPE and attention softmax at float32 (as shown here)
github.com/xjdr-alt/ent...
github.com/xjdr-alt/ent...
November 24, 2024 at 5:23 PM
now that people are paying attention again, here is your periodic reminder. Always run in bf16. always apply ROPE and attention softmax at float32 (as shown here)
github.com/xjdr-alt/ent...
github.com/xjdr-alt/ent...