(fp8)

That's easily at least 10^30 flops

And for all those harping about cost: efficiency frontiers do come down as evident from AI2's OLMo work, that synthetic data seems to largely be helping with learning better abstractions so why wouldn't these absurdly long CoT dumps? Even simple known methods would lead to a 100x decrease!

This obsession with whether it's AGI or not digresses from the very impressive evals perf. The last few times it was immediately clear that something represented a breakthrough was when ResNets did exceedingly well on ILSVRC and transformers did the same with BLEU etc.

Nonetheless I will never cease to be amazed by the fact that I end up with slower, but definitely more than zero progress even when I'm clicking buttons and running things for claude.

For anyone on the more applied side wondering why they should care, I think conformal prediction is going to be instrumental for building good reward models for sparse reward domains, among other (more obvious) things!

I'm reading this compilation of short stories "Contemporary Taiwanese Women Writers: An Anthology."
I found it thanks to a Google search typo and couldn't be more grateful!! It's mesmerizing. ✨

No sane person is questioning the existence of anti-semitism, but it is not anti-semitic (nor Islamophobic for that matter) to call out war crimes and seek an end to violence. #FreePalestine

So proud!!!!!!