Deep learning isn’t hitting a wall—our compute has been lagging. GPT-4 ran on 4-yr-old A100 GPUs (~25k). GPT-5 likely on ~70k H100s. But Blackwell (GB200) GPUs are 8-32x faster. First clusters are just shipping; in ~12-18 months, expect new LLMs trained on these.

Data: @epochai.bsky.social

Hate when this happens

Since GPT-4 came out, open source LLMs have scaled datasets much more than compute, at least from the few data points we have.

Data from EpochAI

I don't like the new Google Gemini model, it feels very stilted and just talks in bullet points constantly. Maybe it is good at some specific tasks, but not as a partner chatbot