Thank you for having me 🙌

3️⃣ An older version – Llama 3.3 70B-Versatile – performed even better than Llama 4 Maverick.

The benchmark – designed by the
@rootly.com AI Labs – tests models' ability to pick the correct pull request for a given bug description. The full findings 👉 rootly.com/blog/llama-4...

2️⃣ Second, we wanted to test it against models tailored for coding tasks. Unsurprisingly, it performs way under those. Llama 4 Maverick achieved only a 70% accuracy score. Alibaba’s Qwen2.5-Coder-32B is ranking the best at (90%), closely followed by GPT o3-mini (89%).

1️⃣ First, we wanted to reproduce Meta's findings that Llama 4 outperformed GPT-4o, Gemini 2.0 Flash, and DeepSeek v3.1—we found the exact opposite.

It came last, 6% less than the next best-performing model (DeepSeek) and 18% behind the overall top-performing model (GPT-4o).

☕️ Or just meet for a coffee; DM me 😊

🎤 Interview guests wanted: speak about your favorite AI tool.
scheduler.default.com/7992/member/...

🔭 Join our Code to Clarity event: The Future of Monitoring, Observability, and Reliability with our friends at Checkly & @coralogix.bsky.social
lu.ma/fhl522f4

📧 We are hiring across the board and are looking for contractors for the AI Lab – shoot me a DM if you are interested!

💡 The AI Lab mission is to leverage AI to improve incident management and systems operations. We’ll be building POCs, open-sourcing tools, and benchmarking models.

👨‍💻Joinly Rootly feels like the perfect next step. My career has always been about SREs—I worked as one, trained them, and helped startups engage with them.