🚨New blog: The AI Evaluation Chart Crisis 📝

From misleading bar heights to missing error bars, recent model launches have sparked debate on AI evals. In our new blogpost, we dig into what’s broken, why it matters and how they should be presented 👇

evalevalai.com/documentatio...

We are committed to making meaningful progress in machine learning research through open collaboration. Follow this 🧵to stay on top of our research contributions.

Dear computer vision researchers, students & practitioners🔇🔇🔇

Remi Denton & I have written what I consider to be a comprehensive paper on the harms of computer vision systems reported to date & how people have proposed addressing them, from different angles.

PDF: cdn.sanity.io/files/wc2kmx...

The baseline is around 65% accuracy with a simple fine-tuned BERT and ~95% with a LLM like GPT4o-mini.... your mission is to beat it while using the least energy possible!
Check out the dataset here:
huggingface.co/collections/...
And stay tuned for the last task later this week!🔥

Today we’re launching the text challenge of the Frugal AI Challenge for the AI Action Summit: detecting climate misinformation in the media (Press, TV, Radio), sponsored by the French non-profit QuotaClimat.
The goal of the task is to detect climate-based misinformation and to categorize its type 📃

Interesting result, even after you correct for anthropomorphizing language

The key takeaway is that providing information about the training condition (explicitly or implicitly) to an LM makes it only "align" (update the probability distribution) in that condition

www.anthropic.com/research/ali...

📣📣 Wanna be an Area Chair or a Reviewer for @aclmeeting.bsky.social or know someone who would?

Nominations and self-nominations go here 👇

docs.google.com/forms/d/e/1F...

Announcing Global-MMLU - an improved MMLU Open dataset with evaluation coverage across 42 languages.

The result of months of work with the goal of advancing Multilingual LLM evaluation.

Built together with the community and amazing collaborators at Cohere4AI, MILA, MIT, and many more.

Day 2 of smol course and the community is building something here.

👷 If you want to get involved, you can do this:
- read (and star) the repo
- check out our new discord channel
- open a PR to submit an exercise on module 1
- open an issue to improve the course
- review another submission

🧵

If you're potentially interested in transitioning into AI safety research, come collaborate with my team at Anthropic!

Funded fellows program for researchers new to the field here: alignment.anthropic.com/2024/anthrop...

Just FYI because it seems relevant and I've seen it misstated a few times, LAION retrained their dataset and provided a diff to migrate over to the filtered Re-LAION-5B dataset

HuggingFace, like most platforms that handle user content, do checks for CSAM too.

laion.ai/blog/relaion...

We've just launched our AI Benchmarking Hub!
This is a new platform for rigorous, independent evaluations of AI model capabilities, featuring interactive visualizations and in-depth analysis. (1/8)

epoch.ai/blog/introdu...

My deep learning course at the University of Geneva is available on-line. 1000+ slides, ~20h of screen-casts. Full of examples in PyTorch.

fleuret.org/dlc/

And my "Little Book of Deep Learning" is available as a phone-formatted pdf (nearing 700k downloads!)

fleuret.org/lbdl/

I made a Chrome Extension to bring Bluesky comments to any URL :)

Get it here: github.com/joneslloyd/b...

Credits to:

- @emilyliu.me
- @coryzue.com
- @louee.bsky.social

Any feedback and/or PRs are welcome.

I threw this together in 1.5 errors, so expect bugs etc.

Just realized that with the API being opened and all we could make a Bluesky RAG.

Short stories for children to read and listen to in many languages. Written with human imagination and translated, illustrated, and read with the help of Artificial Intelligence.

From the original Italian to English, spanish, French, Japanese, and chinese…

www.storiesottolestelle.com