I appreciate your overall point, but just to push back on "this is such a new field" -- AI as a field has been around for at least 70 years, and universities have been training AI researchers for most of that time.

This is not to say that such "role-playing" can't be dangerous in and of itself. In fact, role-playing is a key method for AI "jail-breaking". But that's not the same thing as a "survival drive".

It's a difficult and uncertain time for science in the U.S. and worldwide, so communicating the ideas and results of science to the general public has never been more essential.

More about these awards: www.nationalacademies.org/news/2025/10...

Half the authors might be hallucinations.

Lol. Who among us hasn't hallucinated in the course of a Google Docs ➡️ LaTeX migration?

This one too? URL links to paper with similar-sounding title, some different authors, different journal. Title in this reference does not seem to exist.

Thanks, I will take a look.

Megan, That's wonderful -- congratulations!!

Evaluation of reasoning, and reasoning about evaluation -- both understudied, imo

Excellent statement.

On the other hand, accuracy alone may be *underestimating* this ability in visual settings

It is essential to go beyond accuracy in evaluating such capabilities!

Paper: arxiv.org/abs/2510.02125

Blog post: aiguide.substack.com/p/do-ai-reas...

🧵 10/10

Conclusions: Evaluations like those of the ARC Prize, using accuracy alone, may be *overestimating* abstract reasoning ability of these models in textual setting.

🧵 9/10

With visual inputs, these models all do quite poorly on generating accurate grids. But they do manage to get the correct-intended rule considerably more often than they generate the correct grid.

🧵 8/10