and here I was thinking you were out at the Opera 🤯

It's been a joy working with @jeff-cheng.bsky.social & Ben Van Durme on this project. And huge thanks to @alexmartin314.bsky.social, @miriamsw.bsky.social, @marcmarone.com, @orionweller.bsky.social, and everyone else who gave very helpful feedback over the past weeks.

To our knowledge this is the first work to raise this point in the new area of LLM test-time scaling, but the community has been aware of this for a long time. E.g., the Watson effort on Jeopardy, and a push by Jordan Boyd-Graber to reward systems that hold back dubious answers.

We propose the standard evaluation format of “Jeopardy odds”: win a point when you’re right, lose a point when you’re wrong. Here we see compute scaling distinctions that were hidden when evaluating under a zero-risk setting. Selection functions matter!

We test DeepSeek-R1 and find that scaling test-time compute can substantially increase a model’s confidence in correct answers, drawing a wider gap between correct and incorrect answers.

You might look into behavior cloning agents, which is a pretty robust space (e.g. arxiv.org/abs/2209.05451)

I could be misunderstanding what you're looking for though, since this feels very different from the CogAI/SOAR items you point to.

In many ways, the Vision Pro hits on both categories.

At this point, I would probably buy a cellular phone that they made

I think 17th century English were more likely to be enjoying Tea than Coffee

👋

Did you recently visit an Apple store?

I saw this happen live, it was tragic