(If you're interested, a truly random set of 50 numbers might look like this.)

/end

The most interesting result comes from Deepseek R1, where the internal monologue shows that it *knows* it isn't doing it right.

After an agonizing debate with itself and a lot of heartrending effort at making it more random it eventually ends up with:

3/N

On the other hand, if you're an LLM, apparently the distribution will look like this, with each digit appearing *exactly* as often as the others.

(This isn't random at all!)

I get identical results for ChatGPT, Claude and Deepseek. 2/N

Want a test to check if you're talking to a human or an AI?

Ask for a list of 50 random numbers between 0 and 9. Don't let it use any code.

If you're talking to a human (e.g. me) the distribution of numbers might look like this:

1/N

in a first for me, i'll be at neurips this year (!) where we have a paper on ml and duality at the machine learning for physical sciences workshop.

come and say hi if you're there!

ml4physicalsciences.github.io/2024/files/N...