Can you pls expand on how social phenomena resist quantification?

Yes, I think that this may lead to better scores, even though I expected o1 to make some reasoning "path" that would be vaguely similar to what you propose. Maybe "enforcing" this with an agentic flow may work

From looking at the log, I think that one of the reasons may be that it tries to pay too much attention to the "variables" in the prompt, without "thinking out of the box" (which, in this case, is actually a simple solution that most people with knowledge in the field would do)

Very interesting paper! I was wondering though: what do you think about figure 1? Is the last "verify" stage really needed, or can we replace it with e.g., well-known non-parametric hypothesis tests on 30 runs? This may avoid running too many experiments just to know that two methods are comparable