Ah! I like that example!

It’s effectively pointing out that we might be particularly prone to do implicit meta studies across our own works (inferring V as a particular important control from several papers). Fixed seeds can bias conclusions from those.

I can very much see the practical implication *within* a project. Across projects it feels less clear. Picking a different seed introduces random variation in results (conclusions) across projects which does not seem particularly useful

Maybe an argument is to make projects “meta-analysis-friendly”

I’m curious: How would you articulate the problem with using the same seed across different projects?

Interesting study but worth noting that “correlation is not causality” applies here. This shows a correlation between political beliefs and conclusions, it does NOT show that political beliefs affect conclusions.

Jeg ville jo nok insistere på at det første er lidt et definitionsspørgsmål :)

Tak for oplysning. Nu er jeg med igen (på både substans og sprogbrug)

Ja, mindre samlet trafik ved road pricing, men også en modsatrettet effekt fra af at nye veje nu giver indtægter fra roadpricebetalinger.

Men det sidste du nævner lyder som om den gængse nomenklatur er at se dette som en sænkning af nettoomkostningerne (selvfinansiering) frem for højere afkast?

Hvis man regnede helt rigtig, burde vejprojekter så ikke se bedre ud med optimal Road pricing? Road pricing løser et eksternalitetsoroblem som sikrer optimal udnyttelse af vejnettet = større samf. afkast

Yes, nu er jeg med. Så den overordnet gode policy er congestion pricing plus vejudvidelse indtil omkostning>=gevinst i sparet rejsetid.

Jeg er nysgerrig om jeg er bagud på min viden her, men jeg troede: 1) udvidelse af transport-infrastruktur løser ikke trængselsproblemer (eksternalitetsomkostninger), 2) congestion road pricing gør. Er det ikke det der står i tweetet? (Plus noget tvivlsomt om kollektiv transport)

TL;DR: The novel data is the universe of human written text. The LLM is just a way to process this data.

I share your skepticism but there is a coherent argument for why this could work:

An LLM is (approx) trained on the universe of text to behave like humans do.

To someone who has read and understood the universe of text; the LLM will never surprise. But none of us have!

Let’s just say my enthusiasm for accepting referee tasks dropped for a while after.

I’m sure I have lots of bad (subjective) opinions as reviewer. But this was not that.

Later I got the revised version to review. The authors had BEAUTIFULLY addressed 1…

… but also doubled down on the bias claim, STILL without properly defining an estimand. 🤯

Situation was weird after that. Eventually ended with editor half-agreeing with both of us and accepting then paper. 🤷‍♂️

N/N

… the paper never defined the estimand! And in fact for a sensible choice of estimand, there is NO bias in the existing method.

I wrote a positive referee report detailing how I would fix 1. Plus suggesting that claim 2. be taken out (the paper’s contribution was easily high enough without it)

2/N

I’ll keep this vague since I hold no ill will here. But basically:

I refereed a really good paper with only two drawbacks:

1. The exposition/framing was doing the results a disservice (IMO)

2. The paper claimed to fix a bias in an existing method that I have worked on extensively, except…

1/N