A ManyModels project would be a valuable contribution. Almost all published literature using LLMs is out of date by time of peer-review; maybe a living report to show 'current' state of LLMs? What's the experimental design?

Fantastic, thanks for doing this and making it available!

Thanks Nick! Much appreciated :)

Add me please :) 👋

this may end up becoming how evidence synthesis is done in the future (with teams of AI agents collaborating on each step of the process and continuously updating reviews/meta-analyses when new evidence becomes available)

Challenging indeed! Regression based models will only get us so far. I think formal models (eg ordinary differential equations) are needed at the idiographic level. I'm hopeful that personal AI agents will be developed to do this type of continual data collection and modeling in the future.