CopyBench (EMNLP 2024, led by @tomchen0112.bsky.social)
Oral at regulatableml.github.io & Poster at redteaming-gen-ai.github.io

tldr: We benchmarked LLMs' literal/non-literal copying of copyrighted content—risks found even in 8B models.

Detais: www.arxiv.org/abs/2407.07087

MassiveDS (led by @rulinshao.bsky.social) Wednesday Poster at 11-2 pm at West Ballroom#7203

TLDR: We demonstrated scaling retrieval corpora of Retrieval-Augmented LMs to 1.4T helps & achieves more compute-optimal scaling

Details: retrievalscaling.github.io

Oh that's a screenshot of my website. Here's link to my CV akariasai.github.io/assets/pdf/a...

I would love to hear about any opportunities that might be a good fit!! You can find my contact info and CV on my website. akariasai.github.io. I am attending NeurIPS in person so let’s chat!

🏆 Recognition & Impact: My work has earned EECS Rising Stars 2022, the MIT Tech Review Innovator Award (Japan 2024), paper awards at ACL & NeurIPS, and the IBM Fellowship. My work has been featured in medias like MIT Tech Review, Forbes and VentureBeat.

🌍 Making Real-World Impacts
Retrieval-Augmented LMs tackle critical challenges like:
1️⃣ Unreliable LMs in expert domains
2️⃣ Information access inequity across languages
I launched OpenScholar for scientific synthesis—20k+ demo requests in week 1! Details: allenai.org/blog/opensch...

🛠 Building the Foundations:
Retrieval-augmented LMs need more than off-the-shelf models. I developed advanced training/inference algorithms & architectures, including Self-RAG (ICLR 2024 Oral; NeurIPS Workshop Hon. Mention) for adaptive retrieval & self-critique.
Learn more:
selfrag.github.io

🔍 Establishing the Necessity
My work showed that scaling LLMs alone doesn’t solve issues like hallucinations or obsolete knowledge and is compute suboptimal, and Retrieval-Augmented LMs address these challenges. See our ACL 2023 Best Video Award paper:
aclanthology.org/2023.acl-lon...

8/ ❤️Acknowledgements:
OpenScholar is the result of a collaborative effort UW, Ai2 and many others!
Huge thanks to our incredible team including experts from CS, Bio, and physics, for making this possible!
We’d love your feedback! Reply or email us with questions, ideas, or use cases✨

8/ 🧪 Summary
Try it out: openscholar.allen.ai
Read more: allenai.org/blog/opensch... – we discuss more details as well as limitations of OpenScholar, based on our beta testing with CS researchers!
Code & data: github.com/AkariAsai/Op...
Paper: openscholar.allen.ai/paper

7/ 🌐 What’s next?
We're just getting started with OpenScholar! 🚀
Expanding domains: Support for non-CS fields is coming soon. Public API: Full-text search over 45M+ papers will be available shortly.
Try the OpenScholar demo and share your feedback!
openscholar.allen.ai

6/ 💾 Open Access [2]:
📂 OpenScholar Datastore (45M+ papers up to 2024/10): huggingface.co/datasets/Ope...
📊 ScholarQABench: github.com/AkariAsai/Sc...
👩‍🔬 Human evaluation interface: github.com/AkariAsai/Op...

6/ 💾 Open Access [1]:
Prior work in this area has relied on proprietary LMs and/or released only a subset of datastore
We're releasing
Demo: openscholar.allen.ai
🔓 Code & model checkpoints:
github.com/AkariAsai/Op...
huggingface.co/collections/...

5/ 📊 Exert Evaluation Results:
We further conduct expert evaluations with scientists across CS, Bio and Physics, comparing OS against expert answers.
Scientists preferred OpenScholar-8B outputs compared to human-written answers in majority of the times, thanks to its coverage

5/ 📊 Automatic Results:
So how good OpenScholar?
On ScholarBench, OpenScholar-8B surpassed GPT-4o, concurrent PaperQA2, and other models in factuality & citation accuracy despite being many times cheaper!

4/ 🧪New dataset: ScholarBench
A benchmark for evaluating scientific language models on real-world, open-ended questions requiring synthesis across multiple papers. 🌟
📚 7 datasets across four scientific disciplines
🧑‍🔬 2,000+ expert-annotated question and 200 answers
📊 Automated metrics