Just as human chemists learn through diverse materials and experiences—textbooks, laboratory work, research papers, and problem-solving —ChemPile’s varied content types aim to provide a comprehensive learning
arXiv: arxiv.org/pdf/2505.12534
read more: chempile.lamalab.org

We introduce the ChemPile, the largest natural language chemistry dataset (>75B tokens).
dataset: huggingface.co/collections/...

 Not sure where to start? Our documentation has step-by-step guides for every scenario
lamalab-org.github.io/chembench/

✨Public Datasets & Leaderboard – All datasets are live on HuggingFace, alongside a real-time performance leaderboard! huggingface.co/datasets/jab...

What's new?
✨Multimodal Support – Handle text, data, and chemistry-specific inputs seamlessly
✨Redesigned API – Now standardized on LiteLLM messages for effortless integration
✨Custom System Prompts – Tailor benchmarks to your unique use case

🌟LLM limitations persist: Still lagging in 3D molecular spatial reasoning
#LLMs #MachineLearning #OpenScience

🌟System prompt insights: Ablation studies show no effect on evaluation outcomes
🌟VLLMs dominate: Outperform specialized models like Decimer in benchmarks

Supported by Carl Zeiss Foundation, Intel, Merck, Alexander von Humboldt Foundation, Friedrich-Schiller-Universität Jena, IIT Delhi.

📜Manuscript: arxiv.org/abs/2411.16955
👩‍💻GitHub: github.com/lamalab-org/...

For instance, one would expect vision models to perform very well and better than text models on spatial reasoning - such as identifying the correct isomeric relation between two compounds.

But this is not the case!

But we did not stop there! We dug deeper with ablations to understand the bottlenecks in applicability.
We compared different modalities, multi-step vs single step reasoning, guided prompting, etc.

We observed a striking disparity in performance across tasks. Models can identify lab equipment but struggle with identifying safety violations in real-life laboratory scenarios.

We and M3RG-Group from IIT Delhi created MaCBench: a multimodal materials and chemistry benchmark. (2137 questions)

We focus on the tasks we consider crucial for scientific development, practical lab scenarios, Spectral Analysis, US patents, and more.