We introduce the ChemPile, the largest natural language chemistry dataset (>75B tokens).
dataset: huggingface.co/collections/...

Training large language models for chemistry is bottlenecked by one critical problem: there is no unified dataset that connects all chemical domains.

We're excited to present our posters today at the AI4Mat workshop at #ICLR25 #AI4Mat #Singapore

LAMA Lab at ICLR in Singapore!
#iclr2025 #singapore #AI #ML #chemistry #iclr

we're ready for spring! team building is always more fun when it's outside ☀️

Day 1 of the Foundation Models workshop hosted by the ELLIS Winter School!

 Not sure where to start? Our documentation has step-by-step guides for every scenario
lamalab-org.github.io/chembench/

✨Public Datasets & Leaderboard – All datasets are live on HuggingFace, alongside a real-time performance leaderboard! huggingface.co/datasets/jab...

🌟LLM limitations persist: Still lagging in 3D molecular spatial reasoning
#LLMs #MachineLearning #OpenScience

🌟System prompt insights: Ablation studies show no effect on evaluation outcomes
🌟VLLMs dominate: Outperform specialized models like Decimer in benchmarks

🚀Our revised MaCBench paper is now on arxiv! arxiv.org/pdf/2411.16955

Key updates!
🌟Robust reproducibility: 5x experiment runs + error bars for statistical confidence
🌟Full dataset & leaderboard: Now live on HuggingFace with model comparisons huggingface.co/spaces/jablo...

For instance, one would expect vision models to perform very well and better than text models on spatial reasoning - such as identifying the correct isomeric relation between two compounds.

But this is not the case!

But we did not stop there! We dug deeper with ablations to understand the bottlenecks in applicability.
We compared different modalities, multi-step vs single step reasoning, guided prompting, etc.

We observed a striking disparity in performance across tasks. Models can identify lab equipment but struggle with identifying safety violations in real-life laboratory scenarios.

We and M3RG-Group from IIT Delhi created MaCBench: a multimodal materials and chemistry benchmark. (2137 questions)

We focus on the tasks we consider crucial for scientific development, practical lab scenarios, Spectral Analysis, US patents, and more.

Are Vision Language Models ready for scientific research?
🧑‍🔬🧪

We compared leading VLLMs on the three pillars of chemical and material science discovery: data extraction, lab experimentation and data interpretation.
arxiv.org/abs/2411.16955