Sebastian Joseph
@sebajoe.bsky.social
CS Ph.D. Student at UT Austin
My amazing co-authors: Syed Murtaza Husain, Stella Offner, @stephajuneau.bsky.social, Paul Torrey, Adam Bolton, Juan Frias, @niall2.bsky.social, @gregdnlp.bsky.social, and @jessyjli.bsky.social.
Full support from @nsfsimonscosmicai.bsky.social.
🌐: astrovisbench.github.io
📄: arxiv.org/abs/2505.20538
Full support from @nsfsimonscosmicai.bsky.social.
🌐: astrovisbench.github.io
📄: arxiv.org/abs/2505.20538
AstroVisbench · AstroVisBench
astrovisbench.github.io
June 2, 2025 at 3:42 PM
My amazing co-authors: Syed Murtaza Husain, Stella Offner, @stephajuneau.bsky.social, Paul Torrey, Adam Bolton, Juan Frias, @niall2.bsky.social, @gregdnlp.bsky.social, and @jessyjli.bsky.social.
Full support from @nsfsimonscosmicai.bsky.social.
🌐: astrovisbench.github.io
📄: arxiv.org/abs/2505.20538
Full support from @nsfsimonscosmicai.bsky.social.
🌐: astrovisbench.github.io
📄: arxiv.org/abs/2505.20538
We think this dataset is a great target for AI for science efforts. It zeroes in on an important part of the scientific workflow that is achievable near term and aims to produce tools used by astronomers, not aiming to replace them or automate all of science.
June 2, 2025 at 3:42 PM
We think this dataset is a great target for AI for science efforts. It zeroes in on an important part of the scientific workflow that is achievable near term and aims to produce tools used by astronomers, not aiming to replace them or automate all of science.
Even the best LLMs struggle to execute scientific workflows.
SOTA models including Gemini 2.5 Pro, Claude Opus 4, o3-mini and QwQ crash 30-60% of the time and only produce visualizations without error in less than 16% of the cases.
SOTA models including Gemini 2.5 Pro, Claude Opus 4, o3-mini and QwQ crash 30-60% of the time and only produce visualizations without error in less than 16% of the cases.
June 2, 2025 at 3:42 PM
Even the best LLMs struggle to execute scientific workflows.
SOTA models including Gemini 2.5 Pro, Claude Opus 4, o3-mini and QwQ crash 30-60% of the time and only produce visualizations without error in less than 16% of the cases.
SOTA models including Gemini 2.5 Pro, Claude Opus 4, o3-mini and QwQ crash 30-60% of the time and only produce visualizations without error in less than 16% of the cases.
We generate code from a model, run it, and evaluate the following:
Processing tasks: we compare key variable values.
Visualizations: we use a VLM judge (well correlated w/ pro astronomers) that compares a visualization’s scientific utility to that of the ground truth.
Processing tasks: we compare key variable values.
Visualizations: we use a VLM judge (well correlated w/ pro astronomers) that compares a visualization’s scientific utility to that of the ground truth.
June 2, 2025 at 3:42 PM
We generate code from a model, run it, and evaluate the following:
Processing tasks: we compare key variable values.
Visualizations: we use a VLM judge (well correlated w/ pro astronomers) that compares a visualization’s scientific utility to that of the ground truth.
Processing tasks: we compare key variable values.
Visualizations: we use a VLM judge (well correlated w/ pro astronomers) that compares a visualization’s scientific utility to that of the ground truth.
We created AstroVisBench from expert-curated jupyter notebooks for astronomy tasks, from which we constructed 432 sets of processing and plotting tasks. It tests a diverse set of visualizations and long-tail API use.
June 2, 2025 at 3:42 PM
We created AstroVisBench from expert-curated jupyter notebooks for astronomy tasks, from which we constructed 432 sets of processing and plotting tasks. It tests a diverse set of visualizations and long-tail API use.