Desmond Elliott
@delliott.bsky.social
Reposted by Desmond Elliott
📄 [ACL 2025 main] LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (doi.org/10.48550/arX...)
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case...
doi.org
July 18, 2025 at 10:19 AM
📄 [ACL 2025 main] LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (doi.org/10.48550/arX...)
Reposted by Desmond Elliott
Three invited speakers will share their insights at TokShop! Hear from Yuval Pinter @uvp.bsky.social, Desmond Elliott @delliott.bsky.social, and Adrian Łańcuck on cutting-edge tokenization research. Don't miss these keynote presentations! #ICML2025 tokenization-workshop.github.io/speakers
July 16, 2025 at 9:13 PM
Three invited speakers will share their insights at TokShop! Hear from Yuval Pinter @uvp.bsky.social, Desmond Elliott @delliott.bsky.social, and Adrian Łańcuck on cutting-edge tokenization research. Don't miss these keynote presentations! #ICML2025 tokenization-workshop.github.io/speakers
Reposted by Desmond Elliott
It is with great pleasure that I share MAXMINDS 2.0, a new Max Planck program to support scholars in danger of displacement by war or natural disasters, and who have limited access to resources and institutional support.
If you know affected scholars, please share.
www.maxminds.mpg.de
If you know affected scholars, please share.
www.maxminds.mpg.de
MAXMINDS 2.0 Homepage
MAXMINDS 2.0
www.maxminds.mpg.de
July 7, 2025 at 7:30 PM
It is with great pleasure that I share MAXMINDS 2.0, a new Max Planck program to support scholars in danger of displacement by war or natural disasters, and who have limited access to resources and institutional support.
If you know affected scholars, please share.
www.maxminds.mpg.de
If you know affected scholars, please share.
www.maxminds.mpg.de
📢I am hiring a Postdoc to work on post-training methods for low-resource languages. Apply by August 15 employment.ku.dk/faculty/?sho....
Let's talk at #ACL2025NLP in Vienna if you want to know more about the position and life in Denmark.
Let's talk at #ACL2025NLP in Vienna if you want to know more about the position and life in Denmark.
Postdoc in Natural Language Processing
employment.ku.dk
July 7, 2025 at 12:47 PM
📢I am hiring a Postdoc to work on post-training methods for low-resource languages. Apply by August 15 employment.ku.dk/faculty/?sho....
Let's talk at #ACL2025NLP in Vienna if you want to know more about the position and life in Denmark.
Let's talk at #ACL2025NLP in Vienna if you want to know more about the position and life in Denmark.
Reposted by Desmond Elliott
💡Beyond math/code, instruction following with verifiable constraints is suitable to be learned with RLVR.
But the set of constraints and verifier functions is limited and most models overfit on IFEval.
We introduce IFBench to measure model generalization to unseen constraints.
But the set of constraints and verifier functions is limited and most models overfit on IFEval.
We introduce IFBench to measure model generalization to unseen constraints.
July 3, 2025 at 9:06 PM
💡Beyond math/code, instruction following with verifiable constraints is suitable to be learned with RLVR.
But the set of constraints and verifier functions is limited and most models overfit on IFEval.
We introduce IFBench to measure model generalization to unseen constraints.
But the set of constraints and verifier functions is limited and most models overfit on IFEval.
We introduce IFBench to measure model generalization to unseen constraints.
📣 I am happy to support Ph.D applications to the Danish Advanced Research Academy. My main areas of research include multimodal learning and tokenization-free language processing. Feel free to reach out if you have similar interests! Applications due August 29 www.daracademy.dk/fellowship/f...
Dara
www.daracademy.dk
June 26, 2025 at 2:40 PM
📣 I am happy to support Ph.D applications to the Danish Advanced Research Academy. My main areas of research include multimodal learning and tokenization-free language processing. Feel free to reach out if you have similar interests! Applications due August 29 www.daracademy.dk/fellowship/f...
Reposted by Desmond Elliott
Huge thanks to everyone that attended the Copenhagen NLP Symposium last week. Thanks for our wonderful speakers @kylelo.bsky.social, @najoung.bsky.social, Yohei Oseki, @mziizm.bsky.social, and @loubnabnl.hf.co! @mariaa.bsky.social did a great job of summarizing the talks in these liveposts (quoted).
June 23, 2025 at 3:13 PM
Huge thanks to everyone that attended the Copenhagen NLP Symposium last week. Thanks for our wonderful speakers @kylelo.bsky.social, @najoung.bsky.social, Yohei Oseki, @mziizm.bsky.social, and @loubnabnl.hf.co! @mariaa.bsky.social did a great job of summarizing the talks in these liveposts (quoted).
Thought-provoking interview with Meg about “AGI”
🙀Have we reached AGI?? 🙀
💛 Really grateful to @financialtimes.com and the incredible journalist @melissahei.bsky.social, who gave me space to talk about “AGI” (vs AI, vs ML) and where we’re headed.
Link here!!
www.ft.com/content/7089...
💛 Really grateful to @financialtimes.com and the incredible journalist @melissahei.bsky.social, who gave me space to talk about “AGI” (vs AI, vs ML) and where we’re headed.
Link here!!
www.ft.com/content/7089...
Margaret Mitchell: artificial general intelligence is ‘just vibes and snake oil’
One of the pioneers of AI ethics explains why human needs should be the central driver in the development of the technology
www.ft.com
June 22, 2025 at 6:19 PM
Thought-provoking interview with Meg about “AGI”
Reposted by Desmond Elliott
📯 Best Paper Award at CVPR workshop on Visual concepts for our (@doneata.bsky.social + @delliott.bsky.social) paper on probing vision/lang/ vision+lang models for semantic norms!
TLDR: SSL vision models (swinV2, dinoV2) are surprisingly similar to LLM & VLMs even w/o lang 👀
arxiv.org/abs/2506.03994
TLDR: SSL vision models (swinV2, dinoV2) are surprisingly similar to LLM & VLMs even w/o lang 👀
arxiv.org/abs/2506.03994
June 13, 2025 at 3:15 PM
📯 Best Paper Award at CVPR workshop on Visual concepts for our (@doneata.bsky.social + @delliott.bsky.social) paper on probing vision/lang/ vision+lang models for semantic norms!
TLDR: SSL vision models (swinV2, dinoV2) are surprisingly similar to LLM & VLMs even w/o lang 👀
arxiv.org/abs/2506.03994
TLDR: SSL vision models (swinV2, dinoV2) are surprisingly similar to LLM & VLMs even w/o lang 👀
arxiv.org/abs/2506.03994
I am looking forward to meeting people working on multimodality at #CVPR2025. You can find me hopping between the @vlms4all.bsky.social and Visual Concepts Workshops on Thursday. Feel free to reach out if you want to grab a coffee ☕ or a beer 🍻 during the week!
June 11, 2025 at 12:22 AM
I am looking forward to meeting people working on multimodality at #CVPR2025. You can find me hopping between the @vlms4all.bsky.social and Visual Concepts Workshops on Thursday. Feel free to reach out if you want to grab a coffee ☕ or a beer 🍻 during the week!
Reposted by Desmond Elliott
Announcing our recent work “Multilingual Pretraining for Pixel Language Models”! We introduce PIXEL-M4, a pixel language model pretrained on four visually & linguistically diverse scripts: English, Hindi, Ukrainian & Simplified Chinese. #NLProc
June 4, 2025 at 1:45 PM
Announcing our recent work “Multilingual Pretraining for Pixel Language Models”! We introduce PIXEL-M4, a pixel language model pretrained on four visually & linguistically diverse scripts: English, Hindi, Ukrainian & Simplified Chinese. #NLProc
Reposted by Desmond Elliott
I am excited to announce our latest work 🎉 "Cultural Evaluations of Vision-Language Models Have a Lot to Learn from Cultural Theory". We review recent works on culture in VLMs and argue for deeper grounding in cultural theory to enable more inclusive evaluations.
Paper 🔗: arxiv.org/pdf/2505.22793
Paper 🔗: arxiv.org/pdf/2505.22793
June 2, 2025 at 10:36 AM
I am excited to announce our latest work 🎉 "Cultural Evaluations of Vision-Language Models Have a Lot to Learn from Cultural Theory". We review recent works on culture in VLMs and argue for deeper grounding in cultural theory to enable more inclusive evaluations.
Paper 🔗: arxiv.org/pdf/2505.22793
Paper 🔗: arxiv.org/pdf/2505.22793
Reposted by Desmond Elliott
📢 The Copenhagen NLP Symposium on June 20th!
- Invited talks by @loubnabnl.hf.co (HF) @mziizm.bsky.social (Cohere) @najoung.bsky.social (BU) @kylelo.bsky.social (AI2) Yohei Oseki (UTokyo)
- Exciting posters by other participants
Register to attend and/or present your poster at cphnlp.github.io /1
- Invited talks by @loubnabnl.hf.co (HF) @mziizm.bsky.social (Cohere) @najoung.bsky.social (BU) @kylelo.bsky.social (AI2) Yohei Oseki (UTokyo)
- Exciting posters by other participants
Register to attend and/or present your poster at cphnlp.github.io /1
Copenhagen NLP Symposium 2025
symposium website
cphnlp.github.io
May 26, 2025 at 1:08 PM
📢 The Copenhagen NLP Symposium on June 20th!
- Invited talks by @loubnabnl.hf.co (HF) @mziizm.bsky.social (Cohere) @najoung.bsky.social (BU) @kylelo.bsky.social (AI2) Yohei Oseki (UTokyo)
- Exciting posters by other participants
Register to attend and/or present your poster at cphnlp.github.io /1
- Invited talks by @loubnabnl.hf.co (HF) @mziizm.bsky.social (Cohere) @najoung.bsky.social (BU) @kylelo.bsky.social (AI2) Yohei Oseki (UTokyo)
- Exciting posters by other participants
Register to attend and/or present your poster at cphnlp.github.io /1
Reposted by Desmond Elliott
Interested in multilingual tokenization in #NLP? Lisa Beinborn and I are hiring!
PhD candidate position in Göttingen, Germany: www.uni-goettingen.de/de/644546.ht...
PostDoc position in Leuven, Belgium:
www.kuleuven.be/personeel/jo...
Deadline 6th of June
PhD candidate position in Göttingen, Germany: www.uni-goettingen.de/de/644546.ht...
PostDoc position in Leuven, Belgium:
www.kuleuven.be/personeel/jo...
Deadline 6th of June
Stellen OBP - Georg-August-Universität Göttingen
Webseiten der Georg-August-Universität Göttingen
www.uni-goettingen.de
May 16, 2025 at 8:23 AM
Interested in multilingual tokenization in #NLP? Lisa Beinborn and I are hiring!
PhD candidate position in Göttingen, Germany: www.uni-goettingen.de/de/644546.ht...
PostDoc position in Leuven, Belgium:
www.kuleuven.be/personeel/jo...
Deadline 6th of June
PhD candidate position in Göttingen, Germany: www.uni-goettingen.de/de/644546.ht...
PostDoc position in Leuven, Belgium:
www.kuleuven.be/personeel/jo...
Deadline 6th of June
Reposted by Desmond Elliott
Has anyone written anything about *scraping and text processing* for internet pretraining data? Practical details, which tools are used, which webpage elements are considered, how HTML to text conversion is done?
(I know about work on quality filters, relevant but not quite what I'm looking for)
(I know about work on quality filters, relevant but not quite what I'm looking for)
May 9, 2025 at 10:05 AM
Has anyone written anything about *scraping and text processing* for internet pretraining data? Practical details, which tools are used, which webpage elements are considered, how HTML to text conversion is done?
(I know about work on quality filters, relevant but not quite what I'm looking for)
(I know about work on quality filters, relevant but not quite what I'm looking for)
Reposted by Desmond Elliott
Had fun talking at the Spurious Correlations & Shortcut Learning at ICLR! One example I brought up, which I think provides an uncommon perspective: a case where spurious shortcuts can improve generalization... even to out-of-distribution sets where the spurious feature doesn't generalize! Thread:
May 1, 2025 at 12:32 AM
Had fun talking at the Spurious Correlations & Shortcut Learning at ICLR! One example I brought up, which I think provides an uncommon perspective: a case where spurious shortcuts can improve generalization... even to out-of-distribution sets where the spurious feature doesn't generalize! Thread:
What would you do if someone has rolled your dataset into their benchmark (cool!) but marked it as being available under a much more permissive license (not so cool)?
April 14, 2025 at 1:44 PM
What would you do if someone has rolled your dataset into their benchmark (cool!) but marked it as being available under a much more permissive license (not so cool)?
I'm recruiting a postdoc on an 18-month contract candidate.hr-manager.net/ApplicationI.... The position is about deploying LLMs in the Danish public sector. This is an interdisciplinary project that touches on technical, ethical, and legal aspects of LLM usage. Apply by 1 May 2025.
Postdoctoral Researcher in Natural Language Processing
Postdoc in Natural Language Processing, Department of Computer Science, Faculty of Science, University of Copenhagen The Natural Language Process
candidate.hr-manager.net
April 12, 2025 at 8:49 AM
I'm recruiting a postdoc on an 18-month contract candidate.hr-manager.net/ApplicationI.... The position is about deploying LLMs in the Danish public sector. This is an interdisciplinary project that touches on technical, ethical, and legal aspects of LLM usage. Apply by 1 May 2025.
Reposted by Desmond Elliott
Very excited to release Kaleidoscope—a multilingual, multimodal evaluation set for VLMs, built as part of our open-science initiative!
🌍 18 languages (high-, mid-, low-)
📚 21k questions (55% require image understanding)
🧪 STEM, social science, reasoning, and practical skills
🌍 18 languages (high-, mid-, low-)
📚 21k questions (55% require image understanding)
🧪 STEM, social science, reasoning, and practical skills
April 10, 2025 at 7:52 PM
Very excited to release Kaleidoscope—a multilingual, multimodal evaluation set for VLMs, built as part of our open-science initiative!
🌍 18 languages (high-, mid-, low-)
📚 21k questions (55% require image understanding)
🧪 STEM, social science, reasoning, and practical skills
🌍 18 languages (high-, mid-, low-)
📚 21k questions (55% require image understanding)
🧪 STEM, social science, reasoning, and practical skills
Reposted by Desmond Elliott
🚀 We are excited to introduce Kaleidoscope, the largest culturally-authentic exam benchmark.
📌 Most VLM benchmarks are English-centric or rely on translations—missing linguistic & cultural nuance. Kaleidoscope expands in-language multilingual 🌎 & multimodal 👀 VLMs evaluation
📌 Most VLM benchmarks are English-centric or rely on translations—missing linguistic & cultural nuance. Kaleidoscope expands in-language multilingual 🌎 & multimodal 👀 VLMs evaluation
April 10, 2025 at 8:24 PM
🚀 We are excited to introduce Kaleidoscope, the largest culturally-authentic exam benchmark.
📌 Most VLM benchmarks are English-centric or rely on translations—missing linguistic & cultural nuance. Kaleidoscope expands in-language multilingual 🌎 & multimodal 👀 VLMs evaluation
📌 Most VLM benchmarks are English-centric or rely on translations—missing linguistic & cultural nuance. Kaleidoscope expands in-language multilingual 🌎 & multimodal 👀 VLMs evaluation
I'm really excited about this new benchmark for VLMs! It was great to do this as part of an open-science initiative and to meet so many great people.
Today we are releasing Kaleidoscope 🎉
A comprehensive multimodal & multilingual benchmark for VLMs! It contains real questions from exams in different languages.
🌍 20,911 questions and 18 languages
📚 14 subjects (STEM → Humanities)
📸 55% multimodal questions
A comprehensive multimodal & multilingual benchmark for VLMs! It contains real questions from exams in different languages.
🌍 20,911 questions and 18 languages
📚 14 subjects (STEM → Humanities)
📸 55% multimodal questions
April 10, 2025 at 1:03 PM
I'm really excited about this new benchmark for VLMs! It was great to do this as part of an open-science initiative and to meet so many great people.
Reposted by Desmond Elliott
📢Excited to announce our upcoming workshop - Vision Language Models For All: Building Geo-Diverse and Culturally Aware Vision-Language Models (VLMs-4-All) @CVPR 2025!
🌐 sites.google.com/view/vlms4all
🌐 sites.google.com/view/vlms4all
March 14, 2025 at 3:55 PM
📢Excited to announce our upcoming workshop - Vision Language Models For All: Building Geo-Diverse and Culturally Aware Vision-Language Models (VLMs-4-All) @CVPR 2025!
🌐 sites.google.com/view/vlms4all
🌐 sites.google.com/view/vlms4all
Reposted by Desmond Elliott
Announcing a new event initiative: Feminist AI LAN Party! Katharine and I did a pilot event last year and we're now taking it to @pyconde.bsky.social.
We've also open-sourced event kits to make it easy to host your own, including:
💣 hacking LLMs
📑 data development
✂️ zine making
feministai.party
We've also open-sourced event kits to make it easy to host your own, including:
💣 hacking LLMs
📑 data development
✂️ zine making
feministai.party
April 4, 2025 at 6:37 AM
Announcing a new event initiative: Feminist AI LAN Party! Katharine and I did a pilot event last year and we're now taking it to @pyconde.bsky.social.
We've also open-sourced event kits to make it easy to host your own, including:
💣 hacking LLMs
📑 data development
✂️ zine making
feministai.party
We've also open-sourced event kits to make it easy to host your own, including:
💣 hacking LLMs
📑 data development
✂️ zine making
feministai.party
Reposted by Desmond Elliott
Would you present your next NeurIPS paper in Europe instead of traveling to San Diego (US) if this was an option? Søren Hauberg (DTU) and I would love to hear the answer through this poll: (1/6)
NeurIPS participation in Europe
We seek to understand if there is interest in being able to attend NeurIPS in Europe, i.e. without travelling to San Diego, US. In the following, assume that it is possible to present accepted papers ...
docs.google.com
March 30, 2025 at 6:04 PM
Would you present your next NeurIPS paper in Europe instead of traveling to San Diego (US) if this was an option? Søren Hauberg (DTU) and I would love to hear the answer through this poll: (1/6)