Fabian David Schmidt
@fdschmidt.bsky.social
PhD candidate at Uni of Würzburg working on multilinguality & multimodality | prev. visited visit Mila & LTL@UniCambridge
https://fdschmidt93.github.io
https://fdschmidt93.github.io
Cross-modal topic matching correlates well with other multilingual vision-language tasks!
🤗Images-To-Sentence (given Images, select topically fitting sentence) & Sentences-To-Image (given Sentences, pick topically matching image) probe complementary aspects in VLU
🤗Images-To-Sentence (given Images, select topically fitting sentence) & Sentences-To-Image (given Sentences, pick topically matching image) probe complementary aspects in VLU
February 21, 2025 at 7:46 AM
Cross-modal topic matching correlates well with other multilingual vision-language tasks!
🤗Images-To-Sentence (given Images, select topically fitting sentence) & Sentences-To-Image (given Sentences, pick topically matching image) probe complementary aspects in VLU
🤗Images-To-Sentence (given Images, select topically fitting sentence) & Sentences-To-Image (given Sentences, pick topically matching image) probe complementary aspects in VLU
X-modal to text-only perf. *gap* shows that VL support decreases from high to low-resource language tiers:
Images/Topic→Sentence (for I/T, pick S): narrows with less textual support (left)
Sentences→Image/Topic (for S, pick I/T): increases with less VL support worse (right)
Images/Topic→Sentence (for I/T, pick S): narrows with less textual support (left)
Sentences→Image/Topic (for S, pick I/T): increases with less VL support worse (right)
February 21, 2025 at 7:46 AM
X-modal to text-only perf. *gap* shows that VL support decreases from high to low-resource language tiers:
Images/Topic→Sentence (for I/T, pick S): narrows with less textual support (left)
Sentences→Image/Topic (for S, pick I/T): increases with less VL support worse (right)
Images/Topic→Sentence (for I/T, pick S): narrows with less textual support (left)
Sentences→Image/Topic (for S, pick I/T): increases with less VL support worse (right)
Strong vision-language models (VLMs) like GPT-4o-mini maintain good performance for top-150 languages, only to drop to performing no better than chance for the lowest resource languages!
February 21, 2025 at 7:46 AM
Strong vision-language models (VLMs) like GPT-4o-mini maintain good performance for top-150 languages, only to drop to performing no better than chance for the lowest resource languages!
Introducing MVL-SIB, a massively multilingual vision-language benchmark for cross-modal topic matching in 205 languages!
🤔Tasks: Given images (sentences), select topically matching sentence (image).
Arxiv: arxiv.org/abs/2502.12852
HF: huggingface.co/datasets/Wue...
Details👇
🤔Tasks: Given images (sentences), select topically matching sentence (image).
Arxiv: arxiv.org/abs/2502.12852
HF: huggingface.co/datasets/Wue...
Details👇
February 21, 2025 at 7:46 AM
Introducing MVL-SIB, a massively multilingual vision-language benchmark for cross-modal topic matching in 205 languages!
🤔Tasks: Given images (sentences), select topically matching sentence (image).
Arxiv: arxiv.org/abs/2502.12852
HF: huggingface.co/datasets/Wue...
Details👇
🤔Tasks: Given images (sentences), select topically matching sentence (image).
Arxiv: arxiv.org/abs/2502.12852
HF: huggingface.co/datasets/Wue...
Details👇