Marianne de Heer Kloots
LightNews LightNews
mdhk.net
Marianne de Heer Kloots
@mdhk.net
1.2K followers 490 following 100 posts
Linguist in AI & CogSci 🧠👩‍💻🤖 PhD student @ ILLC, University of Amsterdam

🌐 https://mdhk.net/
🐘 https://scholar.social/@mdhk
🐦 https://twitter.com/mariannedhk
Posts Replies Media Videos
mdhk.net
Finally, downstream performance on Dutch speech-to-text transcription reflects the language-specific advantage for Dutch linguistic feature encoding in model-internal representations: on average, Wav2Vec2-NL has a 27% lower word error rate than the multilingual model.
Word Error Rate results for models fine-tuned for Dutch ASR (speech-to-text transcription), across 4 models and 5 evaluation datasets.
August 27, 2025 at 2:31 PM
mdhk.net
We find that language-specific advantages are well-detected by trained clustering or classification probes, and partially observable using zero-shot metrics. I.e. the encoding of Dutch linguistic features is enhanced in the Dutch model, as compared to models trained on English and multilingual data.
Layerwise phonetic and lexical analyses, across a read speech (MLS, top row) and a dialogue (IFADV, bottom row) dataset of spoken Dutch. Measures marked * involve optimized linear transforms, whereas others are computed zero-shot; shading indicates 95% confidence intervals. The Dutch Wav2Vec2-NL model achieves highest scores across most analyses of Dutch phone and word encoding, though the size of this language-specific advantage varies considerably across analyses.
August 27, 2025 at 2:31 PM
mdhk.net
But they also used different analysis techniques.

We designed the SSL-NL dataset to test the encoding of Dutch phonetic and lexical features in SSL speech representations, while allowing for comparisons across different analysis methods.

We compare both trained probes(*) and zero-shot metrics:
The model comparison set includes Wav2Vec2-NL and 3 other existing Wav2Vec2-base models: facebook's multilingual voxpopuli model, facebook's English base model, and another model trained on nonspeech acoustics. The set of analysis techniques includes probing classifiers (logistic regression), ABX similarities, PCA clustering, LDA clustering, and representational similarity analysis (RSA). Word- and phone-level embeddings were created by mean-pooling model frame embeddings within words and phones respectively. The SSL-NL evaluation dataset is a curated dataset of Dutch speech recordings and accompanying forced alignments, across two domains: audiobooks (MLS) and face-to-face conversations (IFADV).
August 27, 2025 at 2:31 PM
mdhk.net
✨ Do self-supervised speech models learn to encode language-specific linguistic features from their training data, or only more language-general acoustic correlates?

At #Interspeech2025 we presented our new Wav2Vec2-NL model and SSL-NL evaluation dataset to test this!

📄 arxiv.org/abs/2506.00981

⬇️
Interspeech paper title: What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-training Authors: Marianne de Heer Kloots, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema, Martijn Bentum
August 27, 2025 at 2:31 PM
mdhk.net
Last but not least, I personally can’t wait for the social event on Thursday night that we’ve been planning for the past year ✨
It features a *live brain-controlled music act* by the AIAR collective 🧠🎶 2025.ccneuro.org/social-event/ Get one of the last remaining tickets at the registration desk now!
August 12, 2025 at 2:19 PM
mdhk.net
So exciting, #CCN2025 in Amsterdam started today! We have stroopwafels!!

Catch me at my poster on Friday to chat about the role of context in neural representational alignment to spoken language systems (C34) 🙌



🔗 2025.ccneuro.org/poster/?id=K...
August 12, 2025 at 2:19 PM
mdhk.net
We are having an impromptu overflow room around the corner 😅
July 28, 2025 at 12:26 PM
mdhk.net
Next week I’ll be in Vienna for my first *ACL conference! 🇦🇹✨

I will present our new BLiMP-NL dataset for evaluating language models on Dutch syntactic minimal pairs and human acceptability judgments ⬇️

🗓️ Tuesday, July 29th, 16:00-17:30, Hall X4 / X5 (Austria Center Vienna)
The BLiMP-NL dataset consists of 84 Dutch minimal pair paradigms covering 22 syntactic phenomena, and comes with graded human acceptability ratings & self-paced reading times. An example minimal pair: A. Ik bekijk de foto van mezelf in de kamer (I watch the photograph of myself in the room; grammatical) B. Wij bekijken de foto van mezelf in de kamer (We watch the photograph of myself in the room; ungrammatical) Differences in human acceptability ratings between sentences correlate with differences in model syntactic log-odds ratio scores.
July 24, 2025 at 3:30 PM
mdhk.net
If you see this, post a concert picture you took this year.
December 29, 2024 at 4:15 PM
mdhk.net
red square against budget cuts in higher education
December 4, 2024 at 9:44 AM
mdhk.net
In a bizarre undemocratic turn of events, the massive national protest against our government's plans for higher education was cancelled last week.

We'll be back stronger next Monday in The Hague! 🟥
poster announcing the protest on November 25th, 1pm, Malieveld, The Hague
November 19, 2024 at 12:43 PM
mdhk.net
Or this from within The Netherlands: campagnes.degoedezaak.org/campaigns/st...
and come to Utrecht on Nov 14th! www.fnv.nl/cao-sector/o...
October 29, 2024 at 5:13 PM
mdhk.net
Bluesky now has over 10 million users, and I was #497,227! 😎
September 18, 2024 at 6:52 AM
mdhk.net
I will be presenting this work tomorrow (Thursday) at #INTERSPEECH2024, 10.00-10.40 in the Acesso room!

Looking forward to discuss how we can learn from human speech science to interpret end-to-end neural speech models 💡

The paper is here: www.isca-archive.org/interspeech_...
paper title: Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0
September 4, 2024 at 6:35 AM
mdhk.net
It turns out the accuracy of dependency structures decoded from LM hidden layers (measured by Labelled Attachment Score) strongly correlates with similarity to brain activity in sentence reading! 🧠
This correlation disappears in a control condition with scrambled inputs.
Figure with two scatterplots, showing a strong correlation between dependency accuracy (Labelled Attachment Score) and brain alignment (Representational Similarity Score) on the left, and no correlation in a scrambled control condition on the right.
July 23, 2024 at 8:53 PM
mdhk.net
Language model internal states show surprising similarity to human brain activity in language comprehension — but how does this relate to their accurate representation of structured linguistic information, like syntactic dependencies? (i.e. links between words in a sentence)
Dependency parse for the sentence "De overtreder die de smeris ontvlucht was is een kronkelig paadje ingerend" (Dutch for: "The offender who had escaped from the cop ran into a winding path"). The picture shows all dependency links between the words of the sentence, such as the links between verbs and their subjects (Subj) and nouns and their determiners (Det).
July 23, 2024 at 8:50 PM
mdhk.net
Excited for #CogSci2024 this week!

In session T.24 on Friday morning (10.30-12), Bram will present our work on representational alignment between LMs, brains, and syntactic structure 🤖🧠💬

w/ Rochelle Choenni, @mheilbron.bsky.social & @wzuidema.bsky.social

📑 escholarship.org/uc/item/1fp7...

⬇️
Paper title ('Language Models That Accurately Represent Syntactic Structure Exhibit Higher Representational Similarity To Brain Activity') and overview figure
July 23, 2024 at 8:49 PM
mdhk.net
📏 We also compare three analysis methods for decoding phoneme preference from model internals, and find interesting differences between them!

➡️ Read more in the paper: arxiv.org/abs/2407.03005
July 8, 2024 at 5:40 AM
mdhk.net
💡 We find similar adaptation to phonotactic context in Wav2Vec2 models, emerging around the 4th layer of their Transformer module. This effect is amplified by finetuning for text transcription, but also present in fully self-supervised models (when trained on English speech).
July 8, 2024 at 5:39 AM
mdhk.net
One case of such contextual biasing effects comes from phonotactic constraints.

For example in English: TL << TR, SL >> SR

This has been demonstrated in human listeners a while ago! (doi.org/10.3758/BF03...)
July 8, 2024 at 5:38 AM
mdhk.net
Feeling very inspired about ✨Using ANNs for Studying Human Language Learning and Processing (ann-humlang.github.io )✨ after the workshop that Tamar Johnson and I organized this week at the ILLC in Amsterdam! Many thanks to all our speakers and participants for such a great event,
June 13, 2024 at 3:19 PM
mdhk.net
A nice session at KNAW tonight looking back on the year since the launch of ChatGPT — Katia is giving a short technical glimpse behind the curtains (🥁) of LLMs right now, that I made some illustrations for! Livestream: www.youtube.com/live/Nn41XWA...
November 30, 2023 at 6:55 PM
mdhk.net
Finally, there's some useful settings you can tune to make things better on your home feed as well!
I currently have this in Home Feed and Thread Preferences settings (forgot which ones are different from the defaults)
October 22, 2023 at 8:11 PM
mdhk.net
I'm a bird!
August 22, 2023 at 9:31 AM