Kayo Yin
@kayoyin.bsky.social
PhD student at UC Berkeley. NLP for signed languages and LLM interpretability. kayoyin.github.io
🏂🎹🚵♀️🥋
🏂🎹🚵♀️🥋
Other interesting findings:
- FV heads have relatively high induction scores and vice versa compared to other heads
- FV heads emerge later in training than induction heads
- ICL accuracy rises around the same time induction emerges during training, but increases more gradually
- FV heads have relatively high induction scores and vice versa compared to other heads
- FV heads emerge later in training than induction heads
- ICL accuracy rises around the same time induction emerges during training, but increases more gradually
February 28, 2025 at 4:16 PM
Other interesting findings:
- FV heads have relatively high induction scores and vice versa compared to other heads
- FV heads emerge later in training than induction heads
- ICL accuracy rises around the same time induction emerges during training, but increases more gradually
- FV heads have relatively high induction scores and vice versa compared to other heads
- FV heads emerge later in training than induction heads
- ICL accuracy rises around the same time induction emerges during training, but increases more gradually
We also find evidence of induction heads that evolve into FV heads.
Several instances of FV heads have a high induction score earlier in training (around when induction heads first emerge). However, the reverse (induction heads with high FV scores earlier) does not occur.
Several instances of FV heads have a high induction score earlier in training (around when induction heads first emerge). However, the reverse (induction heads with high FV scores earlier) does not occur.
February 28, 2025 at 4:16 PM
We also find evidence of induction heads that evolve into FV heads.
Several instances of FV heads have a high induction score earlier in training (around when induction heads first emerge). However, the reverse (induction heads with high FV scores earlier) does not occur.
Several instances of FV heads have a high induction score earlier in training (around when induction heads first emerge). However, the reverse (induction heads with high FV scores earlier) does not occur.
2 mechanisms have been proposed to explain ICL: induction heads that find and copy relevant tokens, and FV heads that compute a latent encoding of the task from examples.
Our ablations show that FV heads are crucial for few-shot ICL, whereas induction heads are not necessary.
Our ablations show that FV heads are crucial for few-shot ICL, whereas induction heads are not necessary.
February 28, 2025 at 4:16 PM
2 mechanisms have been proposed to explain ICL: induction heads that find and copy relevant tokens, and FV heads that compute a latent encoding of the task from examples.
Our ablations show that FV heads are crucial for few-shot ICL, whereas induction heads are not necessary.
Our ablations show that FV heads are crucial for few-shot ICL, whereas induction heads are not necessary.
Induction heads are commonly associated with in-context learning, but are they the primary driver of ICL at scale?
We find that recently discovered "function vector" heads, which encode the ICL task, are the actual primary mechanisms behind few-shot ICL!
arxiv.org/abs/2502.14010
🧵👇
We find that recently discovered "function vector" heads, which encode the ICL task, are the actual primary mechanisms behind few-shot ICL!
arxiv.org/abs/2502.14010
🧵👇
February 28, 2025 at 4:16 PM
Induction heads are commonly associated with in-context learning, but are they the primary driver of ICL at scale?
We find that recently discovered "function vector" heads, which encode the ICL task, are the actual primary mechanisms behind few-shot ICL!
arxiv.org/abs/2502.14010
🧵👇
We find that recently discovered "function vector" heads, which encode the ICL task, are the actual primary mechanisms behind few-shot ICL!
arxiv.org/abs/2502.14010
🧵👇
took my dog to the beach today. His name is apollo and he’s 14 years old :)
December 15, 2024 at 12:35 PM
took my dog to the beach today. His name is apollo and he’s 14 years old :)
this is driving me crazy, does anyone recognize this song?? It’s stuck in my head but I can’t remember where I heard it
November 26, 2024 at 9:51 AM
this is driving me crazy, does anyone recognize this song?? It’s stuck in my head but I can’t remember where I heard it
multiple people told me I seem to be in a visibly better mood the past 2 days which coincides perfectly with Berkeley finally getting rainy weather
November 22, 2024 at 10:53 PM
multiple people told me I seem to be in a visibly better mood the past 2 days which coincides perfectly with Berkeley finally getting rainy weather
What about perceptual effort - could it be correlated with English usage?
Perceptual effort to distinguish between 2 handshapes is very weakly correlated with how often the 2 letters appear in similar contexts in English, and in the "wrong" direction for efficiency. 7/8
Perceptual effort to distinguish between 2 handshapes is very weakly correlated with how often the 2 letters appear in similar contexts in English, and in the "wrong" direction for efficiency. 7/8
November 21, 2024 at 5:40 AM
What about perceptual effort - could it be correlated with English usage?
Perceptual effort to distinguish between 2 handshapes is very weakly correlated with how often the 2 letters appear in similar contexts in English, and in the "wrong" direction for efficiency. 7/8
Perceptual effort to distinguish between 2 handshapes is very weakly correlated with how often the 2 letters appear in similar contexts in English, and in the "wrong" direction for efficiency. 7/8
We also look at handshapes in ASL fingerspelling (used to spell out English words, 1 handshape = 1 letter) and their correlation with letter frequency in English text.
No significant correlation between fingerspelling handshapes and English letter frequency! 6/8
No significant correlation between fingerspelling handshapes and English letter frequency! 6/8
November 21, 2024 at 5:40 AM
We also look at handshapes in ASL fingerspelling (used to spell out English words, 1 handshape = 1 letter) and their correlation with letter frequency in English text.
No significant correlation between fingerspelling handshapes and English letter frequency! 6/8
No significant correlation between fingerspelling handshapes and English letter frequency! 6/8
We compute the correlation between articulatory effort and handshape frequency in a lexicon of ASL signs (ASL-LEX).
In core signs native to ASL (left), frequent handshapes are easier to produce!
In initialized and loan signs borrowed from English (right), no correlation! 5/8
In core signs native to ASL (left), frequent handshapes are easier to produce!
In initialized and loan signs borrowed from English (right), no correlation! 5/8
November 21, 2024 at 5:40 AM
We compute the correlation between articulatory effort and handshape frequency in a lexicon of ASL signs (ASL-LEX).
In core signs native to ASL (left), frequent handshapes are easier to produce!
In initialized and loan signs borrowed from English (right), no correlation! 5/8
In core signs native to ASL (left), frequent handshapes are easier to produce!
In initialized and loan signs borrowed from English (right), no correlation! 5/8
For perceptual effort, we measure handshape similarity.
When two handshapes have similar finger joint angles, they appear more alike, making it harder to distinguish between them perceptually. 4/8
When two handshapes have similar finger joint angles, they appear more alike, making it harder to distinguish between them perceptually. 4/8
November 21, 2024 at 5:40 AM
For perceptual effort, we measure handshape similarity.
When two handshapes have similar finger joint angles, they appear more alike, making it harder to distinguish between them perceptually. 4/8
When two handshapes have similar finger joint angles, they appear more alike, making it harder to distinguish between them perceptually. 4/8
For articulatory effort, we measure finger independence.
The more variation there is in finger joint angles within a handshape, the more difficult it is to produce that handshape. 3/8
The more variation there is in finger joint angles within a handshape, the more difficult it is to produce that handshape. 3/8
November 21, 2024 at 5:40 AM
For articulatory effort, we measure finger independence.
The more variation there is in finger joint angles within a handshape, the more difficult it is to produce that handshape. 3/8
The more variation there is in finger joint angles within a handshape, the more difficult it is to produce that handshape. 3/8
✨TISLR 15 stage presentation✨
Spoken languages exhibit communicative efficiency by minimizing speaker+listener effort.
What about signed languages?
ASL handshapes reflect efficiency pressures - but only in native signs, not signs borrowed from English!
aclanthology.org/2024.acl-lon... 🧵
Spoken languages exhibit communicative efficiency by minimizing speaker+listener effort.
What about signed languages?
ASL handshapes reflect efficiency pressures - but only in native signs, not signs borrowed from English!
aclanthology.org/2024.acl-lon... 🧵
November 21, 2024 at 5:40 AM
✨TISLR 15 stage presentation✨
Spoken languages exhibit communicative efficiency by minimizing speaker+listener effort.
What about signed languages?
ASL handshapes reflect efficiency pressures - but only in native signs, not signs borrowed from English!
aclanthology.org/2024.acl-lon... 🧵
Spoken languages exhibit communicative efficiency by minimizing speaker+listener effort.
What about signed languages?
ASL handshapes reflect efficiency pressures - but only in native signs, not signs borrowed from English!
aclanthology.org/2024.acl-lon... 🧵
Since most of our dataset does not have fingerspelling annotations (we annotated 507 sentences), we train initial models for automatic sign suggestion using self-supervised contrastive learning.
We find that contrastive learning significantly improves model accuracy.
6/8
We find that contrastive learning significantly improves model accuracy.
6/8
November 19, 2024 at 12:19 AM
Since most of our dataset does not have fingerspelling annotations (we annotated 507 sentences), we train initial models for automatic sign suggestion using self-supervised contrastive learning.
We find that contrastive learning significantly improves model accuracy.
6/8
We find that contrastive learning significantly improves model accuracy.
6/8
Studies show DHH students prefer technical terms to be signed instead of fingerspelled.
We propose a new task - *automatic sign suggestion*: given an English sentence and an ASL video, a model detects when the interpreter fingerspells, and suggests ASL signs to use instead.
5/8
We propose a new task - *automatic sign suggestion*: given an English sentence and an ASL video, a model detects when the interpreter fingerspells, and suggests ASL signs to use instead.
5/8
November 19, 2024 at 12:19 AM
Studies show DHH students prefer technical terms to be signed instead of fingerspelled.
We propose a new task - *automatic sign suggestion*: given an English sentence and an ASL video, a model detects when the interpreter fingerspells, and suggests ASL signs to use instead.
5/8
We propose a new task - *automatic sign suggestion*: given an English sentence and an ASL video, a model detects when the interpreter fingerspells, and suggests ASL signs to use instead.
5/8
E.g. interpreting STEM documents is challenging because of the lack of standardized STEM terminology in ASL.
Our dataset reflects how interpreters often use fingerspelling (spelling out the English word using letter signs) for technical terms when the ASL sign is unknown.
4/8
Our dataset reflects how interpreters often use fingerspelling (spelling out the English word using letter signs) for technical terms when the ASL sign is unknown.
4/8
November 19, 2024 at 12:19 AM
E.g. interpreting STEM documents is challenging because of the lack of standardized STEM terminology in ASL.
Our dataset reflects how interpreters often use fingerspelling (spelling out the English word using letter signs) for technical terms when the ASL sign is unknown.
4/8
Our dataset reflects how interpreters often use fingerspelling (spelling out the English word using letter signs) for technical terms when the ASL sign is unknown.
4/8
ASL STEM Wiki is the first continuous signing dataset focused on STEM, with Wikipedia articles translated by certified ASL interpreters.
We release this dataset alongside our paper, identifying several use cases for ASL STEM Wiki informed by its unique characteristics.
3/8
We release this dataset alongside our paper, identifying several use cases for ASL STEM Wiki informed by its unique characteristics.
3/8
November 19, 2024 at 12:19 AM
ASL STEM Wiki is the first continuous signing dataset focused on STEM, with Wikipedia articles translated by certified ASL interpreters.
We release this dataset alongside our paper, identifying several use cases for ASL STEM Wiki informed by its unique characteristics.
3/8
We release this dataset alongside our paper, identifying several use cases for ASL STEM Wiki informed by its unique characteristics.
3/8
🚨New dataset + challenge🚨
We release ASL STEM Wiki: the first signing dataset of STEM articles!
📰 254 Wikipedia articles
📹 ~300 hours of ASL interpretations
👋 New task: automatic sign suggestion to make STEM education more accessible
microsoft.com/en-us/resear...
🧵 #EMNLP2024
We release ASL STEM Wiki: the first signing dataset of STEM articles!
📰 254 Wikipedia articles
📹 ~300 hours of ASL interpretations
👋 New task: automatic sign suggestion to make STEM education more accessible
microsoft.com/en-us/resear...
🧵 #EMNLP2024
November 19, 2024 at 12:19 AM
🚨New dataset + challenge🚨
We release ASL STEM Wiki: the first signing dataset of STEM articles!
📰 254 Wikipedia articles
📹 ~300 hours of ASL interpretations
👋 New task: automatic sign suggestion to make STEM education more accessible
microsoft.com/en-us/resear...
🧵 #EMNLP2024
We release ASL STEM Wiki: the first signing dataset of STEM articles!
📰 254 Wikipedia articles
📹 ~300 hours of ASL interpretations
👋 New task: automatic sign suggestion to make STEM education more accessible
microsoft.com/en-us/resear...
🧵 #EMNLP2024
first post 🦋
food in miami was so good thanks emnlp
food in miami was so good thanks emnlp
November 18, 2024 at 7:30 PM
first post 🦋
food in miami was so good thanks emnlp
food in miami was so good thanks emnlp