Yupei Du
@yupeidu.bsky.social
Postdoc at @lst.uni-saarland.de. LLM reasoning
[Observation 2]
The computation for the correct labels is NOT independent of that for noisy labels: instead, predicting noisy labels relies on computing noisy labels!
We ablate the correct "bridge entities" in THR, and find that noise memorization is heavily influenced
The computation for the correct labels is NOT independent of that for noisy labels: instead, predicting noisy labels relies on computing noisy labels!
We ablate the correct "bridge entities" in THR, and find that noise memorization is heavily influenced
November 1, 2025 at 5:25 PM
[Observation 2]
The computation for the correct labels is NOT independent of that for noisy labels: instead, predicting noisy labels relies on computing noisy labels!
We ablate the correct "bridge entities" in THR, and find that noise memorization is heavily influenced
The computation for the correct labels is NOT independent of that for noisy labels: instead, predicting noisy labels relies on computing noisy labels!
We ablate the correct "bridge entities" in THR, and find that noise memorization is heavily influenced
[Observation 1]
Even after perfect memorization of noisy labels, the computation for correct labels persists within our models!
We find that models still produces the correct labels at eariler layers (red lines, Mem-Corrected) and only **override** them with noisy labels later
Even after perfect memorization of noisy labels, the computation for correct labels persists within our models!
We find that models still produces the correct labels at eariler layers (red lines, Mem-Corrected) and only **override** them with noisy labels later
November 1, 2025 at 5:25 PM
[Observation 1]
Even after perfect memorization of noisy labels, the computation for correct labels persists within our models!
We find that models still produces the correct labels at eariler layers (red lines, Mem-Corrected) and only **override** them with noisy labels later
Even after perfect memorization of noisy labels, the computation for correct labels persists within our models!
We find that models still produces the correct labels at eariler layers (red lines, Mem-Corrected) and only **override** them with noisy labels later
[Obervation 0]
on our tasks, generalization happens earlier than memorization: even on training instances of noisy labels (e.g., a wrong addition result in FDA, or a wrong target person entity in THR), our models first produces the correct answers for them
on our tasks, generalization happens earlier than memorization: even on training instances of noisy labels (e.g., a wrong addition result in FDA, or a wrong target person entity in THR), our models first produces the correct answers for them
November 1, 2025 at 5:25 PM
[Obervation 0]
on our tasks, generalization happens earlier than memorization: even on training instances of noisy labels (e.g., a wrong addition result in FDA, or a wrong target person entity in THR), our models first produces the correct answers for them
on our tasks, generalization happens earlier than memorization: even on training instances of noisy labels (e.g., a wrong addition result in FDA, or a wrong target person entity in THR), our models first produces the correct answers for them
Specifically:
we train GPT2-style LM from scratch on two tasks: four-digit addition (FDA) and two-hop relational reasoning (THR), with 2-10% random label noise injected
Examples:
1. 1357+2473=7143 (FDA)
2. Who is the debtor of the neighbor of Adam? (THR, all facts are known)
we train GPT2-style LM from scratch on two tasks: four-digit addition (FDA) and two-hop relational reasoning (THR), with 2-10% random label noise injected
Examples:
1. 1357+2473=7143 (FDA)
2. Who is the debtor of the neighbor of Adam? (THR, all facts are known)
November 1, 2025 at 5:25 PM
Specifically:
we train GPT2-style LM from scratch on two tasks: four-digit addition (FDA) and two-hop relational reasoning (THR), with 2-10% random label noise injected
Examples:
1. 1357+2473=7143 (FDA)
2. Who is the debtor of the neighbor of Adam? (THR, all facts are known)
we train GPT2-style LM from scratch on two tasks: four-digit addition (FDA) and two-hop relational reasoning (THR), with 2-10% random label noise injected
Examples:
1. 1357+2473=7143 (FDA)
2. Who is the debtor of the neighbor of Adam? (THR, all facts are known)
How do language models memorize noise while reason impressively well?
Our #EMNLP2025 (poster, Nov 5, 11:00-12:30, Hall C) paper shows that memorization reuses internal mechanisms of generalization, even when they are not related to each other!
arxiv.org/abs/2507.04782
Our #EMNLP2025 (poster, Nov 5, 11:00-12:30, Hall C) paper shows that memorization reuses internal mechanisms of generalization, even when they are not related to each other!
arxiv.org/abs/2507.04782
November 1, 2025 at 5:25 PM
How do language models memorize noise while reason impressively well?
Our #EMNLP2025 (poster, Nov 5, 11:00-12:30, Hall C) paper shows that memorization reuses internal mechanisms of generalization, even when they are not related to each other!
arxiv.org/abs/2507.04782
Our #EMNLP2025 (poster, Nov 5, 11:00-12:30, Hall C) paper shows that memorization reuses internal mechanisms of generalization, even when they are not related to each other!
arxiv.org/abs/2507.04782