Isabelle Lee
@wordscompute.bsky.social
ml/nlp phding @ usc, currently visiting harvard, scientisting @ startup;
interpretability & training & reasoning
iglee.me
interpretability & training & reasoning
iglee.me
a harder task- last step prediction: ¬(¬Sunny(x) ∧ Breezy(x)) ↔ [MASK] or last two step prediction. Most LLMs only achieve <50% accuracy on both tasks.
(n.b. since FOL is verifiable, we define correct as any generation that's equivalent to expression.)
7/9
(n.b. since FOL is verifiable, we define correct as any generation that's equivalent to expression.)
7/9
February 11, 2026 at 5:17 PM
a harder task- last step prediction: ¬(¬Sunny(x) ∧ Breezy(x)) ↔ [MASK] or last two step prediction. Most LLMs only achieve <50% accuracy on both tasks.
(n.b. since FOL is verifiable, we define correct as any generation that's equivalent to expression.)
7/9
(n.b. since FOL is verifiable, we define correct as any generation that's equivalent to expression.)
7/9
e.g. masked prediction. we mask an operator randomly and have LLMs guess: ¬(¬Sunny(x) ∧ Breezy(x)) ↔ (Sunny(x) [MASK] Breezy(x)). LLMs are correct ~45.7% on average:
6/9
6/9
February 11, 2026 at 5:17 PM
e.g. masked prediction. we mask an operator randomly and have LLMs guess: ¬(¬Sunny(x) ∧ Breezy(x)) ↔ (Sunny(x) [MASK] Breezy(x)). LLMs are correct ~45.7% on average:
6/9
6/9
So how do we strike a balance? We propose using First-Order Logic (FOL) as a middle ground. We
1. programmatically, randomly generate a bunch of FOL expressions
2. progressively simplify them, verifying their equivalence
3. chain them together
4. NL instantiate them w/ LLMs
4/9
1. programmatically, randomly generate a bunch of FOL expressions
2. progressively simplify them, verifying their equivalence
3. chain them together
4. NL instantiate them w/ LLMs
4/9
February 11, 2026 at 5:17 PM
So how do we strike a balance? We propose using First-Order Logic (FOL) as a middle ground. We
1. programmatically, randomly generate a bunch of FOL expressions
2. progressively simplify them, verifying their equivalence
3. chain them together
4. NL instantiate them w/ LLMs
4/9
1. programmatically, randomly generate a bunch of FOL expressions
2. progressively simplify them, verifying their equivalence
3. chain them together
4. NL instantiate them w/ LLMs
4/9
New dataset 🗂️ coming to #eacl
What is (correct) reasoning in LLMs? How do you rigorously define/measure process fidelity? How might we study its acquisition in large scale training? We made a gigantic, verifiably correct reasoning traces of first order logic expressions!
1/9
What is (correct) reasoning in LLMs? How do you rigorously define/measure process fidelity? How might we study its acquisition in large scale training? We made a gigantic, verifiably correct reasoning traces of first order logic expressions!
1/9
February 11, 2026 at 5:17 PM
New dataset 🗂️ coming to #eacl
What is (correct) reasoning in LLMs? How do you rigorously define/measure process fidelity? How might we study its acquisition in large scale training? We made a gigantic, verifiably correct reasoning traces of first order logic expressions!
1/9
What is (correct) reasoning in LLMs? How do you rigorously define/measure process fidelity? How might we study its acquisition in large scale training? We made a gigantic, verifiably correct reasoning traces of first order logic expressions!
1/9
Really excited to receive Coefficient Giving's Technical AI Safety Research Grant via Berkeley Existential Risk Initiative w/
@nsaphra.bsky.social! We aim to predict potential AI model failures before impact--before deployment, using interpretability.
@nsaphra.bsky.social! We aim to predict potential AI model failures before impact--before deployment, using interpretability.
February 11, 2026 at 5:07 PM
Really excited to receive Coefficient Giving's Technical AI Safety Research Grant via Berkeley Existential Risk Initiative w/
@nsaphra.bsky.social! We aim to predict potential AI model failures before impact--before deployment, using interpretability.
@nsaphra.bsky.social! We aim to predict potential AI model failures before impact--before deployment, using interpretability.
titled: peer review
March 29, 2025 at 4:58 AM
titled: peer review
made something with friends on friendsgiving
November 18, 2024 at 11:14 PM
made something with friends on friendsgiving