Anna Seo Gyeong Choi
LightNews LightNews
Anna Seo Gyeong Choi
@annaseogyeongchoi.bsky.social
25 followers 17 following 8 posts
phd @ cornell information science
Posts Replies Media Videos
We can decompose performance degradation by individual grammar rules.
Three rules – existential “it”, zero copula, and y’all – account for roughly half of a dialect’s accuracy decreases, relative to Standard American English accuracy.
Three bar charts showing how individual grammar rules contribute to dialect performance degradation. Dark blue bars show single rule impact, medium purple shows all obligatory dialect rules, light purple shows complete dialect transformation. Percentages indicate proportion of total degradation explained by that rule set. Existential it: -5 to -8 point drops across Appalachian, African American, and Singaporean English, explaining 45-85% of degradation. Y'all: -4.5 point drops across Southern, Appalachian, and African American English, explaining 64-72% of degradation. Zero copula: -5.5 to -6.5 point drops for African American and Singaporean English, explaining 53-69% of degradation. Single rules account for 64-85% of degradation in American dialects.
November 6, 2025 at 12:10 AM
We used the Multi-VALUE package to transform Standard American English questions from QA datasets into dialectal variants based on grammatical rules.
Table showing three grammar rules that cause LLM errors across dialects. Row 1: Existential 'it' - occurs in Appalachian, African American, and Singaporean English. Standard American English example: 'How many kcal are there in one gram of ethanol?' Grammar rule applied: 'How many kcal is it in one gram of ethanol?' Row 2: Zero Copula - occurs in African American and Singaporean English. Standard American English example: 'Alpha emission is a type of what?' Grammar rule applied: 'Alpha emission a type of what?' Row 3: Y'all - occurs in Southern, Appalachian, and African American English. Standard American English example: 'Can you drive with a beer in Texas?' Grammar rule applied: 'Can y'all drive with a beer in Texas?' The changed words are highlighted in blue in each example, showing how small grammatical changes alter meaning for LLMs.
November 6, 2025 at 12:10 AM
We studied 6 English dialects (African American, Appalachian, Chicano, Indian, Singaporean, Southern) across 3 LLMs using 3 multiple-choice QA benchmarks.
The question: Do dialects affect performance even on easy tasks?
Answer: YES, with worst performance on Singaporean English.
Three tables showing LLM accuracy on dialect variants across different datasets. Each table has a title (BoolQ Yes/No Questions, SciQ Science Exam Questions, MMLU Multitask Knowledge Questions) and explanatory text stating 'Accuracy (%) on questions in benchmark dataset that were answered correctly in the original Standard American English (SAE) formulation. Numbers in parentheses show percentage point decrease from SAE baseline.' All tables show Standard American English at 100% accuracy baseline, with six dialect variants (Chicano, Appalachian, Southern, African American, Indian, and Singaporean English) tested on three models (Gemma-2B, Mistral-7B, GPT-4o-mini). Dialects range from 0.5 percentage points to 21.6 percentage points worse than SAE performance, with Singaporean English consistently showing the largest degradation across datasets.
November 6, 2025 at 12:08 AM
🧵Excited to present our work at #EMNLP2025 “Analyzing Dialectal Biases in LLMs for Knowledge and Reasoning Benchmarks”!
Paper 📄 arxiv.org/abs/2510.00962
w/ Eileen Pan, Skyler Seto, @allisonkoe.bsky.social @maartjeterhoeve.bsky.social
Research poster showing LLMs perform up to 20% worse on non-standard English dialects. Includes tables comparing performance across 6 dialects (African American, Appalachian, Chicano, Indian, Singaporean, Southern) and bar charts showing three grammar rules (existential it, zero copula, y'all) explain roughly half the accuracy decrease
November 6, 2025 at 12:08 AM