All the details about the task are available here 👉 sites.google.com/view/crucive...
📅 Training data release: 22 September 2026

📂 Code available at the following repository: github.com/snizio/Beyon...
🧵(5/5)

✅ Larger models develop more robust substring awareness.
✅ Morphemes are recognized better than meaningless substrings.
✅ Awareness emerges early for suffixes and roots, later for non-morphemic units
✅ Productivity, word frequency and tokenization shape this ability.
🧵(4/5)

🧪 We design a controlled binary task asking models whether a substring appears in a word. Using MorphoLex, we evaluate models from the Pythia family across:
- substring position and length;
- morphemic vs. non-morphemic substrings;
- pre-training checkpoints.
🧵(3/5)

LMs operate on subword tokens and lack explicit access to characters. Despite so, they show a limited ability to recognize spelling-level patterns (i.e. Spelling Miracle). In this work, we take a look at when, where, and how such character-level awareness emerges.
🧵(2/5)

🤗 Models available on Huggingface: huggingface.co/collections/...
📂 Code & Dataset: github.com/snizio/Lexic...
🧵(5/5)

🧠 Our findings show that Transformer-based models can handle lexical composition and meaning inference to some extent—effectively producing and interpreting plausible lexical innovations, though with a notable drop in performance vs. standard lexical items.
🧵(4/5)

Key contributions:
✅ A new framework to assess lexical abilities across tasks & word types
✅ A lexical resource for Italian with definitions & examples
✅ Analysis of model size, multilinguality & linguistic features
✅ Human eval via the Optimal Innovation Hypothesis
🧵(3/5)

In this study, we propose a novel, unified framework to evaluate lexical proficiency in Transformer-based LMs, testing their ability to generate, define, and use words across three lexical categories: commonly lexicalized words, recent neologisms and nonce words.
🧵(2/5)

🔜 More info coming soon!

3) Findings: Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors (with Pedrotti A., Papucci M., Ciaccio C., Puccetti G., Dell'Orletta F. and Esuli A.)