I would like to thank my wonderful co-authors, Anthony Sicilia and
@malihealikhani.bsky.social

Only with their efforts and constant support, this paper could come together. 🧵n/n

Code and checkpoints are open-source!

⚠️These LLMs are not perfect, and we will be updating as new libraries become available.

Dive in and build with us the next generation of signing LLMs:

🔗 github.com/Merterm/sign...
🔗 huggingface.co/merterm/sign...

🧵7/n

🛠️ In the paper, we also explored all avenues to teach LLMs to sign! We tested:
🌸In-Context Learning (Prompt-Tuning)
🌸Supervised Fine-Tuning (SFT)
🌸Multitasking Fine-Tuning 🧵6/n

🛡️ Forget Forgetting! We solve the problem of catastrophic forgetting. When an LLM loses its original spoken language skills after learning a new task. Our Multitasking Fine-Tuning strategy (mixing DGS & spoken language data, OpenOrca) successfully mitigates this data shift.🧵5/n

🚀 Our Core Contribution: We introduced new fine-tuning strategies to build the first text-based and multimodal LLMs capable of SLP. This includes:
✅ Video-Based Input capabilities via fine-tuned LLaVA (V2T task).
✅ Text-Based Models fine-tuned on LLaMA3. 🧵4/n

Unlike other LLMs, our models are trained and tested on 6 additional SLP tasks:
🌿(G2T) DGS to German
🌿(T2G) German to DGS
🌿(V2T) DGS Videos to German
🌿(I-G2T) Intensified DGS to German
🌿(T2I-G) German to Intensified DGS
🌿(G2E) DGS to English

🧵 3/n

🚀 We're proud to be among the first to bring multimodal and text-based SLP models to the open-source community. Our approach is uniquely comprehensive. 🧵 2/n

I would like to thank my wonderful co-authors, Anthony Sicilia, Suvodip Dey, @vardhandongre.bsky.social, @tejassrinivasan.bsky.social, @thomason.bsky.social, @gokhantur.bsky.social, @dilekh.bsky.social, @malihealikhani.bsky.social
With their efforts and support, this paper beautifully came together.

We present a taxonomy of different friction movements, and show that positive friction,

1⃣Extends Dialogue Acts
2⃣Helps Model User Mental States
3⃣Helps Accomplish User Goals

We hypothesize that systems can improve goal alignment, modeling of user mental states, and task success by deliberately slowing down conversations in strategic moments to ask questions, reveal assumptions, or pause.