Anuj Diwan
LightNews LightNews
anujdiwan.bsky.social
Anuj Diwan
@anujdiwan.bsky.social
190 followers 120 following 6 posts
UT CS PhD Student working on generative speech models. Prev. Student Researcher @ Google DeepMind, FAIR (Meta AI) and Adobe Research. 2021 BTech CS IIT Bombay.
Posts Replies Media Videos
anujdiwan.bsky.social
Thanks to my amazing collaborators Zhisheng Zheng, @eunsol.bsky.social and David Harwath!
Paper: arxiv.org/abs/2503.04713
Code: github.com/ajd12342/par...
Dataset: huggingface.co/datasets/ajd...
Model: huggingface.co/ajd12342/par...
Demo: paraspeechcaps.github.io
March 8, 2025 at 4:04 AM
anujdiwan.bsky.social
We finetune Parler-TTS-Mini-v1 on ParaSpeechCaps and achieve significant improvements in both speech-style consistency and naturalness over our best performing baseline (that combines existing smaller-scale style datasets)!
Evaluation results comparing style consistency (CMOS, Intrinsic and Situational Rich Tag Recall), speech quality (NMOS) and intelligibility (IMOS, WER). Mean score and 95% confidence intervals are reported for MOS. Our Base and Scaled models obtain improved style consistency (+5.6% and +7.9% Consistency MOS) and speech quality (+5.5% and +15.5% Naturalness MOS) over baselines.
March 8, 2025 at 4:04 AM
anujdiwan.bsky.social
ParaSpeechCaps contains 282 hrs of human-labelled data and 2427 hours of automatically-labelled data. Human evaluators rate our scaled data to be on par with human-labelled data! We carefully ablate our dataset design choices.
Human evaluation of intrinsic/situational style tag recalls, comparing our datasets and ablations. For rich intrinsic tags, PSC-Scaled (50.3%) achieves a comparable performance to PSC-Base (48.7%), while Std. Embedder (45.3%) worsens it. For rich situational tags, PSC-Scaled (71.3%) achieves a comparable performance to PSC-Base (68.1%), while removing any of Expressivity Filtering (61.0%), Semantic Matching (66.1%), or Acoustic Matching (63.3%) worsens it.
March 8, 2025 at 4:04 AM
anujdiwan.bsky.social
ParaSpeechCaps is the first large-scale dataset that supports both speaker-level intrinsic tags and utterance-level situational tags. Our key contribution is a novel pipeline for scalable, automatic style annotations over such a wide variety of rich styles for the first time.
An overview of our automatic dataset scaling pipeline, for rich intrinsic and situational tags. For intrinsic style tags, we use a perceptual speaker similarity model to identify speakers whose speech resembles that of speakers human-annotated with intrinsic tags. Then, we propagate the intrinsic tags of the similar speaker. For situational style tags, we combine three different types of signals. We first identify expressive speech using an off-the-shelf dominance-valence-arousal speech classifier. Among the selected expressive speech clips, we use a text embedding model to find transcripts that semantically match the desired situational tag. Lastly, we use a large-scale speech-text multimodal LLM to check whether the speech acoustically matches the situational tag.
March 8, 2025 at 4:04 AM
anujdiwan.bsky.social
Thanks for this list! Would appreciate being added :)
November 22, 2024 at 4:37 PM