Learn more about this project:
blog: ai.meta.com/blog/seamles...
paper: ai.meta.com/research/pub...
data explorer: www.aidemos.meta.com/seamless_int...
github: github.com/facebookrese...
HugginFace: huggingface.co/datasets/fac...

Current limitations: Our work is still at a research stage and we recognize limitations that still exist: more semantically-relevant gestures (closely linking gesture generation with language modeling (LLM) to allow gestures that are more semantically relevant), low latency, etc.

In line with FAIR mission, we open source Seamless Interaction Dataset to the community in order to accelerate the field. Download on github github.com/facebookrese... and @hf.co huggingface.co/datasets/fac...

We’ve trained all our dyadic motion models on Seamless Interaction Dataset. Seamless Interaction Dataset is the first-of-its-kind dataset in terms of scale and breadth, with 4000+ hours and 4000+ participants.

The output representations of our motion models are compatible with 2D and 3D renderings.

Illustrative Gestures: our models generate gestures that match the semantic content (e.g. extend the arms when saying the word “fly”).

Models can be controlled to enable more face expressiveness, with potential applications towards building more attentive or empathetic virtual listeners.

The AV Dyadic Motion models go one step further to show visual synchrony by also taking into consideration the visual input of the other party.

We built a family of Audio-Visual (AV) Dyadic Motion research models. Our models, conditioned on speech from two parties, can jointly generate facial expressions and body gestures.