We dooooo

Finally me and Peter would like to thank all our collaborators and colleagues who helped! Especially @leslievosshall.bsky.social and the members of the Vosshall lab! This work would have never been possible without their support, help and passion for animals (with a obvious bias toward mosquitoes)

But what we are really interested in seeing is what sort of behaviors and recordings the community will try it on!

Check out FERAL

Preprint: www.biorxiv.org/content/10.1...
Code + Docs: www.getferal.ai

Feel free to contact us with any feedback, question or suggestion!

🐜 Emergent collective behavior

Finally, with @tomaskay.bsky.social we asked whether FERAL can detect collective dynamics directly from raw frames.

We annotated raiding phases in colonies of ants.
FERAL identifies the onset and duration of colony raids and collective behavior without tracking!

🦍 Camera-trap footage of chimpanzees & gorillas (PanAf500)

We pushed FERAL to handle multi-species, multi-behavior field recordings: day and night, dense vegetation, variable lighting.
FERAL achieves strong accuracy even for visually rare but distinct behaviors like climbing up/down.

This is also a great example of how tracking and FERAL are complementary:
➡️ track the animal to get WHERE it is
➡️ apply FERAL to its video crop to get WHAT it’s doing

(We think this two-step approach has great potential, so if you are interested in this stay tuned)

FERAL goes wild:
🦓 Drone footage of Grevy’s zebras (with @blaircostelloe.bsky.social, Ben Koger and @icouzin.bsky.social)

Pose-estimation pipelines struggled to detect vigilance behavior from overhead drone videos.
FERAL identifies vigilance bouts directly from video.

Ant social interactions (with Janet Zhao & Dominic Frank)

Next we wanted to try FERAL on recordings of social behavior. FERAL detects both self-grooming and allogrooming directly from raw video, even with occlusions and two animals with very different shapes.

How does FERAL perform on lab recordings?

🪱 C. elegans locomotion (with Friederike Buck)

FERAL incorporates temporal dynamics in its learning. In this dataset, it separates forward and reverse crawling: behaviors that look nearly identical frame-to-frame, but are revealed by temporal dynamics!

Performance

On benchmarks, FERAL beats both pose- and video-based baselines: 94.5% mAP on CalMS21 (mouse social interactions). Outperforms Google’s VideoPrism while using only 25% of the data.
Here’s a snippet of FERAL segmenting mouse social behavior:

Give FERAL:
1. videos
2. behavioral annotations

FERAL fine-tunes a video foundation model on your data.

FERAL outputs:
- segmented ethograms
- model weights you can reuse

All in few lines of code (no ML or coding expertise needed). It runs easily on Google Colab.

Try it yourself: www.getferal.ai

As you can see in this pic we practiced this presentation a lot and we also have some FERAL tshirts! #getferalatSfN

Sure thank you!