Read all about it!
➡️Paper: arxiv.org/abs/2506.09108

Huge team effort! Kudos to my intern Evelyn, amazing team @kmr_ayush, @aametwally1, @Orson_Xu, @timalthoff, @pushmeet, @cecim, @xliucs, @danmcduff, and other amazing co-authors!

#AI #wearable #sensor #health #multimodal
(8/8)

Beyond its discriminative power, SensorLM showcases compelling generative capabilities. It can produce hierarchical and realistic captions from input wearable data only, offering more coherent & correct descriptions compared to LLMs like Gemini 2.0 Flash. ✍️✨

(7/8)

SensorLM also demonstrates intriguing capabilities, including interesting scaling behavior over data size, model size, and compute. 📈💡

(6/8)

Experiments across real-world tasks in human activity analysis 🏃‍♀️ & healthcare ⚕️ showcase its superior performance over SOTA models in:
✨ Zero-shot recognition
✨ Few-shot learning
✨ Cross-modal retrieval

(5/8)

SensorLM extends prominent multimodal pretraining architectures (e.g., contrastive, generative) unifying their principles for sensor data. It extends prior approaches, recovering them as specific configurations within a single architecture. 🏗️🔗

(4/8)

This enabled us to curate the largest sensor-language dataset to date: over 59.7 million hours of data from >103,000 people. That's orders of magnitude larger than prior studies! 🚀💾

(3/8)

Despite its pervasiveness, aligning & interpreting sensor data with language remains challenging 📈 due to the lack of richly annotated sensor-text descriptions. 🚫

Our solution? A hierarchical pipeline captures statistical📊, structural🏗️, and semantic🧠 sensor info.

(2/8)

Science News provides a great cover of our paper: www.science.org/content/arti...

Started in 2023, delayed but finally out! Huge congrats & thanks to amazing collaborators: Yujia, @xliucs, @Avanti0609, @Mastrodicasa_MD, Vivi, @ejaywang, @sahani_dushyant, Shwetak 🎉

(6/6)
#AI #health #fairness

Why the gap? These foundation models in medical imaging encode demographic info (age, sex, race) from X-rays—more than humans do! Fascinating, but a challenge for fair healthcare ⚖️.

(5/)

This fairness disparity also holds for unseen pathologies during training, as well as for differential diagnoses across 50+ pathologies. ⚕️

(4/)

While expert-level VLMs can achieve _overall_ diagnosis accuracy on par with clinicians, they show significant underdiagnosis disparity over (intersectional) subpopulations vs. Radiologists 🚨

(3/)

We tested top vision-language models like CheXzero on 5 global datasets 🌍. Result? They consistently show disparities in diagnosis based on race, sex, and age—esp. across marginalized groups—compared to certified radiologists 📷

(2/)

Would love to be added!

Would love to be added, thanks!