📢 My lab at UCLA is hiring PhD students and postdocs!

Please apply to UCLA CS or CompMed and mention my name if you are interested in foundation models and (Gen)AI for health / medicine / science.

More info: cs.ucla.edu/~yuzhe

Beyond its discriminative power, SensorLM showcases compelling generative capabilities. It can produce hierarchical and realistic captions from input wearable data only, offering more coherent & correct descriptions compared to LLMs like Gemini 2.0 Flash. ✍️✨

(7/8)

SensorLM also demonstrates intriguing capabilities, including interesting scaling behavior over data size, model size, and compute. 📈💡

(6/8)

Experiments across real-world tasks in human activity analysis 🏃‍♀️ & healthcare ⚕️ showcase its superior performance over SOTA models in:
✨ Zero-shot recognition
✨ Few-shot learning
✨ Cross-modal retrieval

(5/8)

SensorLM extends prominent multimodal pretraining architectures (e.g., contrastive, generative) unifying their principles for sensor data. It extends prior approaches, recovering them as specific configurations within a single architecture. 🏗️🔗

(4/8)

This enabled us to curate the largest sensor-language dataset to date: over 59.7 million hours of data from >103,000 people. That's orders of magnitude larger than prior studies! 🚀💾

(3/8)

Despite its pervasiveness, aligning & interpreting sensor data with language remains challenging 📈 due to the lack of richly annotated sensor-text descriptions. 🚫

Our solution? A hierarchical pipeline captures statistical📊, structural🏗️, and semantic🧠 sensor info.

(2/8)

🚨 Let your wearable data "speak" for themselves! ⌚️🗣️

Introducing *SensorLM*, a family of sensor-language foundation models, trained on ~60 million hours of data from >103K people, enabling robust wearable sensor data understanding with natural language. 🧵

Why the gap? These foundation models in medical imaging encode demographic info (age, sex, race) from X-rays—more than humans do! Fascinating, but a challenge for fair healthcare ⚖️.

(5/)

This fairness disparity also holds for unseen pathologies during training, as well as for differential diagnoses across 50+ pathologies. ⚕️

(4/)

While expert-level VLMs can achieve _overall_ diagnosis accuracy on par with clinicians, they show significant underdiagnosis disparity over (intersectional) subpopulations vs. Radiologists 🚨

(3/)

We tested top vision-language models like CheXzero on 5 global datasets 🌍. Result? They consistently show disparities in diagnosis based on race, sex, and age—esp. across marginalized groups—compared to certified radiologists 📷

(2/)

Do foundation models in medical imaging see everyone fairly?🤔

Excited to share our new Science Advances paper uncovering & auditing demographic biases of expert-level VLMs, and comparing to board-certified radiologists🧑‍⚕️

📄science.org/doi/10.1126/sciadv.adq0305
💻github.com/YyzHarry/vlm-fairness
(1/)

Hello world! I’m recruiting ~3 PhD students for Fall 2025 at UCLA 🚀

Please apply to UCLA CS or CompMed if you are interested in ML and (Gen)AI for healthcare / medicine / science.

See my website for more on my research & how to apply: people.csail.mit.edu/yuzhe