This project was part of my internship at Meta, and it was a great collaboration with Jia-Bin, Zhengqin, Zhao, Christian, Tuotuo, Michael, Johannes, Shenlong, and Changil 🙌

📍 Meet us at #CVPR2025
🗓️ June 13 (Fri.), 10:30–12:30
🪧 Come by our poster session to chat about IRIS and any research ideas
Looking forward to reconnecting — and meeting new friends — in Nashville! 🎸✨

Please check out our paper, code, and more demo videos!
🌐 Project page: irisldr.github.io
💻 GitHub: github.com/facebookrese...
📝 Paper: arxiv.org/abs/2401.12977

This work is a great collaboration at NVIDIAAI by Chih-Hao Lin, Zian Wang, Ruofan Liang, Yuxuan Zhang, Sanja Fidler, Shenlong Wang, Zan Gojcic

🌐Please check out our project page: research.nvidia.com/labs/toronto...

The weather removal model successfully removes both transient (e.g., rain, snowflake) and persistent effects (e.g., puddle, snow cover), and can even restore sunny-day lighting from rainy/snowy videos.

WeatherWeaver combines two video diffusion models. The weather synthesis model generates realistic, temporally consistent weather, adapting shading naturally while preserving the original scene structure.

By combining and adjusting multiple weather effects, WeatherWeaver can simulate complex weather transitions, e.g. on 🌧️ rainy and ☃️ snowy days — without costly real-world acquisitions.

WeatherWeaver enables precise control of the weather effects by changing the intensity of the corresponding effects. 🌤️➡️🌥️

We train a video diffusion model to edit weather effects with precise control, using a novel data strategy combining synthetic videos, generative image editing, and auto-labeled real-world videos.

Realistic, controllable weather simulation opens new possibilities in 🚗 autonomous driving simulation and 🎬 filmmaking. Physics-based simulation requires accurate geometry and doesn’t scale to in-the-wild videos, while existing video editing often lacks realism and control.

Check out our cool demos. The code is also open-source!
Project website: haoyuhsu.github.io/autovfx-webs...
Code (GitHub): github.com/haoyuhsu/aut...
Paper: arxiv.org/abs/2411.02394

🙏 Huge thanks to our amazing collaborators from UIUC & UMD: Bohan, Yi-Ting, Kuan-Sheng, David, Jia-Bin (@jbhuang0604.bsky.social) , Anand (@anandbhattad.bsky.social) , and Shenlong. This work wouldn't be possible without you all.