Joint work by our awesome research intern Qihang Zhang, together with colleagues Shuangfei, Miguel, Kevin, Alex and Josh at Apple MLR!

Want to dive deeper? Check out our paper for full details

ArXiv: arxiv.org/abs/2412.01821
Project page: zqh0253.github.io/wvd/ (9/n, n=9)

WVD also supports controllable video generation. Given a single image, we estimate the 3D geometry via standard WVD inference, and project it to get partial XYZ images. Finally, WVD generates the RGB images jointly with the projected XYZ images through in-painting. (6/n)

For example, WVD can be directly applied to various single-image tasks. WVD can also take unposed images (video) as input, and infer XYZ images via “in-painting” strategy. With a post optimization procedure, the XYZ images can be converted to camera poses, and depth maps. (5/n)

At inference time, this joint distribution can be leveraged to estimate conditional distributions, such as P (XYZ | RGB) or P (RGB | XYZ). This capability makes WVD a foundation for supporting a wide range of downstream tasks. (4/n)

During training, WVD learns to generate 6D (RGB + XYZ) videos by modeling the joint probability P (RGB, XYZ), effectively capturing their interdependent structures and features. (3/n)

Existing multi-view/video diffusion model usually lack explicit 3D supervision (or guarantee), leading to potential 3D inconsistency and inefficient training.

In contrast, WVD models multi-view images, and explicit 3D geometry. Specifically, we represent the 3D geometry via XYZ images. (2/n)

More interesting research work 🤔