We hope this can provide some insights on how to design diffusion-based NVS methods to improve their consistency and plausibility!

🧩💻🗂️ All code, data, & checkpoints are released!
🔗 Learn more: jason-aplp.github.io/MOVIS/ (6/6)

📊 We also visualize the sampling process of:

🔹 Ours (with biased timestep scheduler) ✅

🔹 Zero123 (without it) ❌

Our approach shows more precise location prediction in the earlier stage & finer detail refinement in later stages! 🎯✨ (5/6)

💡 Key insight in MOVIS: A biased noise timestep scheduler for diffusion-based novel view synthesizer that prioritizes larger timesteps early in training and gradually decreases them over time. This improves novel view synthesis in multi-object scenes! 🎯🔥 (4/6)

🔍We analyze the sampling process of diffusion-based novel view synthesizers and:
📌 Larger timesteps → Focus on position & orientation recovery
📌 Smaller timesteps → Refine geometry & appearance

👇 We visualize the sampling process below! (3/6)

In MOVIS, we enhance diffusion-based novel view synthesis with:
🔍 Additional structural inputs (depth & mask)
🖌️ Novel-view mask prediction as an auxiliary task
🎯 A biased noise scheduler to facilitate training
We identify the following key insight: (2/6)

This line highlights our work in reconstruction and scene understanding—including SSR (dali-jack.github.io/SSR/), PhyScene (physcene.github.io), PhyRecon(phyrecon.github.io), ArtGS (articulate-gs.github.io), etc.—with more to come soon!🙌🙌 (n/n)

Even more!

Our model generalizes to in-the-wild scenes like YouTube videos🎥🌍! Using just *15 input views*, we achieve high-quality reconstructions with detailed geometry & appearance. 🌟 Watch the demo to see it in action! 👇 (5/n)

🏆 On datasets like Replica and ScanNet++, our model produces higher-quality reconstructions compared to baselines, including better accuracy in less-captured areas, more precise object structures, smoother backgrounds, and fewer floating artifacts. 👀 (4/n)

🎥✨ Our method excels in large, heavily occluded scenes, outperforming baselines that require 100 views using just 10. The reconstructed scene supports interactive text-based editing, and its decomposed object meshes enable photorealistic VFX edits.👇 (3/n)

🛠️ Our method combines decompositional neural reconstruction with diffusion prior, filling in missing information in less observed and occluded regions. The reconstruction (rendering loss) and generative (SDS loss) guidance are balanced by our visibility-guided modeling. (2/n)

Checking the digest from scholar-inbox has become my daily routine. A real game-changer!👏👏👏