OpenAi Ghibli style + the new FramePack (ControlNet team). I am very impressed by this model, and it was super easy to run. Is it a commoditization moment for video GenAi?

This work was led by Amir Barda, in collaboration with Matheus
@gadelha.bsky.social
, Noam Aigerman
@noamiko.bsky.social
, Vova Kim @vovakim.bsky.social and Amit Bermano.
Check out our paper for more details 📜: arxiv.org/abs/2412.00518
7/end

In the end, with the editing tool becoming FAST 🐇 , 3D editing becomes really FUN to play with! 6/

Do we teach inpainting to a multiview backbone 🤔? Or do we teach multiview to an inpainting backbone? We show that the latter is much better. Multiview is easier to learn than inpainting. 4/

Now, all we need is a multiview inpainting model 😅. How do we train one? Data is always king. We know inpainting masks can’t be random; they need to be realistic and close to what users would do. We propose 3 strategies, in 3D, to create Objaverse masks, closely resembling what a user would do. 4/

However, SDS remains slow and brittle 🐢💥. Instead, we propose to cast the problem of 3D inpainting as 2D *multiview* inpainting 📸-📸-📸-📸. This is possible thanks to off-the-shelf pre-trained transformer models (LRM), which reconstruct multiview image back to meshes, Gsplats, and NeRFs. Great! 3/

There has been previous attempts to tackle generative mesh editing. Check out Amir Barda’s talk on MagicClay this Thursday at Siggraph Asia, Japan 🇯🇵, using SDS. 2/

🌟 Text-to-3D is awesome ! But how do we iterate on the generated 3D model, to get just the right result? Do we tweak the prompt endlessly? Revert to traditional 3D modeling techniques?
We propose a solution to "3D inpainting” 🤩🎨

Project: amirbarda.github.io/Instant3dit....
A thread. 🧵 1/