checkout 🌐project page for more results, experiments, and details: research.nvidia.com/labs/sil/pro...

💻Code, Models, Data: coming soon

joint work with Donglai Xiang, Vismay Modi, Charles Loop, Clement Fuji Tsang, Anka He Chen, Anita Hu, Gavriel State, @diwlevin.bsky.social @shumash.bsky.social

We can also use VoMP properties to build dynamic 3d scenes with meshes demonstrating stability under gravity (see 🌐project page for comparisons with other methods) and realistic interactions with a bowling ball

or run robots through an interactive world

we can now build realistic dynamic 3D interactive worlds powered by VoMP properties:

make a 3d gaussian splat environment interactive and run a robot through it, or
simulate dynamic 3D worlds with 101, 65, and 18 deformable Gaussian Splats with collisions

VoMP can hallucinate internal volumetric structures from external renders and voxels, capture thin details, and handle noise in 3D assets (see 📜paper for training details).

MatVAE’s training (see 📜paper for training details) also yields properties useful for many other problems.

we train a latent space of physics properties, MatVAE. we train a Geometry Transformer that takes in mesh, splats, SDF, NeRF etc. and produces a per-voxel MatVAE latent

reliable high-quality training data is built by combining VLM with assets, parts, textures, material database

using physical simulation for producing dynamic 3d scenes with rich realistic interaction relies on spatially-varying physically-based mechanical properties throughout the volume of the object

these are typically laboriously hand-crafted for every object with much trial-error

we can now build realistic dynamic 3D interactive worlds powered by VoMP properties:

make a 3d gaussian splat environment interactive and run a robot through it, or
simulate dynamic 3D worlds with 101, 65, and 18 deformable Gaussian Splats with collisions

VoMP can hallucinate internal volumetric structures from external renders and voxels, capture thin details, and handle noise in 3D assets (see 📜paper for training details).

MatVAE’s training (see 📜paper for training details) also yields properties useful for many other problems.

we train a latent space of physics properties, MatVAE. we train a Geometry Transformer that takes in mesh, splats, SDF, NeRF etc. and produces a per-voxel MatVAE latent

reliable high-quality training data is built by combining VLM with assets, parts, textures, material database

using physical simulation for producing dynamic 3d scenes with rich realistic interaction relies on spatially-varying physically-based mechanical properties throughout the volume of the object

these are typically laboriously hand-crafted for every object with much trial-error