Ruojin Cai
@ruojin.bsky.social
PhD Student at Cornell CS
https://www.cs.cornell.edu/~ruojin/
https://www.cs.cornell.edu/~ruojin/
We show that InterPose generalizes across 3 SOTA video models (DynamiCrafter, Runway Gen-3, Luma Dream Machine) and consistently outperforms DUSt3R on 4 diverse datasets (indoor, outdoor, object) using our new benchmark, which selects challenging pairs with little to no overlap.
December 23, 2024 at 5:44 PM
We show that InterPose generalizes across 3 SOTA video models (DynamiCrafter, Runway Gen-3, Luma Dream Machine) and consistently outperforms DUSt3R on 4 diverse datasets (indoor, outdoor, object) using our new benchmark, which selects challenging pairs with little to no overlap.
⚠️ Challenge: Generated videos may contain visual artifacts or implausible motion.
🔑 Solution: We generate multiple videos and use a self-consistency metric to select the most visually consistent sample.
🔑 Solution: We generate multiple videos and use a self-consistency metric to select the most visually consistent sample.
December 23, 2024 at 5:44 PM
⚠️ Challenge: Generated videos may contain visual artifacts or implausible motion.
🔑 Solution: We generate multiple videos and use a self-consistency metric to select the most visually consistent sample.
🔑 Solution: We generate multiple videos and use a self-consistency metric to select the most visually consistent sample.
🤔Can Generative Video Models Help Pose Estimation?
✅Yes!
We find that generative video models can hallucinate plausible intermediate frames that provide useful context for pose estimators (e.g. DUSt3R), especially for images with little to no overlap.
🔗 inter-pose.github.io
✅Yes!
We find that generative video models can hallucinate plausible intermediate frames that provide useful context for pose estimators (e.g. DUSt3R), especially for images with little to no overlap.
🔗 inter-pose.github.io
December 23, 2024 at 5:44 PM
🤔Can Generative Video Models Help Pose Estimation?
✅Yes!
We find that generative video models can hallucinate plausible intermediate frames that provide useful context for pose estimators (e.g. DUSt3R), especially for images with little to no overlap.
🔗 inter-pose.github.io
✅Yes!
We find that generative video models can hallucinate plausible intermediate frames that provide useful context for pose estimators (e.g. DUSt3R), especially for images with little to no overlap.
🔗 inter-pose.github.io