What does the system look like? We build a perception module upon visual foundation models and SLAM to build the actionable 3D relational object graph. Then we serialize graphs and input into foundation models to make decision and execute low-level robot skills. (6/9)

We show that our system can explore diverse environments, such as house-like environments and deformable object, and deploy various robot skills, including checking bottom, opening, lifting, pushing, and flipping. (5/9)

Why bothering to build the actionable 3D relational object graph?
Imagine you want your robot to collect toys spreading and being hidden in the house. This representation can not only guide robot to find all toys but also be used to collect all toys into its blanket. (4/9)

Inspired by human example, we build an **actionable 3D relational object graph** to (1) reason object relations and (2) decide actions for exploration. This clip shows how robot (1) localize unknown spaces and (2) execute skills such as opening, lifting, and pushing. (3/9)

How does human interactively explore the environment?
Human see – we often understand object relations first, such as the space **inside** the cabinet or **behind** the chair.
Human do – then we apply actions to reveal the unknown space, such as opening or pushing. (2/9)

🤔Active robot exploration is critical but hard – long-horizon, large space, and complex occlusions. How can robot explore like human?
🤖Introducing CuriousBot, which interactively explores and builds actionable 3D relational object graph.
🔗https://curiousbot.theaiinstitute.com/
👇Threads(1/9)