HJ Hanxiao Jiang RoboEXP GitHub Open source
Exploration Atlas / 6 scenes

Explore. Remember. Manipulate.

01

Iterative exploration

02

Recursive discovery

03

Obstruction reasoning

04

Multi-stage exploration

05

Scene intervention

06

Targeted re-exploration

01–03 / 06
Drag or use arrows

RoboEXP can zero-shot explore unseen real-world environments, reveal hidden structure through interaction, and reuse the resulting memory for downstream manipulation.

01 Discover the Scene

Observe less. Discover more.

We introduce the task of interactive scene exploration, where a robot autonomously explores an environment and produces an Action-Conditioned Scene Graph (ACSG) that captures its underlying structure. Unlike a static reconstruction, the ACSG records both low-level geometry and semantics and high-level relationships that only become visible through physical interaction.

RoboEXP combines a Large Multimodal Model with an explicit memory design. It reasons about what to explore, how to interact, and when new evidence should update its model of the scene.

Explore first. Act with context.

Interactive exploration constructs the ACSG; the same graph then supports the downstream table-setting task.

RoboEXP / real-world run RGB-D → ACSG → manipulation
The robot incrementally uncovers objects and relations, accumulates an actionable scene memory, and plans with that memory instead of treating every observation in isolation.
02 Construct the ACSG

RoboEXP

A closed loop between perception, explicit memory, decision-making, and physical action.

RoboEXP pipeline with perception, memory, decision-making, and action modules
Perception Explicit memory Decision-making Action

Starting from RGB-D observations, vision foundation models ground objects while the memory module fuses observations into the ACSG. A decision-making module reasons over that graph to propose the next exploration plan, and the action module executes it. Every interaction returns new evidence to the same structured memory.

Two views of one evolving scene.

The high-level graph remains actionable while low-level memory anchors its nodes and relationships in 3D.

High-level memory

ACSG construction

Dynamic

Objects and action-conditioned relationships are added as the robot explores.

Low-level memory

3D scene state

Interactive
RoboEXP low-level scene memory
Interactive visualization · 15 MB

Geometry and semantic observations ground graph reasoning in the physical scene.

03 Adapt and Transfer

Update the memory when the world changes.

RoboEXP detects new or moved objects, identifies the affected region, and selectively re-explores it.

Introducing a new cabinet

01 Detect the new structure and extend the scene graph.

Moving objects between doors

02 Track the change and re-explore only where needed.

The same representation, now on the move.

The interactive exploration pipeline transfers to a Stretch mobile manipulator for household scenarios.

Explore inside drawers

01 Open articulated storage to discover what is inside.

Explore beneath cloth

02 Interact with deformable objects to reveal hidden items.

Generalize across layouts and object types.

Zero-shot exploration with rigid, articulated, nested, occluded, and deformable objects.

Evaluation reel / diverse settings Zero-shot

See the full system in action.

From autonomous exploration to graph construction and downstream manipulation.

BibTeX

If RoboEXP supports your research, please cite the CoRL 2024 paper.

@inproceedings{jiang2024roboexp,
  title     = {RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation},
  author    = {Jiang, Hanxiao and Huang, Binghao and Wu, Ruihai and Li, Zhuoran and Garg, Shubham and Nayyeri, Hooshang and Wang, Shenlong and Li, Yunzhu},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2024}
}