RAPID: Robot Agentic Programming from Demonstrations

Abstract

Coding agents have demonstrated enormous success in solving complex programming problems. We introduce Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs given a single visual human demonstration. Closing this agentic loop requires a testable task specification, manipulation primitives, and an interactive verification environment. RAPID infers all three from the demonstration. To make the resulting program reusable, RAPID uses an object-centric relational program representation that captures the underlying structure of the demonstrated strategy. Primitives are trajectory-optimization programs that realize object-level effects, composed through relational constraints that resolve scene-specific geometry at runtime. We evaluate RAPID on eight contact-rich nonprehensile manipulation tasks in simulation and on a real Franka arm, and further evaluate its generality on LIBERO-Pro.

The method

Closing the agentic coding loop

Demonstrations as the programming interface

01

Construct & verify

Infer the task specification, reconstruct an interactive simulation, and generate and refine manipulation programs.

02

Represent the strategy

Express object-level effects as trajectory-optimization programs. Relational constraints compose successive primitives.

03

Instantiate & execute

Bind new scene objects to semantic roles in just a few seconds without needing to change the strategy and primitives, then optimize and execute.

Object-centric relational program representation

Building reusable and generalizable manipulation primitives and strategy

Real-world experiments

8 long-horizon, contact-rich nonprehensile tasks.

The robot uses contact with the surrounding scene to grasp objects or move them into narrow spaces.

Example 1 / 3

Task 03 / 08

Reposition, flip, and grasp

Move the object clear of surrounding obstacles. Then push it toward the fixture, flip it, and grasp the exposed edge.

Press move→Push→Flip→Grasp

Franka Research 3 · 10× speed

Simulation evaluation

Object-centric relational programs improve generalization.

50 novel scenes per task, with changes in object pose, appearance, geometry, material, and scene configuration.

75.9%

Average success rate

Across eight nonprehensile tasks.
Mean ± standard deviation: 75.9 ± 2.4% over three trials.

8tasks

50scenes / task

1demo / task

Success rate

0%25%50%75%100%

OReP: object-centric relational programs. SV: scene variants.

BibTeX

@article{liu2026rapid,
  title   = {RAPID: Robot Agentic Programming from Demonstrations},
  author  = {Liu, Yuyao and Mao, Jiayuan and Hsu, David and
             Kaelbling, Leslie Pack and Lozano-Pérez, Tomás},
  journal = {arXiv preprint arXiv:2609.30249},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.30249}
}

Walkthrough Video