Beyond Reconstruction

A Real2Sim2Real Harness for Robot Policy Improvement

Parv Kapoor · Zifu Wan · Kaylee Kirkelie · Srujan Deolasee · Ashish Kapoor · Sai Vemprala

Read the paper
From images to real-world policy improvementPhase 1 reconstructs scene geometry and completes interactions from a single image, functionality descriptions, and priors, with geometric and physical validation. The resulting interactive simulator supplies head-camera RGB observations to the same frozen base policy used for real observations. White and red arrows illustrate the real-input and simulated-input predicted end-effector motions; their gap guides targeted simulator corrections in appearance, viewpoint, embodiment, and state feedback. The action arrows are schematic, not measured data. Once aligned, the simulator is fixed, a residual is learned from simulated oracle demonstrations, and the base plus residual is deployed in reality. Live clips show the socket probe, a simulated rollout from the head camera, and the real robot running base plus residual.
FIGURE 1 Build an interactive world, align it to the frozen policy, then improve the policy. Select an image to enlarge it, or a stage to jump to it.

Approach

A simulator must reproduce a policy’s behavior before it can improve it.

Geometric and physical validity

Claude drives twelve checkpointed reconstruction stages through an MCP server: Depth Anything V3, SAM3, Hunyuan3D 2.1, FoundationPose and GPT-6 Astra, with five review gates and checks for completeness, registration, scale, penetration and settling.

Interaction validity

From a plain-language description, the agent fits hinges, socket cavities or threads to the reconstructed mesh, keeps its appearance, and tests each mechanism with a physics probe.

Objects and scenes

Behavioral fidelity

The frozen π0.5 policy is the discriminator: compare its actions at matched real and simulated states and correct the simulator until it reproduces real behavior. Then learn a small residual in simulation.

Alignment

Reconstruction

Objects and scenes from photographs, with sockets, threads and hinges added.

Power strip

Real / source photograph

Real power strip with yellow sockets.

Simulation / robot-free insertion

Reconstructed strip with authored socket cavities. A three-prong probe seats to 15.9 mm and holds; rotated and offset plugs are rejected.

Robot-free probe of the delivered mechanism. Socket geometry.

Jar

Real / source photograph

Real clear jar with metal lid.

Simulation / thread mechanism

Reconstructed jar with a procedural thread: torque unscrews the lid, an axial pull does not.

Fixture test, no robot. Reconstructed asset.

Box

Real / source photograph

Real cardboard box.

Simulation / corrected object

Reconstructed box with corrected articulated flaps.

Hinged lid and side flaps fitted to the reconstructed box.

Rigid-panel approximation. Folded fit.

Pickup scene

Real / input photograph

Phone photograph of the robot bench with a small camera-shaped object beside a blue plastic bin.

Simulation / reconstructed scene

Preview of the pickup scene; load the 3D view to inspect the reconstructed environment.
Load the reconstructed bench, pickup object and bin.

Interactive reconstruction of the pickup bench, object and bin; robot omitted.

Tabletop reconstruction

Real / input photograph

Real tabletop with mugs, a bottle, paddle and pen.

Simulation / camera tour

Input photograph beside the reconstructed tabletop.

Camera tour; no manipulation shown.

Reconstructed tabletop with mugs, a bottle, paddle and pen.
Full tabletop scene · 78 MB · loads on request.

Desk scene from one photograph

Real / input photograph

Desk with a clear ribbed cup, an ibuprofen bottle, a red pen, a black pen and a black keyboard.

Simulation / settled scene

Reconstructed desk scene: cup, bottle, two pens and keyboard on a support slab.
Five objects · 15.6 MB · loads on request.

One phone photograph, five simulated rigid bodies that settle in 142 steps.

Preview: the name plate was removed because the physics stability check found no predicted support for it, so it was not stable in simulation. Masses are VLM estimates.

Robot interactions

Real recordings beside their reconstructed simulations.

Camera pickup

Real / wrist camera, pick and place

Simulation / learned residual, wrist + overview

Real pick and place beside the learned residual grasping in simulation.

Different trials. Simulation at 2× from a saved grasp state.

Box folding

Real / demonstration 0133

Simulation / scripted oracle

Real demonstration beside the scripted folding sequence in simulation.

Scripted oracle, not a learned policy.

Plug insertion

Real / teleoperated insertion, base + wrist

Simulation / learned residual

Real teleoperated insertion beside a learned-residual rollout in simulation.

Teleoperated demonstration, not a policy rollout. Simulation at 3×.

Jar opening

Real / bimanual opening, wrist cameras

Simulation / bimanual scripted oracle

One arm holds the jar while the other turns the lid.

Scripted oracle in simulation; different trials.

Policy-guided alignment

The first render passed every check, yet the frozen policy started acting in only 6/12 simulated observations. Using the policy as the judge, appearance corrections brought that to 12/12.

Accepted changes, in order · head camera as the policy saw it · home-state descent unless noted

Real robot head-camera frame at the home state.
ReferenceReal head camera12/12
Simulated head camera with the initial pitch of about 50 degrees.
ViewpointInitial fit, pitch ≈50°grasp +2.6 cm
Simulated head camera refit to 36.7 degrees pitch, uncorrected colour.
ViewpointRefit to 36.7°, raw colour6/12
Simulated head camera after a per-channel colour affine.
Appearance+ colour affine on head12/12
Simulated head camera with hard shadows disabled.
Appearance+ hard shadows off10/10
Tried and rejected
Simulated scene in a dark room.
AppearanceDark room4/6
Head view cropped to the real table footprint.
ViewpointCropped to real table0/12
Board extended under the robot.
EmbodimentBoard under the robot0/3 episodes
Robot mounting plate added.
EmbodimentRobot mounting plate4/10
Search space · plug insertion, π0.5 checkpoint 14999
AxisSearchedKept
AppearanceColour gain, gamma, blur and per-channel affine on each camera; shadows; room lighting; table cropPer-channel colour affine on the head camera only; hard shadows off
ViewpointHead-camera extrinsics, fit by least squares to landmarks in a real framePitch 36.7°, yaw −1.6°, 0.62 m above the table. The first fit (≈50°) put every grasp 2.5–4 cm toward the robot
EmbodimentRobot base pose (9 variants), wrist-camera housing, gripper forceBase z +3 cm, y −2.5 cm; CAD wrist housing; ±20 N grip
State and promptGripper reading fed to the policy; instruction stringReal finger readout; the training prompt

Insertion still failed after alignment, by as little as 7 mm. The residual closes that gap.

Residual adaptation

Two tasks, two π0.5 checkpoints. The base stays frozen; a residual trained in simulation corrects it.

Frozen base vs. base + residual
TaskSettingFrozen baseWith residual
Plug insertionSimulation7 / 2013 / 20
Plug insertionReal robot6 / 2011 / 20
Camera pickupSimulation2 / 2414 / 24
Camera pickupReal robot2 / 2013 / 20

Plug insertion

A residual adds a sideways shift of up to 20 mm to π0.5’s actions during insertion.

Real robot · without gate
Phone recording, real time.
Real robot · with gate
Phone recording, real time.
Base · episode 0 · failure
One of the matched initial configurations.
Residual · episode 0 · geometric success
Selected illustration.

Top: one real trial per condition, real time. Bottom: simulation, one matched start, 3× speed.

Camera pickup

Away from its demonstrated placement the base policy misses; the residual moves the grasp onto the object.

Frozen π0.5 · 2 / 24 held

-20020406080-50-2502550forward offset (mm)sideways (mm)seed 60000 · forward 40 mm · sideways 18 mm · missseed 60001 · forward -11 mm · sideways -32 mm · missseed 60002 · forward 45 mm · sideways 18 mm · heldseed 60003 · forward 38 mm · sideways -2 mm · missseed 60004 · forward 76 mm · sideways 8 mm · missseed 60005 · forward -16 mm · sideways -1 mm · missseed 60006 · forward 50 mm · sideways 37 mm · missseed 60007 · forward 44 mm · sideways 5 mm · missseed 60008 · forward 45 mm · sideways 31 mm · missseed 60009 · forward -11 mm · sideways 49 mm · missseed 60010 · forward 86 mm · sideways 49 mm · missseed 60011 · forward 87 mm · sideways 6 mm · missseed 60012 · forward 41 mm · sideways -11 mm · missseed 60013 · forward 37 mm · sideways -26 mm · missseed 60014 · forward 33 mm · sideways 48 mm · heldseed 60015 · forward -15 mm · sideways -26 mm · missseed 60016 · forward -9 mm · sideways -36 mm · missseed 60017 · forward -20 mm · sideways 29 mm · missseed 60018 · forward 66 mm · sideways 39 mm · missseed 60019 · forward -5 mm · sideways 14 mm · missseed 60020 · forward 84 mm · sideways -21 mm · missseed 60021 · forward 85 mm · sideways 11 mm · missseed 60022 · forward 67 mm · sideways -4 mm · missseed 60023 · forward 50 mm · sideways -34 mm · miss

π0.5 + residual head · 14 / 24 held

-20020406080-50-2502550forward offset (mm)sideways (mm)seed 60000 · forward 40 mm · sideways 18 mm · heldseed 60001 · forward -11 mm · sideways -32 mm · lifted onlyseed 60002 · forward 45 mm · sideways 18 mm · heldseed 60003 · forward 38 mm · sideways -2 mm · missseed 60004 · forward 76 mm · sideways 8 mm · heldseed 60005 · forward -16 mm · sideways -1 mm · lifted onlyseed 60006 · forward 50 mm · sideways 37 mm · heldseed 60007 · forward 44 mm · sideways 5 mm · heldseed 60008 · forward 45 mm · sideways 31 mm · missseed 60009 · forward -11 mm · sideways 49 mm · heldseed 60010 · forward 86 mm · sideways 49 mm · heldseed 60011 · forward 87 mm · sideways 6 mm · heldseed 60012 · forward 41 mm · sideways -11 mm · heldseed 60013 · forward 37 mm · sideways -26 mm · lifted onlyseed 60014 · forward 33 mm · sideways 48 mm · heldseed 60015 · forward -15 mm · sideways -26 mm · heldseed 60016 · forward -9 mm · sideways -36 mm · missseed 60017 · forward -20 mm · sideways 29 mm · lifted onlyseed 60018 · forward 66 mm · sideways 39 mm · missseed 60019 · forward -5 mm · sideways 14 mm · heldseed 60020 · forward 84 mm · sideways -21 mm · lifted onlyseed 60021 · forward 85 mm · sideways 11 mm · missseed 60022 · forward 67 mm · sideways -4 mm · heldseed 60023 · forward 50 mm · sideways -34 mm · held

heldlifted onlymissedOffset from the object’s position in the demonstrations.

Simulation / far placement, seed 60010

π0.5 alone (left) hovers short of the object; with the residual head (right) it reaches and holds.

Simulation / near placement, seed 60015

π0.5 alone never grasps; with the residual head it picks and holds.

Simulation at 3× speed; selected clips illustrating the maps above.

On the real bench

  1. 0:00–0:14PlacedGrasp beside the bin, into the bin.
  2. 0:16–0:23PlacedGrasp, short carry, release.
  3. 0:28–0:37PlacedObject further away; grasp and place.

With the correction, the policy picks and places three times in a row, which the base policy never managed.

References

NVIDIA. SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation. 2026.

Li et al. Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024.

Torne et al. Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation. RSS 2024.

Luo et al. SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images. arXiv 2026.