Summary
Introduces “1001 Demos,” a trajectory-level data augmentation framework that turns a single real-world eye-in-hand demonstration (captured with a portable single-fisheye-camera parallel gripper) into many visually realistic, physically feasible synthetic demonstrations by reconstructing the scene, optimizing collision-free alternate trajectories, and rendering matching fisheye observations via 3D Gaussian Splatting.
Key Contributions
- Combines scene reconstruction, trajectory optimization (for physically feasible, collision-free alternate paths), and 3DGS rendering into a single pipeline that augments both visual observations and action labels consistently
- Targets the failure mode of visuomotor policies breaking under minor initial-configuration changes or novel obstacles by directly generating varied initial configurations from one seed demo
- Uses a portable, low-cost eye-in-hand fisheye capture rig, keeping the data-collection hardware barrier low
Strengths
- Jointly handling visual realism (3DGS) and action-label physical feasibility (trajectory optimization) in one augmentation pipeline is more principled than augmenting pixels or actions independently
- Backed by a strong, applied research group (Stanford/Columbia/TRI) with real-robot eye-in-hand hardware experience
Weaknesses
- Reconstruction-based augmentation quality is bounded by the fidelity of the initial 3DGS scene reconstruction, which may degrade for reflective, transparent, or highly dynamic scenes
- The “one demo is worth a thousand trajectories” framing is a strong claim; the actual downstream policy improvement per augmented trajectory versus per real collected trajectory needs careful accounting to substantiate
Open Questions
- How does policy performance trained on 1001-augmented demos compare against policies trained on an equivalent budget of real diverse demonstrations?
- Does the approach extend cleanly to non-fisheye, third-person camera setups, or is it fisheye/eye-in-hand-specific?
Significance
A concrete, hardware-grounded contribution to the high-priority data-augmentation/synthetic-data theme, notable for jointly solving the visual-plus-action consistency problem that simpler augmentation methods often ignore.