Summary
Argues that VLA continued pre-training should be treated as distilling robot trajectories into reusable visual-action representations rather than simply fitting more action data, and proposes a recipe (shallow-layer freezing + caption mixing to preserve VLM priors, parallel continuous action heads, physically-aligned shared action dimensions across embodiments) that outperforms naive data-scaling baselines.
Key Contributions
- Reframes VLA continued pre-training as representation distillation rather than action-fitting at scale
- Concrete recipe: shallow-layer freezing, caption mixing, multi-head parallel continuous action prediction, cross-embodiment action-dimension alignment
- Demonstrates strong downstream transfer using only open-source data and a 16-GPU budget
Strengths
- Compute-efficiency angle (16 GPUs) is a genuinely useful counterpoint to the dominant “just add more data/compute” narrative in VLA scaling papers
- Ablates specific architectural/training choices (freezing depth, caption mixing) rather than just reporting an aggregate score
Weaknesses
- “Representation-centric” framing is somewhat qualitative; the paper would benefit from more direct representation-quality probes (e.g., linear probing, representation similarity) beyond downstream task success
- Cross-embodiment action alignment claims need testing on embodiments more different than the ones evaluated
Open Questions
- Does the recipe scale favorably as pretraining data grows further, or is its advantage specific to the modest-compute regime?
- How sensitive are results to the choice of which layers are frozen?
Significance
Directly engages the “data augmentation/scaling real data pipelines for VLA” high-priority theme by showing that recipe design, not just data volume, is a major lever for VLA pretraining quality under constrained compute.