Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

1. Quick overview of the paper

Figure 1. Overview diagram: input multi-view images, robot status and text task description; output action blocks, future status and values. Later in the report, this picture will be split into two parts: "input organization method" and "training/inference responsibility allocation".

2. Motivation

2.1 What problem should be solved?

The paper is concerned about: the basic video model has learned temporal causality, implicit physics and motion patterns. Can these priors be used directly for robot control, instead of just using VLA pre-trained with static images and text as the backbone. The author hopes to obtain a unified strategy model that can output actions, imagine future states, and evaluate whether the future is worth executing.

The specific pain point here is not as general as "the robot control is not strong enough", but two things exist at the same time: first, the action distribution of many tasks is multi-modal, for example, when grabbing candy, you can start with the left hand and then the right hand, or vice versa; second, many tasks are extremely sensitive to accuracy, such as zipper bag sliders, and millimeter-level differences will lead to failure. The author believes that the video model has natural advantages in modeling complex, high-dimensional, multi-modal time series distribution, so it is worth directly migrating to control.

2.2 Limitations of existing methods

• The video strategy route is usually multi-stage: first adapt the video model to the robot data, then train the action module separately, or add inverse dynamics / action diffuser. In this way, the training chain is longer, there are more modules, and there are more sources of errors.
• Although the unified video-action model avoids a multi-module structure, it is often not initialized from a large-scale pre-trained video model, but a custom structure is trained from scratch, and the spatio-temporal prior learned from Internet videos is not available.
• VLAs are powerful, but they are mainly built on static image and text pre-training. The author's judgment is that this type of backbone is more semantic than the dynamic physical process itself, so it may not be the most natural inductive bias for high multi-modality, long time series, and fine control.
• In terms of planning, if you only look at demonstration, world model and value function, you will see mostly successful trajectories; once the actual execution enters a state of failure or deviation from the distribution, their generalization will become worse. The author emphasizes both in the text and in the planning experiment: for planning to be useful, rollout experience must be introduced.[Text §4.3]

2.3 Solution ideas of this article (high level)

The author's core idea is "Don't add a new head to the video model, but put everything needed for the robot's task back into its most familiar latent sequence." The specific method is: arrange the current observation $s$, action block $a$, future observation $s'$, and future value $V(s')$ into a latent variable frame sequence, and let the diffusion model continue to do what it is most familiar with: recovering these frames from noise.

In this way, the same model can learn three conditional distributions through different condition masks during training: policy $p(a, s', V(s')\mid s)$, world model $p(s', V(s')\mid s, a)$, and value function $p(V(s')\mid s, a, s')$. You can use it only as a direct policy during testing, or you can connect it to a best-of-N planner after fine-tuning the rollout.

3. Summary of related work

3.1 Related work of the thesis self-description

• Video-based robot policies: One type first fine-tunes the video model and then trains the independent action module; the other type directly trains the joint video-action model. The author positions himself as the third line: still retaining unified generation, but initializing directly from the large video model without structural modifications.
• Vision-language-action models: RT-2, OpenVLA, π_0.5, UniVLA, and CogVLA represent the large backbone + robot fine-tuning route. The difference in the paper is not that "there is also a large backbone", but that the backbone is a video generation model rather than a static graphic and text model.
• World models and value functions: Traditional Dyna/MBPO/TD-MPC/Dreamer to recent FLARE, SAILOR, and Latent Policy Steering all show that planning is valuable, but policy/world model/value must usually be built separately. The distinguishing point of this article is that the same pre-trained video model assumes the three functions at the same time.

3.2 Direct comparison with previous works

4. Detailed explanation of method

4.1 Method overview

The entire pipeline can be understood by "Input Organization -> Unified Training -> Two Deployment Methods":

• Input: multi-view images at the current moment, robot proprioception, text task description.
• Presentation side: The image will be transformed into a latent frame through VAE; the author then copies, normalizes and writes the action block, current/future robot state and value function into certain latent frame positions.
• Training end: Different batch samples use different condition masks, corresponding to three condition generation tasks: policy, world model, and value function.
• Deployment side: If you only do direct policy, generate $(a, s', V)$ in parallel and then only execute the action; if you do planning, take the action first, then the future state, then the value, and use best-of-N to select the action.

Figure 2. Core diagram. The author does not add action heads to the network, but inserts different modes into the latent diffusion sequence. This decision directly brings about the possibility of "learning policy / world model / value at the same time on the same network".

Appendix Figure. A more detailed implementation diagram. The blank placeholder image is first encoded into a latent frame by VAE, and then overwritten by the action/state/value; it also explains why there is a blank frame at the beginning of the sequence.[Appendix A.1]

4.2 Method evolution

4.3 Core design and mathematical derivation

Figure 12. Training distribution map of batch 50/25/25. It shows that policy, world model, and value function are not three networks, but three training instances of the same latent diffusion sequence under different condition masks.[Appendix A.4.1]

4.4 Implementation points

Figure 9. The author changed the noise distribution of the base model to a more noisy hybrid distribution to improve the test-time generation accuracy.[Appendix A.2.1]

5. Experiment

5.1 Experimental setup

Datasets and tasks

baseline method

The paper compares the diffusion policy (Diffusion Policy, Dita) trained from scratch, the video model route (UVA, UWM, Video Policy), and the VLA route (π₀, π_0.5, OpenVLA-OFT, CogVLA, UniVLA, DP-VLA, GR00T-N1.5). Focus on comparing Diffusion Policy, OpenVLA-OFT+, and π in real ALOHA₀ and π_0.5.

Evaluation index

• LIBERO / RoboCasa: success rate.
• ALOHA: Staged completion scores on a scale of 0-100, rather than binary success. The scoring details are given in detail in the appendix. For example, "put candy in ziploc bag" is divided into five 20-point stages: grabbing the zipper, grabbing the corner of the bag, opening the mouth of the bag, grabbing the candy, and putting it in the bag.[Appendix A.3.2]

Training configuration and hardware

All non-image modalities are normalized to $[-1, +1]$. The basic models are all fully parameter fine-tuned.[Appendix A.2.2-A.2.4]

Code repository and reproducibility

The paper clearly states that model checkpoints, training data and code will be released; the public entrance is the project page research.nvidia.com/labs/dir/cosmos-policy/. The Reproducibility Statement also states that training and evaluation scripts will be provided with the release.

5.2 Main results

LIBERO main results

Interpretation: Cosmos Policy is not the first in all items, Spatial is π_0.5 Higher, but it gets the best values in the Object, Goal, and Long columns, so the final average is the highest. In particular, Long mentioned OpenVLA-OFT from 94.5 to 97.6, indicating that the advantages of timing modeling emphasized by the author are mainly reflected in long-term tasks.

RoboCasa main results

Interpretation: RoboCasa The most important information in this table is not that 67.1 is only 0.7 higher than 66.4, but that it only uses 50 real-person demonstrations per task. The paper uses this table to emphasize data efficiency, not just absolute accuracy.

REAL ALOHA MAIN RESULTS

Interpretation: The average score advantage of Cosmos Policy mainly comes from the last and most difficult ziploc bag task: 85.4 compared to π_0.5 of 61.5, a huge advantage. candies in bowl on π_0.5 It is higher, but ziploc bag has higher requirements for fine control. The paper uses it as a representative case of "the video model is better at high-precision control a priori".

Figure 4. The histogram of ALOHA's four tasks visualizes that Cosmos Policy is not always far ahead in the first three tasks, but its lead is more obvious in the average score and the most difficult task.

Figure 3. Qualitative results of real dual-arm tasks are shown, indicating that the four tasks selected by the author all contain long-term, multi-modal grasping or high-precision contact operations.

Figure 5. Two types of failure modes summarized by the Authors: π_0.5 High-precision grasping on the ziploc bag is unstable; OpenVLA-OFT+ will reach between two candies in the candies task, reflecting inaccurate action distribution modeling.

IID / OOD Segmentation Results for ALOHA

This is important: 93.6 in the main results table does not mean that Cosmos Policy is the strongest in every generalization subscenario. The appendix clearly states that when broken down to the average OOD score, π_0.5 Slightly higher.[Appendix A.3.2]

Appendix Figure. ALOHA's IID initial state representation.[Appendix A.3.2]

Appendix Figure. ALOHA's OOD initial state representation. The author adds unseen T-shirts, unseen bowls, unseen ziploc bags with higher loading levels, etc. into OOD.[Appendix A.3.2]

5.3 Ablation experiment

LIBERO ablation

Interpretation: auxiliary supervision brings 1.5 points; video prior brings 3.9 points. The biggest drop is still in the Long column, which once again shows that the timing prior mainly helps long-time tasks.

RoboCasa ablation and additional experiments

There are two conclusions from this table. First, the pure action strategy dropped from 67.1 to 44.4, indicating that "letting the policy predict the future state at the same time" is not the icing on the cake, but a key component. Second, using only one denoising step can still maintain 66.4, which is almost unchanged, indicating that the number of diffusion inference steps has obvious room for compression in this task.[Appendix A.4.1]

5.4 Supplementary experiments (from appendix)

Plan an experiment

The authors only conduct planning evaluations on two more difficult ALOHA tasks: put candies in bowl and put candy in ziploc bag. The rollout data pool consists of two parts: 505 rollouts accumulated during the previous direct policy evaluation, plus an additional 143 rollouts collected for the ziploc bag, for a total of 648 rollouts. The author believes that additional ziploc data is critical because this task has severe self-occlusion, high environmental randomness, and millimeter-level motion differences that will affect the results.

Figure 6. The world model of base policy can only imagine "a successful future"; after fine-tuning rollout, the planning model can more easily imagine the real consequences of failure, so it can avoid wrong actions.

Figure 7. Compare model-based planning without planning, based on $V(s')$, and model-free planning based on $Q(s, a)$. The authors report that the model-based version achieves an average score improvement of 12.5 points on two difficult tasks.

The author compares two forms of value training:

• $V(s')$: The world model first predicts the future state, and then scores the future state. This is the model-based version.
• $Q(s, a)$: Score the current state and actions directly, without the need for an explicit world model. This is the model-free version.

The result is that $V(s')$ is better. The author's own explanation is: when the rollout data is very limited, explicitly using future states and environmental dynamics is more sample efficient than directly learning high-dimensional Q functions, and is less likely to overfit.

Reasoning delay

This is also the most realistic cost of the paper: planning is useful, but very slow. The author lists this as the primary limitation in the Discussion.

6. Analysis and Discussion

6.1 Analysis and explanation of the results given in the paper

• The authors argue that the video model prior is an efficient initialization and thus can outperform partially strong VLA on real robots even without additional large-scale action labels. The text directly attributes this to the video model learning spatiotemporal dynamics and implicit physics, not just static semantics.
• The author uses the failure case in Figure 5 to explain why high multi-modality and high-precision tasks are a watershed: a deviation will pull the action between the two targets, or miss in millimeter-level grasping.
• The author uses Figure 6 to illustrate that the planning model after rollout fine-tuning can better predict the consequences of failure, so the planning stage can avoid mistakes that are easy to make with basic strategies, such as the slipper of a zipper bag.
• The value of auxiliary supervision and joint objective is clearly verified in ablation: not all modules are equally important, but "letting policy predict future states together" is very critical.

6.2 Limitations of the author's statement

6.3 Applicable boundaries and discussions clearly stated in the paper

• The "state" in this article is actually closer to observation, because the real environment is not completely observable; the author specifically states this in Preliminaries for the sake of simplicity. In other words, the world model predicts future observations, not the complete state in the strict sense.
• The benefits of planning are mainly shown in difficult tasks where the basic direct policy still has obvious room for error; for the first two easier tasks of LIBERO or ALOHA, the paper does not emphasize further planning gains.
• Both RoboCasa and ALOHA embody methods that can do multi-perspective input, but do not build in the input history; the state only takes the current moment, and only predicts $t+K$ in the future, so the model is not a long horizon autoregressive world model.
• The future work given at the end of the paper also strictly focuses on the original method: accelerating search, reducing rollout requirements, and extending deeper planning depth.