World Action Models are Zero-shot Policies

1. Quick overview of the paper

2. Motivation

2.1 What problem should be solved?

The author believes that the current mainstream route for robot foundation models is VLA: extending pre-trained VLM to action output. This route is very good at semantic generalization, such as understanding complex language, identifying new objects, and aligning "what to do" to language knowledge; but it is not good at "how to do", that is, new physical actions, environmental changes, and fine motor control. The typical counterexample given at the beginning of the paper is: VLA can understand "move the Coke can to Taylor Swift" because it knows who and where Taylor Swift is; but if "untie the shoelace" has never been in the robot data, VLA will often not grow this new action on its own.

Therefore, what the paper wants to solve is not simply "multi-task robot control", but a more acute problem: whether the basic robot model can learn new skills from a data distribution more like the real world without specifically collecting a large number of repeated demonstrations, and generalize it to zero samples on new tasks, new environments, and even new embodiments.

2.2 Limitations of existing methods

• Pre-trained partial static semantics of VLA: What VLM learns from static graphics and text are semantic concepts, not precise spatiotemporal dynamics. The authors explicitly state that this leaves them lacking geometric, dynamic, and motor control representations of action execution.
• VLA often relies on large, repetitive, mission-centric data: To perform stably on robots, existing VLAs often require a lot of embodiment-specific robot data, and these data are usually a large number of repeated demonstrations.
• The old WAM has not fully exploited the generalization potential.: Existing video-action joint models have proven that the world modeling goal is helpful, but many works are still verified in repeated demonstration scenarios, or do not systematically study data diversity, architectural form, and real-time reasoning.
• It is very slow to directly use the video model for closed-loop control: Video diffusion itself requires multi-step denoising and has a large number of parameters, so it is naturally not suitable for millisecond-level reactive control. The authors single this out as one of the three core challenges.

2.3 Solution ideas of this article (high level)

DreamZero's proposition is: Instead of letting the model directly clone the "observed actions", it is better to let it predict future world states and actions at the same time, turning action learning into an implicit inverse dynamics guided by the visual future. Intuitively, if the model already has a good idea of ​​what the world should look like next, then translating this future visual plan into actions should generalize more easily than directly regressing actions from current observations.

To this end, the author uses a pre-trained video diffusion model as the backbone, allowing the model to jointly generate future videos and actions given language, visual history, and ontology status. At the same time, autoregressive video generation, teacher forcing, KV cache, and real observations are used to replace the closed-loop reasoning of predicted frames to avoid the common error accumulation problem in pure video AR.

3. Summary of related work

3.1 Related work of the thesis self-description

• Vision-Language-Action Models: The paper puts RT series, OpenVLA, GR00T, π series, etc. into this line. They put language semantics and action output in the same end-to-end model, but the pre-training priors mainly come from static images and text.
• Video Model-based Robot Policies: One type is to generate a video first and then use inverse dynamics/optical flow/planning to extract actions; the other type is to directly generate video-action jointly. The authors collectively refer to the latter as WAM.
• Alternative world model architecture: In the appendix, the author also specifically compares latent-space world model and 3D point-cloud world model. The author believes that most of these solutions learn from $p(s_{t+1}\mid s_t, a_t)$, and are deployed with inverse dynamics or explicit search; WAM directly learns joint distribution and outputs action trajectories.

3.2 Direct comparison with previous works

4. Detailed explanation of method

5.1 Method overview

DreamZero's calculation graph can be understood at four levels: "input organization, joint modeling, closed-loop execution, and real-time optimization":

• Input: visual history $\mathbf{o}_{0: l}$, current proprioception $\mathbf{q}_l$, language command $\mathbf{c}$.
• Model: pre-trained video diffusion backbone + a small number of new state encoder, action encoder, and action decoder.
• Goal: Jointly predict future video chunk $\mathbf{o}_{l: l+H}$ and action chunk $\mathbf{a}_{l: l+H}$.
• Deployment: The action block is executed asynchronously. After execution, the real observations are rewritten into the KV cache to replace the visual future just generated by the model, thereby preventing video AR errors from snowballing.

5.2 Method evolution

5.3 Core design and mathematical derivation

Figure 11. Flash noise schedule graph. Red is Flash: the video has high noise, but the movements are still uniform; blue is standard DreamZero: the video and movements are scheduled uniformly.

5.4 Implementation points

• Multi-view processing is simple: Multi-view is not about changing the backbone structure, but directly splicing different views into a single frame and sending it to the video model.
• Only the video is AR, the action is not pure AR rollout: The author said that this can avoid the error propagation of closed-loop action prediction while retaining the speed benefits brought by KV cache.
• Chunk design and control frequency are strongly bound: AgiBot is 5FPS video, 30Hz motion, 48-step horizon, exactly 1.6 seconds per chunk; DROID is 5FPS video, 15Hz motion, 24-step horizon, also 1.6 seconds. The total context length defaults to 4 chunks, corresponding to 6.6 seconds of visual history.[Appendix Model Details]
• Training update range: Update all DiT blocks, state encoder, action encoder, action decoder; freeze text encoder, image encoder and VAE. The author also specifically mentions that LoRA was tried but not very effective.[Text §4.1]
• Asynchronous execution is a necessary condition: Not to be faster and better looking, but to change the reactive constraint from "reasoning must precede action" to "reasoning only needs to be completed before the current action chunk expires". The target latency given by the author is less than about 200ms.
• action chunk smoothing: The generated action blocks are first upsampled by 2x, then filtered with Savitzky-Golay, and finally downsampled back to the original resolution to specifically suppress high-frequency jitter.[Appendix Real-time execution]

5. Experiment

6.1 Experimental setup

Datasets and Platforms

training configuration

baseline

The main baselines are two types of VLA: GR00T N1.6 and π_0.5. Each baseline is divided into two types of initialization:

• from-scratch: Only take pre-trained VLM weights, without using robot pre-training checkpoint, and compare "apples to apples" with DreamZero.
• from-pretrained: Use the official robot pre-training weights, and then continue training on the author's data.

Evaluation Agreement

• AgiBot seen tasks: 10 seen tasks, 4 robots, 8 rollouts per task, a total of 80 times/checkpoint; divided into three categories: PnP Easy, PnP Hard, and Contact-Rich.[Appendix AgiBot Eval]
• AgiBot unseen tasks: 10 tasks outside the training distribution, such as untie shoelaces, ironing, painting, shaking hands, pulling cart, etc., the same 80 times/checkpoint.[Appendix AgiBot Eval]
• DROID: 20 seen tasks + 20 unseen-verb tasks, each task has 2 rollouts, a total of 80 times; the indicator looks at both task progress and success rate.[Appendix DROID Eval]
• post training task: shirt folding, fruit packing, table bussing; 10 rollouts per task, the indicator is task progress.

Task list given in appendix

6.2 Main results

Q1: Can strategies be learned from heterogeneous and non-duplicate data?

The authors first look at zero-shot environment generalization on the AgiBot seen task. The core conclusion is: DreamZero's average task progress reaches 62.2%, while the best pretrained VLA baseline has only 27.4%; from-scratch VLA is almost zero. The author explains this result as: WAM uses video prediction priors and does not need to hard-learn dynamics from sparse state-action correspondences, so it is better able to digest heterogeneous data.

Q2: Can zero samples be generalized to unseen tasks?

On the AgiBot unseen task, DreamZero achieves an average of 39.5% task progress, while the pretrained VLA baseline is 16.3%, the from-scratch VLA is less than 1%. The author specifically mentions Remove Hat from Mannequin with 85.7% and Shake Hands with 59.2%.

Under the DROID setting, DreamZero is also clearly ahead: 49% task progress, 22.5% success rate; GR00T N1.6 is 31% / 12.5%, π_0.5 It's 33% / 7.5%. The author here emphasizes unseen verbs, that is, the verbs in the command themselves do not appear in the training set.

Q3: Can generalization be retained after task-specific post-training?

The author does post-training on three tasks: shirt folding, fruit packing, and table bussing. The paper does not give a precise numerical table of each method in the main text, but the conclusion in the figure is clear: DreamZero at least matches VLA on all three tasks, and is significantly better on fruit packing. The author uses this set of experiments to support that "environmental generalization does not disappear after task-specific post-training".

Q4: Is video-only cross-embodiment transfer useful?

Just adding 10-20 minutes of video-only demonstration can pull the average progress of unseen tasks from 38.3 to about 54, a relative increase of more than 42%. The author believes that this is a unique extension direction of WAM: it can continue to improve the task understanding of the world model with a large number of human videos without action labels.

Q5: Is few-shot new embodiment adaptation feasible?

The author uses DreamZero-AgiBot checkpoint ~30 minutes, 55 tracks, 11 tasks The post-training of YAM play data shows that language following and certain zero-sample capabilities are still retained on the new dual-arm robot. The results here are mainly qualitative display, and there is no numerical table as in the previous sections, but the argument of this part is: if action learning mainly involves inversely understanding actions from future videos, then when migrating to a new embodiment, the model only needs a small amount of data to calibrate the mapping of "visual future to action".

Q6: Can Flash maintain performance when denoising in fewer steps?

The key information here is: if you directly push DreamZero from 4 steps to 1 step, the performance will drop a lot; while Flash training can bring back most of the 1-step performance, from 52% to 74%. So the value of Flash is not just to be faster, but to make "very few step reasoning" a trainable work point.

6.3 Ablation experiment

This set of ablations very directly supports the three methodological claims of the paper: The data must be diverse, the model must be large, and autoregression is more suitable for closed-loop WAM..

6.4 Supplementary experiments and real-time execution details

Real-time execution speed source

The overall latency narrative given by the author is: naive single GPU DreamZero requires approximately 5.7 seconds/action chunk, and after superimposing all optimizations on GB200, it can be reduced to about 150ms / chunk, thus meeting the 7Hz closed-loop control under asynchronous execution.

6. Analysis and Discussion

7.1 Explanations already given in the original text of the paper

• The authors attribute DreamZero's advantage to "the strong prior provided by the video predictions" rather than the larger size of the simple model. Especially for heterogeneous data, WAM does not need to learn action rules from noisy state-action pairs like VLA.
• The author repeatedly emphasizes that DreamZero's action and video predictions are highly aligned, so failure is often not "the action decoder is wrong" but "the video plan is wrong first". This is why they believe that improving the video backbone can directly improve the quality of the strategy.
• The benefits of cross-embodiment transfer are explained by the author as "the world model learns task dynamics first, and action mapping only needs a small amount of new embodiment data to realign".
• AR and bidirectional are close in task progress, but AR is smoother and faster. The authors therefore argue that the core benefit of WAM is not only the final success rate, but also the engineering usability of video-action-speech alignment.

7.2 Limitations of the author's statement

7.3 Failure cases and applicable boundaries

• DreamZero's strongest boundaries right now are: Open environment, unseen actions, cross-embodiment. This is the main battlefield that the paper really wants to prove.
• It is not an explicit planner and does not do test-time MPC/MPPI; it relies on the implicit visual planning brought by the joint video-action generation itself.
• The results of the few-shot adaptation embodiment are still mainly qualitative, indicating that this direction has potential, but it is not yet a fully mature quantitative conclusion.
• The author specifically pointed out that if larger-scale egocentric human video can be accessed in the future, the upper limit of WAM may be much higher than shown in the current experiment, because the scale of video data is naturally much larger than the action annotation robot data.