Act2Goal: From World Model To General Goal-conditioned Policy

1. Quick overview of the paper

Difficulty rating: ★★★★☆. Requires understanding of visual goal-conditioned policy, flow matching, Video DiT/3D VAE world model, action expert, hintsight goal relabeling, LoRA finetuning, and real robot evaluation.

Keywords: Goal-conditioned policy, visual world model, Multi-Scale Temporal Hashing, flow matching, action expert, HER, LoRA, online autonomous improvement.

Figure 1. Method overview: Input a visual goal, the goal-conditioned world model first generates a visually imagined trajectory leading to the goal, and then the strategy executes it in the real world.

2. Motivation

2.1 Why use visual targets instead of language targets?

Robotic operation requires task descriptions that are both flexible and precise. Language can express a variety of tasks, but it is often not specific enough in fine-grained operations: such as "arrange the dessert like this", "write this word", "put the insert in the hole", the spatial relationship, shape, orientation and final state details of the target are difficult to completely describe in natural language without ambiguity. Visual goal images directly encode object configurations, spatial relationships, and final state constraints and are therefore more suitable for goal-conditioned manipulation.

2.2 Core shortcomings of existing GCP

Many visual goal-conditioned policies map the current observation and goal image directly to actions. This works for short-term tasks, but long-term tasks require the strategy to continuously know "where am I now from my goal?" If the model has no intermediate representation of progress, it is easy to match only local state-action patterns in the training data, accumulating bias in OOD objects, layouts, or long control chains.

The diagnosis of the paper is that GCP needs an explicit visual dynamics model to predict plausible intermediate states from the current state to the target state. In this way, action expert not only looks at the end point, but also gets a visual route from thick to thin.

2.3 Why is MSTH still needed?

Generating an entire future video is still not enough. If long trajectories are all intensively predicted, the calculation cost is high and the remote details may not be reliable; if you only look at the short term, the strategy will be responsive but it is easy to lose sight of the long-term goal. The role of MSTH is to split the time scale: the near end uses dense frames to provide local closed-loop control, and the far end uses logarithmic sparse frames to maintain global direction.

Figure 2. System overview: After offline imitation learning, the model can be generalized in ID/OOD scenarios with zero samples; in the online stage, it continues to improve through the rollout-goal relabel-optimize loop.

3. Related work context

4. Detailed explanation of method

4.1 Overall process: three-stage learning

Act2Goal's learning process is divided into three stages:

1. Stage 1 joint training: Fine-tune the pre-trained world model to generate MSTH-distributed multi-view transition videos under the current observation and target image conditions, while jointly training the action generation task.
2. Stage 2 action adaptation: Use behavior cloning to fine-tune the complete model end-to-end, using only action flow matching loss to make the visual representation more suitable for action planning.
3. Stage 3 online autonomous improvement: After deployment, collect its own rollout, relabel the next observation that actually arrives as goal, and only update the LoRA layer.

4.2 Model architecture: GCWM + Action DiT

Figure 3. Architecture diagram: The GCWM on the left uses the VAE latents of the current multi-view image and the target image as conditions, and generates MSTH latent frames through Video DiT; the Action DiT on the right receives the robot state and world model multi-scale features, and generates MSTH-structured actions through cross-attention.

The Goal-Conditioned World Model is modified based on Genie Envisioner: it retains the video generation capability, removes language conditions, and adds visual goal conditions. The current observation latent is recorded as $z_t$, the target image latent is recorded as $z_g$, the random noise is $\epsilon$, and the world model generates intermediate latent frames:

Action expert is architecturally isomorphic to world model, but smaller in width. It receives the proprioceptive state $c_p$ and the hierarchical transition features $c_w=\{h_{\mathrm{world}}^1, \ldots, h_{\mathrm{world}}^L\}$ of the world model, and generates actions:

4.3 Multi-Scale Temporal Hashing (MSTH)

MSTH is the core mechanism of this paper. Given the total imagined trajectory length $K$, the near-end horizon $P$, and the visual sampling stride $r$, MSTH splits the future visual state into two segments.

The actions also adopt the same multi-scale structure, but there is a key difference: the near-end action is predicted at every time step, $\{a_{t+1}, \ldots, a_{t+P}\}$, for actual execution; the remote action $\{a_{t+d_m}\}_{m=1}^M$ is only used for long-range guidance and is not executed at deployment time. The appendix gives the specific implementation: the world model predicts 4 latent frames, including 2 proximal and 2 distal, which are decoded by 3D VAE into 9 proximal and 9 distal visual frames; the action expert outputs 54 proximal actions, of which 50 are executed, and the other 9 distal actions are output only for guidance Appendix A.1.

Figure 8. Examples of videos generated by GCWM: proximal frames are closer to local action details, while distal frames correspond to longer time spans and bear target consistency constraints.

4.4 Two-stage offline training objectives

Stage 1 jointly optimizes visual generation and action generation. The visual part uses flow matching loss:

The meaning of this design is: the first stage makes the generated visual trajectory "look reasonable and useful for the action"; the second stage aligns the entire pipeline to expert actions.

4.5 Online self-improvement: HER + LoRA

The online stage does not require external rewards, nor does it require manual marking of success/failure. The robot executes the strategy and stores each transition in the replay buffer: $(o, c_p, a, o')$. Then the actually reached $o'$ is used as the new goal $g'$, and a supervision sample is constructed as "$g'$ can be reached by executing $a$ under $o$". After the buffer reaches the threshold, only the LoRA layer is fine-tuned using these relabeled transitions, and the base model is frozen.

Act2Goal online autonomous improvement:
1. initialize replay buffer B and LoRA parameters
2. execute policy for one episode and store transitions (o, c_p, a, o')
3. for each transition, relabel achieved observation o' as goal g'
4. when B reaches threshold:
   sample batches from B
   optimize action prediction toward stored action a under relabeled goal g'
   update only LoRA parameters
   clear B
5. repeat rollout -> relabel -> optimize until performance converges

Appendix implementation details: Deployed on AgiBot Genie-01 + NVIDIA RTX 4090; LoRA rank 64; replay buffer size 20; 10 epochs per round, including rollout, backpropagation and environment reset of about 5 minutes; inference latency of 50 executable actions is about 200 ms Appendix A.1.

5. Experiment

5.1 Experimental questions

The experiment revolves around three questions: whether offline imitation training can be generalized to ID/OOD scenarios; whether online autonomous improvement is effective; and whether MSTH is a key component of long-term tasks.

5.2 Robotwin 2.0 simulation generalization

The author selects four tasks from Robotwin 2.0: Move Can Pot, Pick Dual Bottles, Place Empty Cup, Place Shoe. Easy mode is the seen/no-noise setting and Hard mode is the unseen/noisy setting. The target image is passed through a fixed environment seed, and the final state is extracted from the successful trajectory as the goal condition.

Act2Goal is best in all Easy tasks; best in Hard mode except Move Can. This result supports the paper's claim that world model + MSTH is helpful for OOD long-term scenarios, but it also shows that not all hard scenarios beat $\pi_{0.5}$-GC.

5.3 Generalization to real robots

Figure 4. Authentic assessment tasks: Whiteboard Word Writing, Dessert Plating, Plug-In Operation, each task has ID and OOD configuration.

Three types of tasks respectively test different abilities: whiteboard writing requires long-range stroke combinations, and OOD is training unseen words; dessert plate placement requires fine-grained spatial arrangement according to the target image, OOD introduces dessert types, plate styles, and background changes; Plug-In ID requires inserting metal artifacts seen in training, and OOD involves inserting cylindrical beverage bottles into cup holders.

The appendix adds details of real tasks: when writing/painting, the marker is first manually placed in the clamping claw. During long writing trials, the marker will slide, so it is fixed with tape; in bearing insertion, the bearing exceeds 2 kg, the diameter of the bottom is about 1 cm, and the diameter of the hole is about 1.5 cm; the dessert plate uses silicone toy desserts to ensure repeatability Appendix A.2.

5.4 Online self-improvement

Figure 5. Four OOD hard scenarios for online self-improvement in Robotwin 2.0.

Figure 6. Simulation online training: The success rate continues to increase after multiple rounds of training, and converges after about 3 rounds; a comparison of data selection strategies for replay buffers shows that using all rollouts is the best, and failed-only rollouts can also bring improvement.

In the simulation, the author selected one scenario from each of the four hard-mode tasks to perform multiple rounds of online training. The text of the paper gives a qualitative conclusion: it converges after about 3 rounds, and the maximum success rate can be increased up to 8 times that of the pre-training baseline; using all rollouts is better than using only successful or failed rollouts. This result is consistent with HER logic: failure trajectories also contain learnable achieved-goal transitions.

Figure 7. Real whiteboard unseen pattern online training: progressive improvement from poor initial drawings. The main text also reports that the OOD Plug-In success rate improves from 0.30 to 0.90 with online training.

5.5 MSTH ablation

This table is the strongest evidence supporting MSTH: fixed horizon chunking is okay for short words, but it collapses quickly when the words become longer, especially OOD long words, which is 0.00; MSTH still maintains 0.90/0.88 for long words in ID/OOD. The explanation is that the fixed horizon is more prone to goal confusion as the sequence becomes longer, while the distal sparse anchors of MSTH maintain the goal direction.

6. Reproducible auditing

6.1 Data, training resources and model scale

6.2 Evaluation Caliber

• Real-world success rates are manually annotated with 40 model rollouts per experiment.
• The simulation success rate is automatically calculated with 90 rollouts per experiment.
• In the online self-improvement experiment, the author saves the model weights after each round of training, and evaluates these checkpoints separately after the online process is completed.

6.3 Minimum recurrence route

1. First implement the visual goal-conditioned policy baseline in simulation: current image + goal image + proprio state → action chunk.
2. Connect to a pre-trained video/world model, change it to current observation + goal image condition, and output transition latent frames.
3. Implement MSTH: near-end dense frames, far-end log-spaced distal frames; action-end prediction of dense proximal actions + sparse distal guidance actions.
4. Use Stage 1 $\mathcal{L}_v + 0.1\mathcal{L}_a$ for joint training, and then use Stage 2 $\mathcal{L}_a$ for action adaptation.
5. First, reproduce the four tasks of Robotwin 2.0; reproducibility on a real machine requires a robot platform, target image acquisition and a safe reset pipeline.
6. Finally, online HER + LoRA is implemented: train for 10 epochs after buffering to 20, only update LoRA, and compare all/success/failure rollout strategies.

6.4 Main recurring risks

7. Analysis, Limitations and Boundaries

7.1 The most valuable part of this paper

The most valuable part of this article is that it concretizes the long-term problem of visual goal strategy: instead of just letting the policy see the goal image, it first constructs a visual intermediate route that "evolves toward the goal", and then compresses this route into a control-friendly multi-scale representation. The idea of ​​MSTH is simple but effective: it should be dense near and sparse far away; proximal actions are executed, and far-end actions are only constrained.

The second value point is that the online self-improvement process is sufficiently engineered. HER-style relabeling + LoRA is very practical on real robots: no reward function is required, and no manual labeling is required at each step. Failure trajectories can also be converted into training signals through achieved-goal relabeling.

7.2 Why the results hold up

• There are levels of task coverage: Simulation Robotwin 2.0, real writing, dessert plating, and insertion operations cover different difficulties.
• OOD settings are explicit: Unseen words, visual changes to desserts/plates/backgrounds, migration from bearing insert to cup holder insert.
• MSTH ablation is strong: Writing long words on the whiteboard w/o MSTH almost fails, while w/ MSTH maintains a high success rate, directly supporting the core of the method.
• The evidence for online training is consistent: Simulation multi-round improvement, real unseen drawing improvement, Plug-In OOD from 0.30 to 0.90, in the same direction.

7.3 Limitations and points that need to be questioned

7.4 Questions that can be asked in group meetings

1. How to choose MSTH's $K, P, r, M$? Do different tasks require adaptive time scales instead of fixed log spacing?
2. Remote actions are not executed, but only serve as guidance. Where does their supervision come from, and is it possible that it may conflict with the near-end execution of actions?
3. Stage 2 only optimizes action loss, but the gradient is also passed back to the world model. Will this reduce the realism of the video generated by the world model and only retain action-useful features? Is this an advantage or a risk?
4. HER relabeling treats $o'$ as goal, but if the action results in a collision or meaningless state, will this relabeling reinforce bad behavior? all rollouts best illustrates that the model may be able to absorb noise, but where are the boundaries?
5. Compared to directly using goal image + transformer memory for long context control, is the additional generation cost of GCWM necessary for all tasks?

Attachment: This report covers inspections