Dream What I Can: Sparse Visual Planning for
Long-Horizon Robot Manipulation

Chongkai Gao^1,2 Wenlong Huang¹ Haonan Chen^1,3 Lin Shao² Yilun Du³ Jiajun Wu¹ Ruohan Zhang⁴ Li Fei-Fei¹

¹ ² ³ ⁴

The world is not made of words

We introduce DreamCan, a sparse visual planner that can generate a sequence of sparse keyframes for robot manipulation task planning. Rather than generating dense frames, DreamCan generates visual subgoals that balance reachability—what the robot can achieve—and task utility—what moves the task forward.

Video

Key Results

Sparse visual planning enables long-horizon planning. We achieve substantially higher planning success rates than the prior methods on Language-Table long-horizon tasks.
Sparse visual planning enables learning from suboptimal data. We demonstrate that DreamCan benefits from scaling suboptimal action-free video data on RoboMimic-MH Square, without additional policy training.
Sparse visual planning enables test-time search and language steering. We show that DreamCan benefits from increased test-time compute and can be steered with language on Language-Table tasks.
Sparse visual planning can effectively guide a goal-conditioned low-level policy. We show that our planner can guide a small goal-conditioned flow-matching policy and achieve comparable results with large VLA/WAMs on LIBERO-Long, with no gradient or shared features between the planner and policy.
Sparse visual planning is effective on real-world tasks. We show that our method can deal with different real-world tasks including cardboard box folding (contact-rich task), box packing (compositional generalization task), and make-a-line (long-horizon symbolic planning task).

Method

DreamCan first employs an unsupervised approach to discover sparse visual subgoal sequences from robot video datasets that balance subgoal reachability and utility, then fine-tunes a pretrained video generation model (WAN 2.2 TI2V-5B) to generate the full sequence of subgoals.

Select a stage to view its details.

Input: Robot video datasets.
Output: A reachability value network that estimates the reachability $R \in [0, 1]$ between two states.

DreamCan first learns a Reachability Value Function from action-free videos to measure transition difficulty between two frames.

What does reachability measure?

An RL value. In an MDP, $V_{R}^{π}$ is the expected discounted return for reaching frame $s_{j}$ from frame $s_{i}$ : −1 per step until arrival, then 0.
A score between 0 and 1. Normalizing this value gives reachability. Earlier arrival means a higher score; $τ_{ij}$ is the first hitting time.

R^{π} (s_{i}, s_{j}) = E_{π} [γ^{τ_{ij}}]

= 1 + (1 - γ) V_{R}^{π} (s_{i}, s_{j}) \in [0, 1]

How is it learned?

Learn from video transitions. Fit $V_{R}^{θ}$ using an action-free, goal-conditioned variant of Implicit Q-Learning (IQL), an offline RL method.
Favor higher returns. An upper-expectile loss emphasizes transitions that reach the conditioning frame more efficiently.

Training objective & sampling

For video transitions $D$ , termination flag $d_{i}$ and target head $\bar{θ}$ , the TD target and expectile loss are:

y_{i} = r_{i} + γ (1 - d_{i}) V_{R}^{\bar{θ}} (s_{i + 1}, s_{j})

δ_{i} = y_{i} - V_{R}^{θ} (s_{i}, s_{j})

L_{α} = E_{D} [| α - 1_{δ_{i} < 0} | δ_{i}^{2}]

The upper expectile favors higher-return transitions. We use $α = 0.75$ , $γ = 0.95$ and Polyak rate $ρ = 0.005$ , with $V_{R} (s_{j}, s_{j}) = 0$ . Second frames are sampled equally from future frames with geometric offsets and other trajectories.

Frame i observation history and frame j pass through a shared frozen RADIO encoder. A temporal Transformer encodes the history, an MLP encodes frame j, and a language encoder provides instruction context. Their features are fused by a value MLP to predict V R of state i and state j, then normalized to reachability R in zero to one.

Explore the learned value

Language Instruction:

Loading learned reachability data…

Input: Robot videos and language instructions.
Output: A task-conditioned utility $U \in [0, 1]$ that measures progress toward task completion.

DreamCan uses a Task Utility Value Function to measure progress toward the instruction $ℓ$ .

RoboMeter · clear, simple tasks

A video prefix and instruction are enough for RoboMeter to estimate task progress.

Build supervision. Pair demonstration clips with instructions; derive progress labels from relative frame positions. Add reversed, rewound and mismatched-instruction clips.
Fine-tune where needed. Freeze the backbone and visual encoder; train LoRA adapters and progress/success heads.

ℒ_{FT} = ℒ_{progress}^{CE} + ℒ_{success}^{BCE}

Prediction & loss details

u_{t} = f_{ϕ} (o_{0 : t}, ℓ)

Progress uses ten-bin soft-target cross-entropy (CE); the success head uses binary cross-entropy (BCE). In the LoRA update, $W_{0}$ stays frozen while $A$ and $B$ are trained.

Video-prefix frames pass through a visual encoder; the instruction passes through a text tokenizer. A causal RoboMeter vision-language backbone, optionally adapted with LoRA, produces progress-token features for a progress MLP to predict raw task progress.

VLM + SAM · complex, long-horizon tasks

For tasks needing explicit spatial reasoning, score the arrangement of detected objects.

Define and detect. A VLM defines task criteria; SAM extracts object masks from each frame.
Measure progress. A task-specific scorer evaluates those masks. For make-a-line, it rewards alignment and even spacing.

u_{t} = f_{ℓ} (M_{t})

Make-a-line scoring example

M_{t} = SAM (o_{t})

Alignment depends on distance $d_{k}$ to the line, tolerance $τ$ and softness $β$ . $S_{t}$ scores even spacing among $N = 8$ blocks; $M_{t}$ denotes their masks.

a_{k} = σ (\frac{τ - d_{k}}{β})

u_{t} = \sqrt{\frac{1}{N} \sum_{k = 1}^{N} a_{k}} (0.8 + 0.2 S_{t})

VLM + SAM · Make a line

Explore the task utility

Language Instruction:

Loading task utility data…

Experimental Results

Select an option to explore the experiments.

Dataset: Three long-horizon tasks from the Language-Table dataset: Make a line, All to region, and Canonical shape.
Train & Eval: We train on 1,424, 1,303, and 1,378 videos for Make a line, All to region, and Canonical shape, respectively, and evaluate on 100 videos per task. Planning success requires reaching the instructed goal, preserving all blocks, and maintaining valid transitions.

Language Instruction: Loading planning examples…

Planning success rate ↑
Method	Success rate
Dense planning	71%
Uniform planning	69%
VLP	56%
Ours	86%

DreamCan (Ours)

Baselines

Dense planning

Uniform planning

VLP

Video to be added

Dataset: Language-Table Make a line and the real-world Box Packing task.
Train & Eval: We use 1,424 training videos for Language-Table make a line task, and 100 videos for evaluation; we use 550 demonstrations for the real-world box packing task and 95 cases for testing.

Figure 7 reproduced: final task progress versus generator calls per episode for subgoal best-of-N, anchored search, Dense video and Uniform subgoal, with mean plus or minus SEM.

What does the y-axis measure?

Final task progress ∈ [0, 1]: a continuous score for how well eight detected blocks align with the instructed line and remain separated. Higher is better.
Mean ± SEM: average the last two evaluated frames of each plan, then average across cases. Error bars show the standard error of this mean.

Two ways of test-time search

Whole-sequence best-of-N: generate N complete plans and keep the one with the highest task-progress score.
Anchored search: keep the best subgoal prefix as clean latents and regenerate the remaining sequence.

Findings

DreamCan can benefit from test-time search compute, especially from anchored search.
Dense planning can also improve with extra compute, but its final task progress remains below DreamCan's initial result.
Uniform subgoal planning cannot benefit from test-time compute.

Generated plans at different budgets

Trajectory 0

Language Instruction: put the blocks in a diagonal line from the top right to bottom left

1 generator call · no search

4 generator calls

8 generator calls

Trajectory 1

Language Instruction: put the blocks in a diagonal line from the top right to bottom left

1 generator call · no search

4 generator calls

8 generator calls

Trajectory 2

Language Instruction: put all the blocks in a vertical line on the center of the board

1 generator call · no search

4 generator calls

8 generator calls

Dataset: PushT and LIBERO-Long.
Train & Eval: We use 206 demonstrations for PushT, and 379 demonstrations for 10 tasks in LIBERO-Long. We use the same low-level policy architecture across different baselines. We use 50 episodes for evaluation per task.

A goal-conditioned flow-matching policy converts visual subgoals into robot actions. The planner and controller have no shared features or gradients.

Observation history, generated visual subgoals, language and robot state form a condition prefix for a flow-matching Transformer; iterative velocity integration generates a 16-step action chunk.

PushT

PushT planning method comparison
Method	Success rate
E2E	58.0%
Dense	70.0%
Uniform	72.0%
DreamCan (Ours)	86.0%

LIBERO-Long

LIBERO-Long planning method comparison
Method	Success rate
E2E	79.2%
Dense	84.0%
Uniform	82.4%
DreamCan (Ours)	91.2%

Planning and execution in simulation

PushT

Push the T-shaped block into the target region.

Completed

Execution

Planner output

LIBERO-Long

Put both the alphabet soup and the tomato sauce in the basket.

Completed

Execution

Planner output

Dataset: Three dual-arm YAM robot tasks: Cardboard Box Folding, Box Packing, and Make a Line.
Train & Eval: We collect 100 demonstrations for the cardboard box folding task, 550 demonstrations (110 combinations × 5) for the box packing task, and 90 (3 variations × 30) demonstrations for the real-world make a line task. Each task is evaluated over 20 episodes; packing holds out 95 unseen object combinations.

Cardboard Box Folding (Contact rich task)

Task success rate
Method	Success
End-to-end policy	5%
Dense video planning	0%
Uniform subgoal planning	10%
DreamCan	40%

Box Packing (Combinatorial generalization task)

Task success rate
Method	Success
End-to-end policy	0%
Dense video planning	5%
Uniform subgoal planning	20%
DreamCan	65%

Make a Line (Long-horizon symbolic planning task)

Task success rate
Method	Success
End-to-end policy	10%
Dense video planning	0%
Uniform subgoal planning	25%
DreamCan	55%

Targets: We evaluate learned subgoal reachability and the effectiveness of the different dynamic-programming terms.
Train & Eval: For the reachability study, we use LIBERO-Long and RoboMimic Square; for the DP-term ablation, we use LIBERO-Long tasks.

Figure 8: two stacked violin plots show the distribution of segment success rates across subgoal stages for ten LIBERO-Long tasks. DreamCan has mean 0.98; Uniform subgoal planning has mean 0.88.

Experimental setting

Ten LIBERO-Long tasks. Evaluate consecutive subgoal pairs with the trained goal-conditioned policy.
Reset, replay, then execute. Replay to the first subgoal, then test whether the policy reaches the next state within 5 cm. Run 50 trials per segment.
Read the distributions. Dots show stage success rates; diamonds show task means. Violins show within-task variation, and dashed lines show the mean across ten tasks.

Figure 8 uses object-only success for DreamCan and object-plus-end-effector success for Uniform subgoal planning.

Findings

DreamCan’s subgoals are highly reachable. Its mean segment success rate is 0.98, with task means between 0.91 and 1.00.
Uniform subgoals show larger variation. Their reported mean is 0.88, and several tasks contain stages with substantially lower success.

Which terms guide subgoal discovery?

R rewards reachable transitions, U rewards task progress, and S controls temporal spacing. The manuscript compares five discovery objectives with the planner and policy trained under the same stated protocol.

LIBERO-Long task success · manuscript Table IV
DP terms	R	U	S	Task success
Spacing only (S)	—	—	✓	81.8%
Reachability + spacing (RS)	✓	—	✓	74.2%
Utility + spacing (US)	—	✓	✓	78.0%
Reachability + utility (RU)	✓	✓	—	83.8%
Ours (RUS)	✓	✓	✓	91.2%

Interactive Demo

Drag the eight Language-Table blocks into any layout, pick a make-a-line instruction, and the DreamCan planner (Wan 2.2 TI2V-5B at 640×384 with our subgoal LoRA) generates the 16-step visual plan from that start frame. Every plan runs anchored test-time search with a detection-based block guard: each accepted subgoal is checked against the previous one and hallucinated blocks are removed before the next step.

Cold start. The GPU container starts on demand; the first visit after an idle period takes about two minutes to load the model.
Planning time. One plan is 30-60 generator calls, roughly 8-14 minutes on an A100; the page shows the queue position and progress. Set candidates to 1 for a faster, less careful plan.
Shuffle draws a layout from real episode start frames (the training distribution); the demo works best when blocks are clustered like that.

Open the demo in a new tab

FAQ

What is DreamCan?

DreamCan is a hierarchical robot manipulation system with a high-level sparse visual planner and an independently trained, goal-conditioned low-level policy. Given the current image observation and a language instruction, the planner predicts a short sequence of visual subgoals. The policy turns these subgoals into robot actions, and the planner is periodically called again using the latest observation for closed-loop replanning.

Why plan with sparse visual subgoals instead of language or dense videos?

Language plans can leave contact locations, geometric relationships, and intermediate object configurations ambiguous. Dense video plans are expensive to generate and can accumulate prediction errors or overconstrain the controller. Uniformly spaced keyframes can include redundant motion, regressions, or transitions that are difficult to execute. DreamCan selects sparse subgoals that balance reachability with useful task progress.

What does the reachability value function measure?

The normalized reachability score estimates how quickly and reliably one visual state can be reached from another. It is a discounted first-hitting-time value, rather than a success probability within a fixed horizon. The model uses action-free, goal-conditioned expectile learning with frozen RADIO visual features.

The learned score is a proxy for dataset-supported reachability, assuming a downstream controller whose capabilities are broadly compatible with the recorded behavior. It is not simply a penalty for elapsed video time: the paper evaluates demonstrations with detours and repeated states to distinguish the two.

How do the planner and low-level policy communicate?

Their interface is the generated subgoal images; the planner and policy do not share gradients or hidden representations. The low-level controller is a 0.1B-parameter goal-conditioned flow-matching Transformer. It uses current visual observations, proprioceptive state, the language instruction, and several future subgoals to produce an action chunk.

The 0.1B parameter count refers only to the low-level policy. The high-level planner is adapted from a 5B-parameter pretrained video model.

How do test-time search and language steering work?

The paper evaluates whole-sequence best-of-N search and anchored search. Anchored search samples candidate continuations, commits a selected prefix as clean latents, and regenerates the remaining subgoals. Candidate plans are evaluated with task-progress scores, and additional generator calls improve planning progress in the evaluated settings.

The planner is conditioned on language instructions. For test-time steering, the paper also uses a vision-language model to write instruction-specific task-progress functions that select plans matching the requested outcome. Examples include different line arrangements in Language-Table and preferred object placements during box packing.

Citation

bibtex

@article{gao2026dreamcan,
  title={Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation},
  author={Gao, Chongkai and Huang, Wenlong and Chen, Haonan and Shao, Lin and Du, Yilun and Wu, Jiajun and Zhang, Ruohan and Li, Fei-Fei},
  journal={},
  year={2026}
}

Video

Key Results

Method

What does reachability measure?

How is it learned?

Reachability value network

Trajectory frames

Reachability curve

RoboMeter · clear, simple tasks

RoboMeter · progress prediction

VLM + SAM · complex, long-horizon tasks

VLM + SAM · Make a line

Trajectory frames

Task utility curve

Balance three needs

Optimize the whole sequence

Full demonstration

Selected keyframe sequence

Align sparse targets

Keep the prefix clean

Train only LoRA

Experimental Results

Baselines

Better demo

Worse demo

Findings

What does the y-axis measure?

Two ways of test-time search

Findings

Generated plans at different budgets

Trajectory 0

Trajectory 1

Trajectory 2

Language Table: change the target arrangement

Box Packing: steer spatial preferences

Goal-conditioned low-level policy architecture

PushT

LIBERO-Long

Planning and execution in simulation

PushT

Execution

Planner output

LIBERO-Long

Execution

Planner output

Cardboard Box Folding (Contact rich task)

Box Packing (Combinatorial generalization task)

Make a Line (Long-horizon symbolic planning task)

Experimental setting

Findings

Which terms guide subgoal discovery?

Interactive Demo

FAQ

Citation