AnyManip | Ziyin Xiong

BIGAI · Manipulation · Imitation Learning · Optical Flow

AnyManip treats motion as the bridge between visual understanding and robot control. A diffusion model first predicts task-relevant optical flow; a robot policy then combines that prediction with RGB observations and proprioception to produce actions.

Stage 01 Predict Motion Image + Language → Optical Flow

Stage 02 Execute The Skill RGB + Flow + State → Robot Actions

01 · Optical Flow Prediction

Task-Conditioned Motion

The flow model focuses on motion that matters for the instruction: where the gripper should move, which object should be manipulated, and how the interaction should unfold over time.

Training targets come from optical flow estimated in robot and human manipulation videos, including BridgeData V2, DROID, and RH20T. From an initial scene and a language instruction, the model predicts future dense motion for both the end effector and the object.

Predicted optical flow for stacking a green block on a yellow block — Stack the green block on the yellow block.

Predicted optical flow for placing a carrot on a plate — Put the carrot on the plate.

02 · Robot Policy

From Pixels To Actions

The action model combines the current RGB observation, predicted motion, and robot state. It predicts a sequence of Cartesian position, orientation, and gripper commands; the examples below show the scene and motion cues for two simulated tasks.

Carrot placement scene alongside object and gripper motion representations — **Carrot Placement Setup** RGB scene, interaction regions, and predicted motion for placing a carrot on a plate.

Block stacking scene alongside object and gripper motion representations — **Block Stacking Setup** The same scene and motion representation applied to a compositional task.

03 · Offline Action Evaluation

Predicted Actions Across Tasks

Later experiments compare policy predictions with demonstrated actions across seven channels: x, y, z, yaw, pitch, roll, and grasp. Each result pairs sampled episode frames with the predicted and reference action traces.

Yellow cloth episode frames above predicted and demonstrated Cartesian, orientation, and grasp action traces — **Yellow Cloth Into A Basket** Offline action comparison for placing a cloth into a basket. Blue: prediction; orange: demonstration.

Mushroom placement episode frames above predicted and demonstrated Cartesian, orientation, and grasp action traces — **Mushroom Into A Pot** A second instruction and object configuration, evaluated with the same seven action channels.

04 · Perspective

Why Flow?

Optical flow provides a robot-agnostic description of how a task should evolve. By separating motion prediction from action generation, AnyManip can learn from heterogeneous visual data while leaving embodiment-specific control to the policy.

This project was conducted at the Beijing Institute for General Artificial Intelligence with Peiyuan Zhi and Yang Yang, advised by Dr. Tengyu Liu and Dr. Siyuan Huang.