Dreamer 4 pushes the frontier of world model accuracy, speed, and learning complex tasks from offline datasets.
co-led with ![David Watson 🥑](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="96" height="96"><rect fill-opacity="0"/></svg>)
Enabled by imagination training, Dreamer 4 is the first agent to mine diamonds in Minecraft entirely from offline data!
This setting is crucial for fields like robotics, where online interaction is not practical. The task requires 20k+ mouse/keyboard actions from raw pixels
Dreamer 4 learns a scalable world model from offline data and trains a multi-task agent inside it, without ever having to touch the environment. During evaluation, it can be guided through a sequence of tasks.
These are visualizations of the imagined training sequences
We first pretrain the WM, finetune agent tokens into the same transformer to predict policy & reward, and then improve the policy by imagination training
Shortcut forcing builds on diffusion forcing and shortcut models, training a sequence model with both the noise level and requested step size as inputs
This enables much faster frame-by-frame generations than diffusion forcing, without needing a distillation phase 
On the offline diamond challenge, Dreamer 4 outperforms OpenAI's VPT offline agent despite using 100x less data
It also outperforms modern behavioral cloning recipes, even when they are based on powerful pretrained models such as Gemma 3
We find that imagination training not only makes policies more robust but also more efficient, so they achieve milestones towards the diamond faster
Moreover, using the WM representations for behavioral cloning outperforms using the general representations of Gemma 3
I honestly think Minecraft will be a great testbed for the next few years of agent and robotics research! There is a lot more to do
Any chance you would be open to speak about it?
