Dreamland: Controllable World Creation with Simulator and Generative Models

Sicheng Mo* , Ziyang Leng* , Leon Liu , Weizhen Wang , Honglin He , Bolei Zhou

University of California, Los Angeles

* Equal Contribution
Code | Paper

TL; DR

:national_park: Dreamland is a hybird generation pipeline that connects simulators and generative models to achieve controllable and configurable world creation.

:robot: Dreamland demonstrates great quality and controllability in scene generation, and improves the adaptation of embodied agents to the real world.

Simulator-Conditioned Generation
● ●
Diverse Scene Generation
● ● ●
Safety-Critical Scene Generation
● ●

Dreamland Architecture

Image

Dreamland pipeline consists of three key stages: (1) Stage-1 Simulation : scene construction with physics-based simulator, (2) Stage-2 LWA-Sim2Real : transferring the Sim-LWA from simulation to Real-LWA with an instructional editing model and user instructions, (3) Stage-3 Mixed-Condition Generation : rendering an aesthetic and realistic scene with a large-scale pretrained image or video generation model

Experiments

Scene Image
● ● ● ●
Image

Dreamland demonstrates great quality and controllability, with scalability that benefits from stronger pre-trained model deployed for Stage-3.

Scene Editing

Image

Dreamland supports editing a generated scene by adjusting the corresponding source scene.

@article{mo2025dreamland,
  title={Dreamland: Controllable World Creation with Simulator and Generative Models},
  author={Sicheng Mo and Ziyang Leng and Leon Liu and Weizhen Wang and Honglin He and Bolei Zhou},
  journal={Preprint},
  year={2025}
}

Acknowledgement

Cosmos-Predict1 and Cosmos-Transfer1 were used as the base generation models for video results.