Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving

Yu Yang^1,*, Jianbiao Mei^1,*, Yukai Ma¹, Siliang Du^2,†, Wenqing Chen², Yijie Qian¹, Yuxiang Feng¹, Yong Liu^1,†
1 Zhejiang University ² Huawei Technologies
^* Equal Contribution ^† Corresponding Authors

📢 News

• [2024-12-10] Our Drive-OccWorld is accepted by AAAI 2025 (Oral)!

• [2024-10-14] We released our project website here.

• [2024-08-26] The paper can be accessed at arxiv.

---

🎯 Abstract

4D Occupancy Forecasting and Planning via World Model. Drive-OccWorld takes observations and trajectories as input, incorporating flexible action conditions for action-controllable generation. By leveraging world knowledge and the generative capacity of the world model, we further integrate it with a planner for continuous forecasting and planning.

📝 Getting Started

• Installation

• Prepare Dataset

• Train and Evaluation

🎥 Demo of 4D Occupancy and Flow Forecasting

Drive-OccWorld understands how the world evolves by accurately modeling the dynamics of movable objects and the future states of the static environment.

Scene 1 (Lane Change)

Scene 2 (Pedestrian Crossing)

Scene 3 (Vehicle Following)

🚗 Demo of Continuous Forecasting and Planning (E2E Planning)

Drive-OccWorld plans trajectories through forecasting future occupancy state and selecting optimal trajectory based on a comprehensive occupancy-based cost function.

Scene 1 (Turn Left to Avoid Stopped Vehicle)

Scene 2 (Slowing Down to Wait for Crossing Pedestrians)

Scene 3 (Turn Right to Avoid Stopped Vehicle)

Acknowledgments

We utilized the following repos during development:

• ViDAR
• Cam4DOcc
• ST-P3
• UniAD

Thanks for their Awesome open-sourced work!

🔖 Citation

If you find our project useful, please kindly cite us via:

@article{yang2024driving,
  title={Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving},
  author={Yang, Yu and Mei, Jianbiao and Ma, Yukai and Du, Siliang and Chen, Wenqing and Qian, Yijie and Feng, Yuxiang and Liu, Yong},
  journal={arXiv preprint arXiv:2408.14197},
  year={2024}
}