TIER IV’s research proposal selected by JAXA Space Exploration Innovation Hub: Accelerating practical application of world models for autonomous driving through neural simulator development

 

This research aims to create high-quality and large-scale digital twins to simulate AD environments, capitalizing on the groundbreaking NeRF world model. The project is set to　develop a neural simulator capable of pre-constructing NeRF models from camera and LiDAR sensor data from autonomous vehicles, paving the way for realistic and cost-effective sensor simulations. The goal is to create an AD simulation environment adaptable both on Earth and in space.

 

Scheduled to kick off in April 2024, this research underscores TIER IV’s role as a tech company firmly committed to reimagining intelligent vehicles through innovation. The company is posed to amplify its efforts to advance AD technology and foster the realization of a sustainable society with each breakthrough.

 

Since 2020, TIER IV has partnered with the Matsuo Institute to pioneer the development of AI technology for autonomous driving, conducting basic component development into world models. World models can be used to approximate complex systems based on observations. By applying the models to autonomous vehicles, the companies aim to enable prediction of future conditions based on current sensor observations, and planning and learning of driving based on those predictions.

 

Through collaboration, TIER IV and the Matsuo Institute aim to drive the development of a scalable world model, utilizing multimodal data from autonomous vehicles, such as sensor, mapping and scenario data. The collaboration will also incorporate recent research involving large language models (LLMs) and NeRF, and the companies will build their own world model with their know-how and resources. The goal is to use the world model for prediction and driving planning in AD systems, including robotaxis, as well as for safety evaluations and AI model training.

World model overview

Example camera sensor output of a neural simulator (ground truth and model prediction outputs)

Example LiDAR sensor output of a neural simulator (ground truth and model prediction outputs)

Example reconstructed mesh output of a neural simulator (3D reconstruction results)

 

“World models have the potential to solve various challenges in autonomous driving, including robotaxis,” said Shinpei Kato, founder, CEO and CTO of TIER IV. “By leveraging TIER IV’s strengths in the development of AD technology in cooperation with the research institute and the educational institution, we will work to develop the technology and implement it in society. As the company leading the development of Autoware *2, we are committed to openly sharing this cutting-edge technology to create a foundation for more partners to develop technology not only for Earth but also for space.”

 

“The Matsuo Institute has been promoting the societal implementation of advanced technologies created in academia through joint research with the private sector, aiming to create a spiral of innovation through industry-academia collaboration,” commented Tofuku Kawakami, the president and representative director of the Matsuo Institute. “The purpose of the joint project with TIER IV is to realize autonomous driving by researching and developing the latest recognition technology using deep learning and world models, which have attracted a lot of attention recently. We aim to implement advanced technologies at high speed and promote initiatives to maximize their impact on society.”

 

“World models are a technological approach to modeling human intelligence in relation to the external environment, and companies in many countries have begun paying attention to and utilizing the technology,” remarked Prof. Yutaka Matsuo of the University of Tokyo’s Matsuo Laboratory. “I believe various services will become smart, intelligent and autonomous as world models get implemented in society in the future. I look forward to TIER IV making everyday life more comfortable through autonomous driving.”

 

*1 A technology that uses multiple photographs taken from various angles to reconstruct 3D scenes and generate images from new perspectives.

*2 Autoware is a registered trademark of the Autoware Foundation.

 

About TIER IV

TIER IV, the pioneering force behind the world’s first open-source software for autonomous driving Autoware, offers a range of advanced AD products and solutions, encompassing both software and hardware across multiple platforms. The company is steering the development of safe and efficient autonomous driving technology, aiming to reimagine intelligent vehicles through the art of open source. A founding member of the Autoware Foundation, TIER IV conducts cutting-edge research and development in collaboration with partners worldwide, harnessing Autoware to accelerate the rollout of autonomous vehicles that will benefit society as a whole.

 

About Matsuo Institute

Matsuo Institute was established to cultivate an innovation-centric "ecosystem" centered around the Matsuo Laboratory at the University of Tokyo’s Graduate School of Engineering. The research institute develops and implements research outcomes and technologies from the Matsuo Laboratory, aiming for widespread societal adoption and contributing to the enhancement of Japan's industrial competitiveness.

 

Media Contact
pr@tier4.jp