At ROSCon JP 2024, TIER IV software engineer Yutaka Kondo shared insights into some of the challenges in autonomous driving AI and delved into TIER IV’s use of multi-container development within Autoware, the world’s first open-source software for autonomous driving. An edited excerpt of Yutaka's presentation follows.
I'd like to share some of the challenges we're facing with AI in autonomous driving and talk about the work being done with multi-containerization. TIER IV is heavily involved in the development of Autoware, which is built on Linux and ROS 2. We're working on its development alongside the Autoware Foundation, a global industry organization. Autoware is compatible with over 30 vehicle models and is used by more than 500 companies. It’s available on GitHub and currently has about 9,000 stars. We’re aiming to hit 10,000 by the end of the year, so if you haven’t already, please give it a star!
An autonomous bus service operates year-round in Komatsu, Ishikawa Prefecture.
Autoware is already powering vehicles on real roads. Together with our partners, TIER IV is running an autonomous bus service between Komatsu Station and Komatsu Airport in Ishikawa Prefecture. It’s been operating since spring, and over 10,000 people have used the service so far.
Autoware is made up of a variety of ROS 2 packages that are grouped into a component that we refer to as the autonomy architecture. It includes over 100,000 lines of code and more than 400 ROS 2 packages, making containerization essential. We also need to compile them for AMD64 and X86 architectures, as well as ARM64 (AArch64). Plus, since we use GPUs, we have to factor in the presence or absence of CUDA drivers and a range of configurations to ensure efficient builds.
The software is licensed under Apache 2, which means we can take advantage of a lot of free services on GitHub. With GitHub-hosted runners, we can run CI/CD pipelines and perform testing at no cost.
From single-stage builds to multi-stage builds
Let's take a moment to introduce the progress made so far in the multi-containerization of Autoware. About six months ago, only single-stage builds were available. The development container stage, labeled devel, was copied into the runtime environment to create the execution container, maintaining a setup close to a single-container configuration.
From that point, we created a separate stage dedicated solely to generating the dependency package list for rosdep installation. This allowed us to split the build stages, improving compilation efficiency. After further stage separation and optimizations, we created development containers that isolate the components for sensing, such as LiDAR and cameras, and the AI vision components, which handle tasks like object recognition, from the all-in-one Autoware container. By providing an execution container as a copy of the final output, we took the first step toward multi-containerization.
Separation of development containers for sensing/perception processing
Now, we also have containers for planning and localization, with development containers and execution containers available for both.
When developing autonomous driving software, it is not always simple to install the necessary applications and sensors in a real vehicle. However, by preparing a simulator, rosbag files, and only the packages you want to work on, you can volume-mount them, build, and debug in the development container, making it possible to develop autonomous driving software on a single computer.
Meanwhile, the execution container is designed to run Autoware on real vehicles, and contains all necessary environment settings and components. The latest version of Autoware is compatible with the container.
Reducing container build time
We have made numerous improvements throughout the transition from single-stage builds to multi-stage builds. As a result, the time it takes to build the container has decreased significantly – from 2.5 hours just a few months ago to about 40 minutes now, and in the best cases, even down to around 5 minutes.
Autoware container build using GitHub-hosted Runner
The size of the container impacts the data transfer size during docker pull and docker push, as well as disk space usage. Previously, the container size was 9.3 GB, but we have reduced it to 4.2 GB. If the CUDA driver is not included, the size is further reduced to about half of that again.
Autoware container image sizes
This shows that with creativity, even in environments like GitHub-hosted runners, which offer somewhat limited CPU resources, we can still develop efficiently.
Container build tips using GitHub Actions
Let me introduce the methods implemented to reduce build time. To start, I want to mention something unrelated to ROS – a CI/CD tool called GitHub Actions. If you don’t enable the Docker build cache properly in this tool, you won’t have to worry about it when you’re doing a local Docker build, but the container build layer cache will not be cached. The cache can be configured to be stored in the registry. As a result, the cached layers can be easily retrieved from the GitHub Container Registry (GHCR) and reused.
If you specify cache-from for both the main branch and the pull request branch, you can utilize the cache even when the pull request gets quite large and needs many builds. This way, the cache from the main branch is also reflected. Plus, when using cache-to, caching all stages ensures that the maximum build results are cached.
However, when working with large ROS package development like Autoware, building hundreds of packages in a single step using colcon build can sometimes take one to two hours. In those kinds of situations, the BuildKit Cache Dance GitHub Action is useful, as it extracts files from the image, restores them and caches them. This action allows us to create and restore a C++ compilation cache with Ccache, making it possible to achieve over 90% cache efficiency when modifying just a few packages.
This is specific to ROS, but we also want to improve the cache retention with rosdep. It’s pretty common for rosdep to reinstall everything, even when only the source code has changed and package.xml hasn’t been touched. To avoid that, we first copy only the necessary parts of the source code in a separate stage, and then we create a dependent package list in advance for the rosdep install. We create that file and install, using it just before the build stage. This allows us to skip the installation of dependent packages if there are no changes in the list of dependencies. This technique is incredibly effective. And just like with Ccache, storing the downloaded APT packages in the GitHub Actions cache allows us to minimize download time.
When bind-mounting the source code, only the necessary parts for each stage should be mounted – such as the sections for sensing and object recognition (perception). Additionally, instead of installing to the current directory, the installation path should be changed to /opt/autoware. This way, it is sufficient to copy the final output in /opt/autoware to the execution container or the next stage. Of course, if we delete the intermediate build directory created in the current directory right away, it won't negatively impact the container image size, which helps keep things lightweight.
Finally, to minimize the size of the execution container, it’s essential to combine the previously mentioned methods. When installing rosdep, be sure to set the dependency type to --dependency-types=exec so that only the exec_depend packages are installed.
Also, by completely removing unnecessary header files and development files from the root directory, and ensuring that only the final output is copied from the development container, a minimal-sized ROS container can be generated.
The procedure for creating an Autoware package using the development container is as follows:
- Clone the Autoware repository.
- Import the submodules using vcs import.
- Launch the development container with docker run, mounting only the necessary packages as volumes.
- Run colcon build inside the container, including the debug option.
This approach allows you to build only what you need for development while attaching GNU Debugger (GDB). By doing this, you can build in your local environment, whether you’re using Mac, Linux, or Windows.
I'm in the process of developing a method using Visual Studio Code, and I look forward to sharing that in the future.
From Autoware to Pilot.Auto
At TIER IV, we aim to create a clear pathway from Autoware to our commercial product, Pilot.Auto. The first step is to remove Autoware's demo functions from the Pilot.Auto container and connect it to the next version. This will streamline the process, allowing Pilot.Auto to inherit features as separate containers.
We plan to remove debugging simulators and evaluation tools to establish a multi-container framework. For instance, if the application is for a logistics robot, we’ll add features specific to logistics. If it’s for a shuttle bus, we’ll include features tailored for shuttle buses. Ultimately, our goal is to integrate this into the Pilot.Auto product lineup.
From OSS to product
A product specifically for race cars is another topic I want to touch on. We plan to extract specific containers from Autoware and integrate them into a system for racing competitions. This will allow us to develop software specifically for race cars.
2024 Japan Automotive AI Challenge
This leads us conveniently to the Japan Automotive AI Challenge, an event aimed at nurturing the next wave of talent in autonomous driving tech, particularly in the context of CASE (Connected, Autonomous, Shared, Electric) and Mobility-as-a-Service (MaaS). Competitors get to develop software focused on real-world applications for self-driving vehicles. It’s designed for everyone – from total newcomers to seasoned developers. Participants have the chance to test their skills on real tracks and connect with industry experts, making it a great place to network and learn.
This year’s event kicked off six months ago, with the qualifying rounds wrapping up in September on a digital twin simulation circuit. The semifinals and finals are taking place in November at Tokyo Bay City Circuit in Odaiba, featuring electric racing karts. If you’re in Tokyo, be sure to check it out!
Yutaka Kondo | Autoware Engineering team
Yutaka joined TIER IV in April 2024 and is currently responsible for Autoware version control, multi-containerization, the CI/CD pipeline, component redesign and overall optimization of the open-source software. Previously, he developed autonomous robots at an industrial robot manufacturer, an AI startup and a household robot company. Yutaka also serves on the program committee for ROSCon and the executive committee of ROSCon JP. His book, “Getting started with ROS 2: Next-generation robot programming,” was released in September 2024.
TIER IV is always on the lookout for passionate individuals to join our journey. If you share our vision of making autonomous driving accessible to all, get in touch.
Visit our careers page to view all job openings.
If you’re uncertain about which roles align best with your experience, or if the current job openings don’t quite match your preferences, register your interest here. We’ll get in touch if a role that matches your experience becomes available, and schedule an informal interview.
Inquiries
- Media: pr@tier4.jp
- Business: sales@tier4.jp
Social Media
X (Japan/Global) | LinkedIn | Facebook | Instagram | YouTube
More