TIER IV is leading efforts to deploy unmanned autonomous mobility services, and as the business expands, Autoware – the open-source autonomous driving software we develop – is being integrated into a variety of vehicles across Japan.
I’m Satoshi Ota, a member of the Planning/Control team, which has been developing the core functionalities required for autonomous driving in urban areas. In this tech blog, I’ll focus on obstacle avoidance, providing an overview of the feature and explaining our evaluation framework.
Planning module overview
Since it's been a while since we last covered a planning-related topic in the tech blog, let’s start with an overview of Autoware’s planning component.
The role of the planning component is to integrate information from other modules, such as perception and localization, and determine the path the vehicle should follow. The current interface defines the primary inputs and outputs of the planning component as follows:
Input
- Odometry (from the localization component)
- Surrounding object recognition results (from the perception component)
- Map data (from the map component)
- Destination (from the API)
Output
- Path the vehicle should follow (to the control component)
The architecture of Autoware's planning component, currently available as open-source software, is divided into three layers: the mission layer, behavior layer, and motion layer. These layers operate sequentially, with the results processed in each layer passed down to the next.
Planning architecture
Mission layer
The mission layer functions like a car navigation system, determining the route the vehicle should take based on its current position and destination. Autoware also provides APIs for setting destinations externally, enabling remote destination configuration through the Fleet Management System (FMS), one of the services in Web.Auto. The information handled at this layer is at a fairly coarse level, such as “which lane to drive in” or “where to turn left.” The behavior layer uses this information to determine more detailed vehicle actions.
Behavior layer
The behavior layer is responsible for ensuring the vehicle adheres to traffic rules and safely navigates on roads. Using the route information planned in the mission layer, the behavior layer makes the necessary decisions and plans to comply with traffic rules, outputting the following information:
- The path the vehicle should follow
- The maximum speed at each point along the path
- The area in which the vehicle can drive
In the simplest use case of lane following, the "path the vehicle should follow" is planned along the centerline of the lane included in the route. The "maximum speed at each point along the path" is the legal speed limit for that lane, and the "area in which the vehicle can drive" is the area within the lane. This is sufficient for simply driving along a designated lane. However, on public roads, the vehicle must also be able to handle various use cases, such as stopping at red lights or changing lanes to make a right turn.
In the current Autoware architecture, most of the functionalities for handling use cases are implemented in the behavior layer, which is the largest layer in the planning component. There are already more than 20 functions for planning routes and vehicle speeds. The module for obstacle avoidance is also included in this layer.
ℹ️ Each function in the behavior layer can be easily toggled on or off in the configuration file, according to user requirements. The system is designed to be adaptable to a wide range of applications, from advanced robotaxis to simple delivery robots.
Motion layer
The role of the motion layer is to plan a path that follows the instructions from the behavior layer as closely as possible, taking into account the vehicle's kinematics and dynamics. Sometimes, the paths output by the behavior and motion layers will be almost identical, but when planning a route for a larger vehicle, the motion layer may adjust the path to ensure the vehicle stays within its lane during turns, for example, by slightly modifying the route to take a tighter turn. Additionally, while the behavior layer provides the maximum speed for each point along the path, the motion layer plans a speed profile (target speeds for each point along the path) within that limit, ensuring the vehicle can travel comfortably and safely.
Details of the various features supported by Autoware can be found in the official documentation.
Obstacle avoidance mechanism
In Autoware, the essential functions required for autonomous driving – such as obstacle avoidance, adaptive cruise control, and lane changing – are implemented as independent modules. Most modules are structured with a decision-making component and a planning component that operates based on the decision results. The obstacle avoidance function follows this same structure.
Decision-making
To avoid obstacles, the system needs to be able to judge which objects are obstacles that need to be avoided. When drivers encounter a parked vehicle in their driving path, they turn on their indicators and steer around the vehicle to avoid it. Similarly, it may be necessary to avoid large vehicles waiting at a traffic light in an adjacent lane or pedestrians on the roadside – there are many situations where avoidance is required. On the other hand, in a scenario where a vehicle ahead is waiting at a traffic light in the same lane, the expected action is to stop behind that vehicle. In normal cases, there is no need to avoid it. For autonomous driving to feel truly seamless, these decisions must be made with precision.
⚠️ An impatient driver might choose to switch to an empty lane when encountering vehicles waiting at a red light, but that would be considered overtaking. As a result, this behavior has been excluded from the scope of obstacle avoidance.
Avoiding a parked vehicle in the same lane
Avoiding a vehicle in the adjacent lane
Avoiding something other than a parked vehicle
Coming to a halt behind a temporarily stopped vehicle
The Planning/Control team has been improving obstacle avoidance decisions by examining the reasoning behind human judgments and applying this information to create implementable conditions. The illustrations above are just a few examples. The process involves defining the "use cases where avoidance is necessary" and the "use cases where avoidance is not required." The decision-making process for "whether to avoid or not" operates based on a set of rules – if/else statements – that are currently configured by engineers.
It can generally be assumed that stationary objects a certain distance away from a vehicle’s travel path do not need to be avoided, regardless of what they are. Therefore, a fixed-width area around the route is defined, and stationary objects outside this range are not treated as obstacles that need to be avoided.
On the other hand, parked vehicles that obstruct a vehicle's travel path are considered obstacles to be avoided. However, vehicles that are temporarily stopped due to traffic lights or congestion must be correctly classified as such, based on a correlation between a vehicle's stopping position and the likelihood that it is a parked vehicle. Specifically, we calculate the ratio of the actual shift distance (i.e., how far the vehicle has moved laterally) to the shiftable length (i.e., how far the vehicle can move when fully shifted to the curb). If the ratio exceeds a certain threshold, the vehicle is classified as a parked vehicle.
Identifying parked vehicles
Extracting parked vehicles
It's a very simple rule, but with this alone, many of the parked vehicles encountered in urban areas can be identified. By layering these conditions, the obstacle avoidance judgment function is improved and, after safety evaluations, it is released. More details can be found in the documentation.
Handling edge cases
There are edge cases that simple rules like the ones above cannot handle. For example, in Tokyo’s Odaiba district, where TIER IV regularly conducts autonomous driving tests, we have encountered vehicles parked far from the curb or even parked on the right, beside the central reservation.
Left: A parked taxi in Odaiba, Tokyo, is positioned with a wide gap between the curb.
Right: A stationary truck on the right is parked beside the central reservation in Odaiba, Tokyo.
In both cases, it is difficult to determine whether or not to avoid the vehicle. However, for Level 4 autonomy, it is necessary to develop decision-making functions that can handle such situations. For instance, a rule could be considered for a vehicle not parked close to the curb, where, if the vehicle has been stationary for a certain period for no apparent reason, it could be classified as a parked car and avoided.
Vehicle in front comes to a halt for no apparent reason.
Vehicle in front comes to a halt for reasons such as a nearby pedestrian crossing.
Performance is gradually being improved by collecting edge cases, refining functions, and repeating the test loop at high frequency. However, when an edge case is found, it's a challenging task to develop explainable assumptions and conditions based on solid reasoning, which has been one of the most difficult aspects for engineers. Thanks to persistent effort, the accuracy of the obstacle avoidance function has improved to the point where it can be used in urban areas. Nevertheless, some edge cases remain that still require further refinement.
To address these edge cases, it’s necessary to enhance not only basic information like the position and velocity of objects, but also recognition performance and the ability to infer the intentions of surrounding drivers based on interactions with nearby vehicles. Machine learning is also increasingly being applied to decision-making. For instance, a paper titled "Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving" explains how the use of the MGAIL imitation learning model, trained on extensive data from skilled drivers, enables the system to make decisions that mirror natural decisions made by humans in many situations.
Autoware currently uses a rule-based decision-making algorithm. However, TIER IV is working to integrate machine learning into the Planning module. We are also developing a framework that allows for the incorporation of different internal algorithms into Autoware, as long as the module interface is compatible. TIER IV aims to provide an architecture that can quickly integrate research outcomes from universities and companies, positioning Autoware as a leading platform for autonomous driving development.
ℹ️ TIER IV has already developed turn signal detection and vehicle intent estimation, with plans to integrate these features into Autoware in the near future. TIER IV is also working on incorporating operator assistance. In cases where the system cannot confidently make a decision, such as unexpected edge cases, we are implementing a mechanism to delegate part of the decision-making process to a human operator.
Path planning
Next up is the planning aspect of obstacle avoidance. There are various methods for path generation, including sampling-based and optimization-based approaches. In global path planning at the behavior layer, spline-based path generation is primarily used. To maintain ride comfort, jerk constraints are applied to prevent abrupt changes in motion. When generating paths that involve steering, such as obstacle avoidance or lane changes, the maximum third derivative of the lateral displacement (l) with respect to the longitudinal displacement (s) in the Frenet coordinate system is constrained to stay within the specified jerk limit, as shown in the figure below.
Although the derivation is omitted, assuming that the speed during avoidance remains constant, the longitudinal distance required to achieve the target displacement can be expressed by the formula below. This formula is based on the target displacement (L), the vehicle speed during avoidance (V), and the upper limit of the third derivative of the lateral displacement (l) with respect to the longitudinal distance (s), denoted as (J).
s = 4.0 * V * (0.5 * L/J)^(1/3)
Profile during shift
In Autoware, parameters can be set to determine how much clearance is needed when passing a parked vehicle, with the lateral displacement during avoidance being determined by the object’s posture and shape, as well as the parameters of the avoidance module. In the robotaxi being developed by TIER IV, the vehicle is configured to pass parked vehicles with a lateral clearance of up to 1.5 meters, assuming the possibility of a door suddenly opening.
The shift must be completed before passing a parked vehicle. The shift completion position and the longitudinal distance to the parked vehicle (referred to as the "front longitudinal buffer" in the figure below) are also configurable parameters. This helps determine the shift completion position, and, using the formula above, the required longitudinal distance needed to keep the jerk within the constraint limit can be calculated. This, in turn, helps identify the starting position for the avoidance maneuver (when steering begins).
After that, the avoidance path is generated by using a curve that connects the starting and ending points of the avoidance maneuver while adhering to the jerk constraint. The same process is used to generate the path that returns to the center of the original lane from the shifted position. Like the decision-making process, this is a simple method. Results from simulations and real-world experiments on public roads with actual vehicles showed that this path generation method can handle straightforward use cases, such as avoiding several parked vehicles in the same direction.
Here, it is assumed that the vehicle's speed during the avoidance maneuver is constant. However, it is also possible to generate a path that ensures the third derivative of lateral displacement over time (lateral jerk) remains below a certain value, even if the vehicle undergoes constant acceleration during the maneuver.
In the behavior layer, there are modules for generating paths for maneuvers such as lane changes, which also involve lateral shifts. These functions have been organized into a library that can be easily included and used by any module. For more details, please refer to the documentation.
Path generation improvement
The division of roles between each layer is as follows:
- Behavior layer: Path planning for obstacle avoidance along the route.
- Motion layer: Fine-tuning to drive along the path planned by the behavior layer, taking into account the vehicle's kinematics.
The granularity of path planning differs between the behavior layer and the motion layer. Rough path planning is performed in the behavior layer, while more detailed planning takes place in the motion layer.
In our tests, a relatively simple sequence of shifting right, going straight, and shifting left has generally been enough to avoid obstacles such as parked cars. This is why it was incorporated into the behavior layer.
However, for more complex scenarios, such as avoiding obstacles on both sides along an S-shaped path or navigating around moving obstacles, the method using spline curves described above may not be optimal due to its limited flexibility in path shape.
Therefore, we are currently considering enhancing the functionality of Autoware by utilizing not only the obstacle avoidance in the behavior layer but also the path planning module of the motion layer, while maintaining the existing architecture of the planning component.
The motion layer of Autoware includes a module that calculates the final path shape that the vehicle can follow by imposing constraints such as drivable area and vehicle kinematics. Originally developed to adjust the path for narrow roads while considering the vehicle's kinematics and ensuring the footprint stays within the drivable area, this feature's strength lies in its ability to plan complex paths through optimization.
Taking advantage of this strength to avoid obstacles even in complex scenarios, the behavior layer not only determines which objects to avoid but also excludes the vicinity of obstacles identified as 'to be avoided' from the drivable area (as shown below). This enables the motion layer to plan a route that avoids obstacles while trying to keep the route within the drivable area.
This feature is currently in the testing phase but is already available as open-source software. By fully utilizing the motion layer, it is possible to handle difficult use cases, such as avoiding a pedestrian walking on the road and navigating between vehicles parked on both sides of the road.
So far, we have gone over Autoware’s obstacle avoidance function, divided into decision-making and planning. While the individual logic components are simple, they provide the essential functionality required for autonomous driving on public roads. In the future, we plan to incorporate advanced techniques such as machine learning and optimization to accelerate improvements in performance while maintaining a strong focus on safety. Previously, efforts were focused on integrating Autoware into Toyota JPN TAXI vehicles for evaluation. Efforts have since shifted to integration into EV buses. Developing modules adaptable to various vehicle types as a platform presents a significant challenge. It requires building a system capable of accommodating differences in vehicle types through tuning and options. The Planning/Control team is developing software with both versatility and performance in mind.
Planning/Control function evaluation
Defects in autonomous driving systems can have a direct impact on human lives. High-quality standards are particularly important in the Planning/Control component, which determines the vehicle's final behavior. However, since Autoware is a large-scale system with many interconnected modules, preventing bugs through mere vigilance during coding is not enough. Below, we will introduce how the functions we have developed are evaluated.
Evaluation infrastructure
To maintain high quality while keeping the development cycle moving efficiently, an automated evaluation system is essential. Even if a great implementation idea comes up, manually checking for degradation in all the diverse use cases that Autoware supports would slow down the development process and ultimately reduce the time available for implementation. Also, while writing unit tests is sufficient when adding a single function to a library, a dedicated evaluation framework is required to assess the overall behavior of the system when integrating new features.
We use a tool called scenario_simulator_v2 to evaluate the Planning/Control component. Developed primarily by the Simulator team, it enables simulation-based evaluations by defining the driving environment and expected system behavior to check if the system performs as intended.
We continuously review how the autonomous driving system should behave in each use case, and these are managed in the form of scenarios. The number of scenarios is vast, but by using this tool for evaluation, we can easily understand which use cases are supported by the current version of Autoware. Additionally, it allows for easy comparison with past evaluation results, helping us quickly detect any degradation.
A related use case for the obstacle avoidance function involves a scenario where a parked vehicle blocks the autonomous vehicle's lane, requiring it to navigate around the obstacle to reach its destination. The expected system behavior can be precisely defined, and in this use case, the success and failure criteria for evaluation are as follows:
Success criteria
All of the following conditions must be met:
- The vehicle triggers its right turn signal when avoiding the obstacle.
- The vehicle signals left when returning to the original lane.
- The vehicle reaches its destination.
Failure criteria
Any of the following conditions will result in failure:
- The vehicle collides with the parked vehicle.
- The distance between the vehicle and the parked vehicle is 1.0 meter or less.
- The vehicle does not reach the destination within 180 seconds (timeout).
This is a simple use case, but because the driving environment can also be configured in detail, it is possible to evaluate various scenarios, including those with multiple parked vehicles or moving objects. Scenarios are classified and managed based on difficulty and context, with more than 4,000 scenarios for the Planning component function evaluation as of 2024. About 10% of these scenarios focus on obstacle avoidance. TIER IV’s CI/CD team has built a system that allows evaluations to be run on a large number of scenarios in the cloud using scenario_simulator_v2. This enables developers to easily conduct regression testing of newly added features via a browser.
ℹ️ TIER IV has also developed an evaluation platform for the perception component, which similarly enables evaluations to be conducted in the cloud. For more details, visit the Driving Log Replayer page on GitHub.
Real-world evaluations
After thorough validation using the evaluation platform mentioned above, we proceed to real-world testing. In real-world evaluations, we cannot use frequently updated mainstream branches, such as the main branch of autoware.universe, due to safety concerns. These branches often include numerous changes unrelated to the functionality developers wish to test, which could introduce unforeseen bugs and disrupt experiments.
To address this, TIER IV conducts experiments using released branches evaluated by the System Integration and Field Integration teams. These branches have already undergone release evaluations, during which identified bugs are resolved. Additionally, we have a clear understanding of the potential issues that might occur with these branches.
This approach minimizes the risk of unexpected bugs. Moreover, if a bug does arise, it is easier to determine whether it has been caused by newly introduced changes. For this reason, the development team adds only the new features to the released branch before proceeding with real-world evaluations.
Released branches play a crucial role in the development process. Releases are made on a product-by-product basis, and the frequency varies, but recently, thanks to improvements in the release process and the efforts of the teams involved, we have been able to update the release branch every two weeks. This has not only created an environment where users can quickly test new features, it has also enabled developers to conduct evaluations in environments that are closer to the final mainstream branch. As a result, we have been able to reduce the number of bugs that appear after merging features into the mainstream. While it may not be a topic that gets much air time, these small improvements play a key role in supporting the development of Autoware.
ℹ️ A dedicated team has been established to focus on improving the release process. The details of TIER IV's release process will be introduced at another opportunity.
Wrap-up
This blog post focused on Autoware’s obstacle avoidance functionality, while also giving an overview of Autoware development, which involves collaboration across multiple teams. Core teams like Planning/Control and Perception are central to this, but other teams also support testing, vehicle integration, environment setups, and the evaluation infrastructure introduced here. Developing autonomous driving technology is a team effort, and people from all backgrounds can make an impact here. If you're interested in joining us, check out our recruitment page.
Satoshi Ota | Planning/Control team
Satoshi joined TIER IV in April 2021 after completing a master's degree at the University of Tokyo’s Department of Aeronautics and Astronautics. He currently leads the development of features such as route and speed planning within the Planning/Control team.
TIER IV is always on the lookout for passionate individuals to join our journey. If you share our vision of making autonomous driving accessible to all, get in touch.
Visit our careers page to view all job openings.
If you’re uncertain about which roles align best with your experience, or if the current job openings don’t quite match your preferences, register your interest here. We’ll get in touch if a role that matches your experience becomes available, and schedule an informal interview.
Inquiries
- Media: pr@tier4.jp
- Business: sales@tier4.jp
Social Media
X (Japan/Global) | LinkedIn | Facebook | Instagram | YouTube
More