This blog post summarizes a presentation from ROSCon JP 2025 by TIER IV engineer Takahiro Ishikawa-Aso.
Overview
- In ROS 2, the presence of the Executor (a middleware-layer task scheduler) leads to nested scheduling between the middleware and the operating system (OS) layer, which complicates real-time scheduling.
- To address this issue, we devised a new Executor called CallbackIsolatedExecutor (CIE).
- With CIE, the nested scheduling problem is eliminated, and developers can directly assign priorities and core affinities to ROS 2 CallbackGroups.
- At TIER IV, the introduction of CIE into the open-source autonomous driving software Autoware—used as the foundation for our robotaxi—resulted in about 5x improvement in worst-case response time (measured).
This presentation, titled “CallbackIsolatedExecutor: A Novel Executor that Eliminates Nested Scheduling in Scheduling Theory,” drew on the academic paper “Work in Progress: Middleware-Transparent Callback Enforcement in Commoditized Component-Oriented Real-time Systems,” which was accepted at RTAS 2025, one of the top conferences in the real-time systems field. The paper is available as a preprint on arXiv and has also been published by IEEE.
In addition, the implementation of CallbackIsolatedExecutor is available in TIER IV’s GitHub repository.
The autonomous driving system developed by TIER IV is built on Autoware, an open-source autonomous driving platform. Autoware is an application implemented on top of the middleware Robot Operating System 2 (ROS 2), where the minimal functional units are provided as individual Nodes implemented as subclasses of the ROS 2 Node class, collectively forming a distributed system. For example, some Nodes correct distortions in point cloud data, while others handle obstacle avoidance.
A collection of Nodes implementing minimal functionalities achieves the overall autonomous driving capability through mutual Publish/Subscribe communications. As illustrated in the figure below, each functional unit, a Node, contains multiple Callbacks. A Callback is executed with the message received from a subscribed Topic as its argument, and then publishes a new message to the next Topic. Through this relay of Publish/Subscribe communications, multiple dataflows are formed across the system as a whole. These dataflows are generally subject to timing constraints, often referred to as end-to-end response time. End-to-end response time is defined as the duration from the occurrence of an event, such as an input to the system, to the completion of the system’s final response to that event.
In real-time systems—where not only logical correctness but also timing constraints are considered part of the correctness of the output—traditional systems have been often composed of simple control loops. For example, there might be an infinite loop triggered every 20 milliseconds, where each cycle must complete its processing within 10 milliseconds. However, in next-generation cyber-physical systems such as autonomous driving systems, it is necessary to handle timing constraints of dataflow-oriented tasks composed of multiple subtasks (Callbacks in ROS 2), as described earlier. If you are interested, try searching for research papers with keywords such as “DAG Scheduling” or “Cause Effect Chain.”
Autoware is also composed of multiple timing-constrained Directed Acyclic Graphs (DAGs). The figure below visualizes the set of timing-constrained DAGs of a robotaxi powered by Autoware, with some details omitted for clarity. We primarily recognize five DAGs as bottlenecks: Top LiDAR Preprocessing (pink), Localization (blue), Perception (green), Planning (yellow), and Control (orange). The red arrows represent DAGs with timing constraints whose execution times are extremely small relative to their deadlines, and thus they do not constitute bottlenecks. To optimize the response times of the five bottleneck DAGs, it is necessary to assign scheduling attributes—such as priorities and core affinities—to the individual Callbacks. In the research field known as DAG Scheduling, it is common practice to assign priorities and core affinities to each subtask within a task in order to satisfy timing constraints.
However, realizing DAG scheduling in ROS 2 systems is not straightforward. As illustrated in the figure below, this is because the responsibility for scheduling the execution of subtasks (Callbacks) lies in two layers—the middleware layer (Executor) and the OS layer (e.g., Linux)—resulting in nested scheduling. In other words, regarding the timing of a Callback’s execution, the middleware layer first decides when the Callback becomes executable and then delegates its execution to an OS thread. The timing of that thread’s execution, in turn, is determined by the OS scheduler. The real-time systems research community has developed scheduling theories under the assumption of such nested scheduling, but this complexity has hindered the effective application of traditional scheduling algorithms. For instance, much work had been conducted on pure DAG scheduling before the problem of nested scheduling emerged, yet those research assets cannot be directly applied. For prior studies on nested scheduling in ROS 2, please refer to the references in the CallbackIsolatedExecutor paper accepted at RTAS 2025.
To address this problem, we devised a new Executor called CallbackIsolatedExecutor, which can eliminate the issue of nested scheduling. The figure below compares the designs of the conventional Executors—SingleThreadedExecutor and MultiThreadedExecutor—with that of the new CallbackIsolatedExecutor. In the case of SingleThreadedExecutor, a single thread is shared among multiple Callbacks, making it impossible to assign independent scheduling attributes to each Callback. With MultiThreadedExecutor, whenever a Callback becomes ready to execute, it is assigned randomly to one of multiple threads, which again prevents assigning specific scheduling attributes. By contrast, CallbackIsolatedExecutor is designed so that each CallbackGroup is mapped one-to-one with a dedicated thread.
It is desirable to directly assign scheduling attributes to Callbacks, which correspond to the subtasks in DAG scheduling, and this can be achieved by allocating an independent CallbackGroup to each Callback. With the introduction of CallbackIsolatedExecutor, the scheduling problem of ROS 2 systems on Linux can be formalized as shown in the figure below. The Executor layer can now be ignored in real-time scheduling models, and from a practical standpoint, the significance of research on nested scheduling has been reduced.
In CallbackIsolatedExecutor, the scheduling attributes of each CallbackGroup can be configured simply by writing a single YAML file for the entire system. Each entry in the array-structured configuration format corresponds to the settings of one CallbackGroup. A detailed explanation of the configuration file is provided in the README.
A similar approach to CallbackIsolatedExecutor is REP-2017. As illustrated in the figure below, this proposal allows a ROS 2 application to pass a YAML configuration file to rcl, which the Executor then reads to set scheduling attributes. Unlike CallbackIsolatedExecutor, however, each entry in the YAML file corresponds to one Executor. As shown in the sample code below, by creating a SingleThreadedExecutor for each CallbackGroup and executing them on separate threads, functionality equivalent to CallbackIsolatedExecutor can be achieved.
Compared to REP-2017, CallbackIsolatedExecutor offers several advantages. First, it requires no modifications to rcl or rclcpp; installation of a standalone package is sufficient. As of September 2025, REP-2017 has not yet been merged, and it is uncertain whether it will be merged in the future. In addition, to configure real-time scheduling policies on Linux, such as SCHED_FIFO, it is usually necessary to grant special privileges like CAP_SYS_NICE to the process. In the case of CallbackIsolatedExecutor, however, such privileges need only be assigned to a single process called the Thread Configurator Node, which minimizes the privilege scope and thus provides a security benefit.
CallbackIsolatedExecutor is currently being introduced into Autoware and has been under early evaluation in TIER IV’s robotaxi. The figure below shows how much the response times of five DAGs were improved by optimizing them with CallbackIsolatedExecutor, compared with running under Linux’s default scheduler CFS without any special tuning (gray line). These results were obtained from measurements taken while the robotaxi was operating on public roads, with the horizontal axis representing elapsed time in seconds. As a result, the worst-case response time (measured) improved by about 5x, demonstrating a significant enhancement in real-time performance. While the scheduling algorithms, such as priority and core affinity assignment, are still heuristic-based, theoretical research is also ongoing.
The introduction of CallbackIsolatedExecutor is easy as described in the README. As of September 2025, it is available only via source build, but preparations are underway for distribution through the ROS build farm. CallbackIsolatedExecutor is implemented not only as a standalone package but also within Agnocast, a zero-copy middleware for ROS 2 developed by TIER IV. A presentation on Agnocast was held at ROS Japan UG #59, with another scheduled for ROSCon 2025. As with ROSCon JP, recordings will be available, so please take a look if you are interested.
With CallbackIsolatedExecutor and Agnocast being integrated into Autoware and TIER IV products, we are demonstrating how challenging research and development can deliver measurable performance in commercial autonomous driving.
Takahiro Ishikawa-Aso | System Software Team
A specialist in operating and real-time systems, Takahiro manages the system software team. He joined TIER IV in April 2022, following a stint as a part-time engineer from 2020. He holds a master's degree from the Graduate School of Information Science and Technology at the University of Tokyo and is currently enrolled in a PhD program.
TIER IV is always on the lookout for passionate individuals to join our journey. If you share our vision of making autonomous driving accessible to all, get in touch.
Visit our careers page to view all job openings.
If you’re uncertain about which roles align best with your experience, or if the current job openings don’t quite match your preferences, register your interest here. We’ll get in touch if a role that matches your experience becomes available, and schedule an informal interview.
Inquiries
- Media: pr@tier4.jp
- Business: sales@tier4.jp
Social media
X (Japan/Global) | LinkedIn | Facebook | Instagram | YouTube
More