This tech blog features content from a ROSCon JP 2026 presentation by TIER IV engineer Atsushi Yano.
The design details and performance evaluations of CIE are summarized in a paper accepted at RTAS 2025, a leading conference in the field of real-time systems. The paper is titled "Work in Progress: Middleware-Transparent Callback Enforcement in Commoditized Component-Oriented Real-time Systems." A preprint of the paper is available on arXiv and has also been published by the IEEE.
The implementation is available on GitHub, where you can also find documentation explaining the setup process.
The design and motivation behind CIE were presented at ROSCon JP 2025. This year's presentation assumed the audience had not seen last year's talk. We kept the design overview brief. Instead, we focused on a hands-on guide covering how to use CIE.
First, a brief overview of the structure of a ROS 2 application, which consists of a collection of nodes that are functional units. Each node has multiple callbacks. These callbacks receive messages from the topics they subscribe to, execute processing, and publish messages to the next topic. This publish-subscribe relay creates multiple data flows across the entire system. Essentially, a ROS 2 application is a data flow made up of multiple callbacks.
When and on which CPU core this callback is executed is determined by two schedulers working in tandem.
POSIX-compliant operating systems such as Linux allow you to set scheduling attributes (policy, priority and affinity) for each thread to control scheduler behavior. There are three typical policies. Affinity allows you to specify the CPU core on which the thread is scheduled.
|
Policy |
Configurable attributes |
Summary |
|
EDF scheduler (SCHED_DEADLINE) |
runtime / period / deadline |
Prioritizes the thread with the earliest deadline. |
|
FIFO scheduler (SCHED_FIFO) |
priority (99 - 1) |
Executes the highest-priority thread first. Uses FIFO when priorities are equal. |
|
CFS (EEVDF from Linux kernel version 6.6) (SCHED_OTHER) |
nice (-20 - 19) |
Allocates CPU time as evenly as possible across all threads. This is the default policy. |
The problem is that, as long as you use the standard ROS 2 Executor, you cannot set these scheduling attributes on a per-callback basis, making it difficult to optimize scheduling. Because SingleThreadedExecutor assigns a single thread to handle multiple callbacks, you cannot apply independent settings to each callback. MultiThreadedExecutor randomly assigns callbacks to multiple threads within a thread pool, preventing specific settings from applying to specific callbacks. NB: the recently added EventsExecutor was implemented for a different purpose – such as reducing timer callback release latency – and does not resolve this issue. This structure is the root cause hindering scheduling optimization for ROS 2 applications. CIE solves this problem.
The core design of CallbackIsolatedExecutor (CIE) creates a permanent one-to-one correspondence between a CallbackGroup and an OS thread. Since each CallbackGroup is assigned a dedicated OS thread, configuring the scheduling attributes for that thread directly applies those settings to the callbacks. This eliminates the need to consider scheduling at the middleware layer and resolves the issue of nested scheduling.
Scheduling attributes can be configured for the entire system with just one YAML file. The YAML template is automatically generated – the process is covered in the hands-on section below. Users simply need to fill in each property.
The Thread Configurator Node reads this YAML file and applies the settings to the CIE threads. The Thread Configurator Node uses system calls such as nice(2) , sched_setaffinity(2) , setpriority(2) , sched_setscheduler(2) to set the scheduling attributes for each thread.
Overview of how CIE works
For more information on CIE's design details and performance evaluations, please refer to the RTAS 2025 paper mentioned above.
Using a sample application, I’ll walk through six steps: from installing CIE to applying Linux scheduling. Similar instructions are also available in the official documentation.
CIE assigns a thread to each CallbackGroup. Therefore, as a preparatory step, we recommend creating an independent CallbackGroup for any callback that requires strict scheduling control.
// Declare and pass a dedicated CallbackGroup for each callback.
group1_ = create_callback_group(rclcpp::CallbackGroupType::MutuallyExclusive);
group2_ = create_callback_group(rclcpp::CallbackGroupType::MutuallyExclusive);
timer_ = create_wall_timer(3000ms, std::bind(&SampleNode::timer_callback, this), group1_);
rclcpp::SubscriptionOptions options;
options.callback_group = group2_; // Specify the subscription via options.
subscription_ = create_subscription<std_msgs::msg::Int32>("topic_in", 1, cb, options);
CIE is an independent package and requires no modifications to rcl or rclcpp. It can be installed via apt from the ROS 2 build farm.
sudo apt install ros-$ROS_DISTRO-callback-isolated-executor
You can also install the sample application used in this guide.
sudo apt install ros-$ROS_DISTRO-cie-sample-application
CIE is also included in Agnocast, a ROS 2-compatible zero-copy middleware developed by TIER IV, and can be installed via the Agnocast package. Use this option if you want to use CIE and Agnocast together.
For standalone nodes, modify the line that creates the Executor in the main function. This typically requires changing just one line.
+ #include "callback_isolated_executor/callback_isolated_executor.hpp"
- auto executor = std::make_shared<rclcpp::executors::SingleThreadedExecutor>();
+ auto executor = std::make_shared<CallbackIsolatedExecutor>();
For composable nodes, simply modify the component container in the launch file. You don't need to change a single line of code in the node.
- <node_container pkg="rclcpp_components"
- exec="component_container" name="sample_container">
+ <node_container pkg="callback_isolated_executor"
+ exec="component_container_callback_isolated" name="sample_container">
To create the YAML template, use CIE's prerun node. Launch the target application while the prerun node is running. The prerun node will automatically generate a YAML template file that lists all detected CallbackGroups.
# Terminal 1
ros2 run cie_thread_configurator prerun_node
# Once the target application has launched, exit with Ctrl+C
# Terminal 2
# Launch the target application
The generated template is in the format shown below. The CallbackGroup ID uses the format "node_name@callback_belonging_to_that_CallbackGroup," so you can tell at a glance which entry corresponds to which CallbackGroup. The scheduling attributes are populated with default values.
callback_groups:
- id: /sample_node@Subscription(/topic_in)
affinity: ~
policy: SCHED_OTHER
nice: 0
- id: /sample_node@Timer(1333000000)
affinity: ~
policy: SCHED_OTHER
nice: 0
- id: /sample_node@Timer(3000000000)
affinity: ~
policy: SCHED_OTHER
nice: 0
Set the scheduling attributes for each CallbackGroup in the generated template. Specify the configuration for CFS, FIFO scheduler and EDF scheduler as shown below. Configuration guidelines are covered in the next section.
callback_groups:
# CFS Example: Set nice value and CPU affinity
- id: xxxxx
affinity: [0,1]
policy: SCHED_OTHER
nice: -10
# FIFO scheduler example: Set real-time priority
- id: yyyyy
affinity: [2,3]
policy: SCHED_FIFO
priority: 50
# EDF scheduler example: Set runtime / deadline / period (unit: ns)
- id: zzzzz
affinity: [0,1]
policy: SCHED_DEADLINE
runtime: 10000000
deadline: 10000000
period: 20000000
For the Thread Configurator to set scheduling attributes for threads in other processes, you must configure the CAP_SYS_NICE capability. There are two methods to do this.
Method 1: Define Thread Configurator as a systemd service and grant CAP_SYS_NICE. This is the easiest approach.
# thread_configurator.service
[Service]
ExecStart=ros2 run cie_thread_configurator thread_configurator_node
AmbientCapabilities=CAP_SYS_NICE
Method 2: This involves executing setcap on the binary. As noted in the official documentation, this requires extra effort because LD_LIBRARY_PATH is disabled for security reasons.
sudo setcap cap_sys_nice+ep \
$(ros2 pkg prefix cie_thread_configurator)/lib/cie_thread_configurator/thread_configurator_node
Only this single Thread Configurator process requires special permissions. The main application requires no permissions at all. This provides a security advantage by minimizing the scope of permissions.
Launch the target ROS 2 application while the Thread Configurator node with the loaded YAML file is running. The scheduling settings are automatically applied.
# Terminal 1 (Not required if defined as a systemd service)
ros2 run cie_thread_configurator thread_configurator_node --ros-args -p config_file:=<your yaml>
# Terminal 2
# Launch the target application
With these six steps, Linux scheduling per callback is ready to run. Replacing the Executor is the only required change to the application code.
CIE enables configuration of scheduling attributes per callback, but how should they actually be set? To address this, the system requirements have to be clarified. Most robotics applications are real-time systems required to produce output within a specified time frame. Autonomous driving systems in particular are hard real-time systems where human lives are at stake. Therefore, a key requirement for autonomous driving systems is to satisfy all time constraints within limited computing resources.
While this problem is not easy to solve, a real-time scheduling research community exists to address this exact challenge. Decades of research assets exist from international conferences such as RTSS, ECRTS and RTAS, and we can utilize these resources. In fact, real-time scheduling research targeting ROS 2 systems alone has produced a vast number of papers, starting with work by Casini et al. (ECRTS 2019). These cover Executor modifications, scheduling attribute assignment algorithms and end-to-end timing analysis. Many of these methods directly analyze the standard ROS 2 Executor – and by extension, nested scheduling.
By treating callbacks as vertices and topic communications as edges, a ROS 2 system can be modeled as a directed acyclic graph (DAG). Real-time scheduling for DAGs has been actively researched since around 2010. Researchers continue working to find optimal algorithms.
Until now, these research findings rarely reached deployed ROS 2 systems. Analysis methods built on the assumption of nested scheduling are overly complex to be routinely applied. Meanwhile, theoretical DAG scheduling algorithms do not assume nested scheduling. By structurally eliminating nested scheduling, CIE enables you to directly apply established theory for independently OS-scheduled task groups to ROS 2 callbacks. In short, CIE serves as a bridge connecting the ROS community and the real-time scheduling research community. Once you find a scheduling algorithm from existing research assets that fits your system requirements, you simply set the scheduling attributes based on that approach.
In Autoware, open-source software for autonomous driving pioneered by TIER IV, almost all key nodes have been replaced with CIE (Autoware GitHub Discussion). We plan to replace the Executors across all nodes with CIE in the future.
Here, we present the results of applying CIE to TIER IV’s robotaxi and optimizing the scheduling attributes of five bottleneck DAGs: Top LiDAR Preprocessing, Localization, Perception, Planning and Control. For priority assignment, we referenced this paper and adopted a strategy of assigning higher priorities to callbacks belonging to longer paths. This ensures the critical path of the DAG is processed as fast as possible.
The graphic below shows the response time improvement (from the release of the source vertex to the completion of the tail vertex) across the five DAGs compared to running the default Linux scheduler (CFS) without custom tuning (gray line). These measurements were captured while driving the robotaxi on public roads in Odaiba. The horizontal axis represents elapsed time in seconds.
As a result, measured worst-case response times showed an approximate 3- to 5-fold improvement, and jitter was also reduced. For systems that must run heavy workloads on limited computing resources, such as autonomous driving platforms, optimized scheduling can deliver significant performance improvements.
It takes only six steps to get from installation to applying schedules. If you have a ROS 2 application facing response time or jitter challenges, give it a try.
This work is based on results obtained from the New Energy and Industrial Technology Development Organization (NEDO) Green Innovation Fund Project “Development of In-vehicle Computing and Simulation Technology for Energy Saving in Electric Vehicles” (JPNP21027).
Atsushi Yano | System Platform Development Department
Atsushi heads the Real-Time Group at TIER IV, working on R&D for OS and middleware to maximize the real-time performance of autonomous driving systems. He is concurrently enrolled in a doctoral program at Saitama University’s Azumi Laboratory, researching real-time scheduling theory for autonomous driving systems.
TIER IV is always on the lookout for passionate individuals to join our journey. If you share our vision of making autonomous driving accessible to all, get in touch.
We’re currently hiring for the following positions:
Visit our careers page to view all job openings.
If you’re unsure which roles fit your experience, or if the current job openings don’t quite match your preferences, you can register your interest here. We’ll contact you when a suitable role becomes available and arrange an informal interview.
Inquiries
Social media
X (Japan/Global) | LinkedIn | Facebook | Instagram | YouTube
More