Zero human labels, zero human code: OSS release of a reference E2E AI model

TIER IV has built an end-to-end (E2E) autonomous driving AI model developed autonomously by an agentic AI, using nationwide Japanese driving data collected on Co-MLOps, a data-sharing platform for democratizing autonomous driving AI development, and labels generated by the autolabeling platform CoMET. Dubbed METEOR, the model does not depend on high-definition (HD) maps: from surround-camera video alone, a single neural network performs bird's-eye-view (BEV) perception of the surroundings through to trajectory generation. Its distinguishing point is that it was developed with no additional human labeling, with the train–evaluate–improve loop executed by an agentic AI. Data augmentation with NVIDIA Cosmos (condition transfer and rare-case generation) is combined to cover conditions that real data alone cannot. The source code, trained model and training recipe have been released as open-source software (OSS). Companies participating in the Co-MLOps project can keep improving the model with shared data while extending it with their own data and features.

Autonomous driving AI development is shifting from a stage where model designs are hand-crafted to one where data is collected, models are trained, and evaluation and improvement are repeated. In 2024, TIER IV established Co-MLOps, a data-sharing platform on which participating companies share driving data and grow autonomous driving AI together. Building on the nationwide Japanese driving data on the platform, we have been working to harness that data to create an autonomous driving AI that runs in a real vehicle, through autolabeling and development driven by agentic AI. This article introduces the result: a reference E2E AI model, covering what is inside the model, and how the agentic AI carries the development forward. The model is scheduled to be demonstrated on an in-vehicle computer at Automotive World 2026 in greater Tokyo, September 9-11.

This post focuses on the reference E2E AI model and its development. For an overview of Co-MLOps data collection, autolabeling with the Collaborative Multi-stage Ensemble-based Teacher Model (CoMET) and NVIDIA Cosmos integration, read the TIER IV tech blog “Building a Dataset Foundation for Autonomous Driving with NVIDIA Cosmos” (August 7, 2026).

What is the reference E2E AI model? A baseline powered by Co‑MLOps

Co-MLOps collects and shares nationwide Japanese driving data from participating companies' vehicles, advancing the democratization of data. Data is the key asset that determines the performance of autonomous driving AI, but data on its own creates no value. Only with a mechanism for labeling and for model development does data contribute to AI performance.

The reference E2E AI model is one of Co-MLOps' answers to this. It serves as a baseline model that Co-MLOps participants can continuously train, update and improve on the platform using the shared data. The source code, trained model and training recipe are released as OSS, and participants can build on these resources to evaluate on their own data, fine-tune, and add their own features. Because labeling is handled by the CoMET autolabeling platform and model development by an agentic AI, an E2E autonomous driving AI model can be developed efficiently starting from driving data.

Fig. 1 shows the Co-MLOps concept. It has been designed as a shared platform where the benefit to everyone grows as more companies join, with the reference AI continuously updated with large-scale data. Fig. 2 illustrates this cycle from the perspective of the reference E2E AI model. CoMET labels data collected by Co-MLOps participants and an agentic AI develops the model. Following the OSS release, participants extend the model independently. The resulting new data and improvements are shared back to feed the next development cycle.

Fig. 1. Co-MLOps overview

Fig. 2. The loop that produces the reference E2E AI model

Model overview: Single camera-only network that does not depend on HD maps

The core of the reference E2E AI model is a single neural network that takes only surround in-vehicle camera video as input and does not depend on HD maps. METEOR stands for Multi-task Estimation of Traffic Elements, Objects & Roads, which describes the design policy of estimating traffic elements, objects and roads in a multi-task manner.

Main functions: BEV perception, E2E trajectory generation, in-vehicle execution

• BEV perception: Road-surface segmentation out to long range (lane lines, stop lines, crosswalks and so on), 3D object detection of vehicles and pedestrians (with heading and velocity), an occupancy grid that divides the surroundings into cells and judges whether each cell contains an obstacle, and traffic-light state, all output simultaneously. Road-surface segmentation is the problem setting of online map construction, estimating road structure in BEV from cameras without a map [1][2].
• E2E trajectory generation: The same network directly outputs the ego trajectory conditioned on a driving command (straight, left or right). A rule-based safety check (guardrail) operates independently alongside the network.
• In-vehicle execution: The model is built from the design stage using only operators that map well onto TensorRT, so it can be deployed to NVIDIA Jetson Orin via ONNX. On real hardware we have confirmed that it runs at a practical latency on a single Jetson Orin GPU (performance figures are being improved continuously)

Architecture overview

Image features from the surround cameras are transformed into the bird's-eye-view (BEV) space with depth, fused over time, and the perception, prediction and trajectory-generation tasks are output simultaneously from a single network. Details of the components are given in Appendix A.

Most of the building blocks are known techniques. Lifting multi-camera image features into BEV space using per-pixel depth distributions follows the lineage that began with Lift, Splat, Shoot [3]; depth supervision from LiDAR [4] and temporal fusion that aligns past BEV features by ego motion [5][6] have both been shown effective in prior work. 3D object detection uses a center-based approach that estimates object centers from a heatmap on the BEV grid [7]. Handling perception and ego-trajectory generation in a single network places the model in the line of recent end-to-end driving research [8][9][10]. What characterizes METEOR is that these are combined using operators that run easily on an in-vehicle SoC, and that the model can be updated continuously on top of the autolabeling and agentic-AI development loop.

Fig. 3. METEOR architecture: surround cameras → depth-aware BEV transform → temporal fusion → simultaneous multi-task output. A rule-based safety check (guardrail) is attached to the trajectory output

Fig. 4 shows inference on public roads not used in training. The top row shows the surround cameras with 2D perception, the middle row camera-only distance estimation, the bottom left a volumetric occupancy grid, and the right a bird's-eye view (lane lines, crosswalks, vehicles; the green line is the E2E driving plan). This frame illustrates the simultaneous outputs of a single network: perception, measurement, analysis and planning.

Fig. 4. Inference on validation data (public roads not used in training): top, surround cameras with 2D perception; middle, camera-only distance estimation; bottom left, volumetric occupancy grid; right, bird's-eye view (green line = E2E driving plan)

In-vehicle implementation: INT8 optimization and deployment by the agentic AI

In-vehicle implementation is also part of the agentic-AI development. The trained model is exported to ONNX, an INT8 quantized engine [11][12] is built with calibration on real driving frames, and implementation optimizations such as a mixed-precision design based on per-layer quantization sensitivity (layers with large impact stay in FP16) [12][13] and a dedicated CUDA plugin for the BEV transform are applied repeatedly by the agentic AI, guided by latency profiles measured on the hardware. Every engine build is automatically checked for latency, output sanity and the accuracy difference before and after quantization, and only builds that meet the criteria are deployed to the in-vehicle computer. If a build fails, the previous generation is restored automatically.

Training data and labels: Co-MLOps data and CoMET

Data-centric development and nationwide data

The performance of physical AI depends on the quality and quantity of data just as much as model design, if not more. We bear this in mind in our data-centric development approach. As a precondition, this requires not only data volume but also quality, diversity and accessibility. Likewise, when accuracy stalls during the improvement cycle of the reference E2E AI model, we look at the data first, identifying weak scenes and addressing the data and labels for those conditions.

Training used nationwide Japanese driving data collected through Co-MLOps. The diversity of regions and conditions provides a level of generalization that is difficult to achieve with data from a single area. These conditions include urban intersections, suburban residential streets, winding mountain roads, nighttime and rain. During development, we observed that adding data from a new region improved perception of that region's characteristic road structures and objects. We believe this data diversity directly expands the range of driving environments the model can handle.

Nationwide collection is carried out by a fleet equipped with the Co-MLOps Data Recording System (DRS) (Fig. 5). Four 120° LiDARs covering front, rear, left and right and eight multi-FoV cameras are recorded in a synchronized, calibration-managed state. The eight-camera input of the reference E2E AI model matches this DRS camera configuration. LiDAR is used only for CoMET label generation (3D bounding boxes and depth); inference uses cameras only. Participants can therefore use DRS data directly for fine-tuning, and with the same vehicle configuration, go straight through to in-vehicle evaluation.

Fig. 5. The Co-MLOps Data Recording System (DRS)

Deriving training labels from general-purpose labels

All training labels for the reference E2E AI model are derived from CoMET autolabeling. CoMET is an ensemble teacher model combining twelve large models, and it generates labels on the scale of millions at consistent quality without the throughput limits of human labor. Its construction and robustness are described in the tech blog mentioned above.

What we want to highlight here is that CoMET's labels are general-purpose rather than task-specific. By keeping them in reusable representations, namely 3D bounding boxes, panoptic segmentation and traffic-light recognition, METEOR's training derives all of the following training labels from the same label asset.

• BEV road-surface segmentation (lane lines, stop lines, crosswalks, etc.): panoptic segmentation associated with the accumulated LiDAR point cloud and transformed into BEV space
• 3D object detection (heading and velocity): 3D bounding boxes with velocity added by temporal association
• Per-pixel depth (all eight cameras): LiDAR projection
• Occupancy grid and risk map: composed from the accumulated point cloud and 3D bounding boxes
• Future trajectory prediction: temporal tracking of 3D bounding boxes
• E2E ego trajectory: the ego driving log and the drivable area above
• Traffic-light state (green/yellow/red, arrows): traffic-light recognition (TLR) labels
• 2D segmentation and 2D detection: image-plane projection of panoptic segmentation and 3D bounding boxes

Many of the labels for understanding the driving environment, such as BEV road-surface segmentation and 2D segmentation, are derived from panoptic segmentation, and the ability of the HD-map-free reference E2E AI model to read road structure rests on this general-purpose label. Adding a new task requires no re-annotation, only one derivation step from the existing asset. Generate labels once as a representation rather than once per task: that is the essence of autolabeling-centered development (Fig. 6). In this project, ingesting data from a new region, from placing the data to starting training, is done with no human involvement.

Fig. 6. From one label asset to every training label: CoMET generates general-purpose label assets (3D bounding boxes / panoptic segmentation / traffic-light recognition) from camera, LiDAR and GNSS data, and every task's training labels are derived from them

Label quality control

The challenge with autolabels is noise. CoMET and METEOR suppress it with the following mechanisms (Fig. 7).

• Agreement check between two label sources: Two independently generated label sources are compared, only pixels on which both agree are kept as labels, and disagreeing pixels are excluded from training. This removes low-confidence regions mechanically.
• Consistency check: Geometric consistency between independently produced labels, such as depth labels against 3D bounding boxes, is measured quantitatively to detect systematic offsets.
• Handling of missing labels: "No object present" and "no label present" are distinguished, and unlabeled regions are excluded from the loss, so that the model does not learn missing labels as "no object."

Fig. 7. Label quality control: Only pixels on which two independently built label sources agree become labels, and the consistency check and the handling of missing labels keep noise out of training.

Data augmentation with NVIDIA Cosmos

For the biases of real driving data (scarce rare cases, difficulty of collecting in bad weather), we are using generated data from NVIDIA Cosmos: condition transfer with Cosmos Transfer (converting to rain, night or snow while keeping CoMET labels), rare-case generation with Cosmos Predict, and data-gap discovery with Cosmos Reason. In training the reference E2E AI model, bad-weather and night data converted with Cosmos Transfer are mixed with real data, and we are evaluating the effect on perception performance under adverse conditions. Here too the generated data are labeled by CoMET, so the autolabeling platform is the common foundation for real and generated data.

Development by agentic AI: Running train–evaluate–improve loop autonomously

The core concept of METEOR is an autonomously evolving perception engine. The aim is to run the whole process, from driving-data collection (DRS) through CoMET labeling, training, improvement and in-vehicle implementation, without human intervention (Fig. 8): development that does not presuppose human labeling or human code.

Fig. 8. The METEOR concept: automating the process from collection → labels (CoMET) → training → improvement → in-vehicle implementation

In developing the reference E2E AI model, the agentic AI executes not only training but the development loop itself: evaluation, failure analysis, implementing improvements and retraining. No additional human labeling was done, and the agentic AI also wrote the model and training code. Improvements are made under pre-registered acceptance criteria (for example, automatic rollback if accuracy degrades), and training, evaluation and hardware verification across several machines run continuously. We view this as a practical example of AI developing AI.

In this development approach, the human role centers on directing what to build and reviewing the results. Given a concept or feature request such as "I want a feature that can do X", the agentic AI carries out model development, ingestion of the autolabeled driving data, data conversion, cleaning, implementation optimization (quantization and edge speed-up) and model operations (deployment and auto-recovery), and keeps running as a self-improvement loop (Fig. 9). It is an attempt to replace the constraint of development resources with the constraint of data and compute.

Fig. 9. How development is done: the human directs, the AI develops. From model development through data processing, implementation optimization and operations, agentic AI executes the work

The agentic AI works under the following principles (Fig. 10):

• One-variable principle: Only one thing changes per training round, and whether it helped is judged only by comparison on the same data under the same conditions.
• Pre-registration and acceptance: Each improvement is submitted with its hypothesis, applicability conditions and pass criteria registered in advance. Measures that fail the criteria are withdrawn automatically and the record is kept in the ledger.
• Disciplined paired comparison: Detectors are compared only on the same objects both detected, and similar procedures remove contamination of the measurement itself (recall differences, threshold differences).
• Failure mining: Scenes the latest model handles poorly are extracted automatically every round and sampled more heavily in the next training.
• Root-cause analysis: Examples include a bug in which the rotation direction of data augmentation was inverted only for the label raster, found and fixed by numerically cross-checking the raster against the point cloud; and a phenomenon in which only the ego trajectory drifted sideways after quantization, isolated layer by layer and by cause and fixed permanently through a design change on the training-recipe side. The agentic AI carries such work through from analysis to fix and verification.
• Revisiting the metrics: Because single-frame trajectory error does not adequately capture behavior on the road [10][14], we also use metrics closer to closed loop, such as accumulated error when following the model's own plan, recovery from lateral offsets, and the intervention rate of the safety check, and include them in acceptance decisions.

Fig. 10. The agentic-AI development loop: one variable per change, pass criteria registered in advance. Measures that do not help are withdrawn automatically

In-vehicle implementation runs inside the same loop. The standard path from ONNX to TensorRT INT8 is used, with real driving frames for calibration. Optimization is driven by latency profiles on the hardware and by quantization sensitivity analysis [12]: layers with large accuracy impact stay in FP16 and the rest are quantized to INT8. In post-training quantization, the quantization scales are determined from the activation distributions of real driving frames, following the standard procedure for integer-arithmetic inference [11][13]. At every engine build, latency, output sanity and the accuracy difference against FP16 execution are measured automatically, and each optimization is accepted or rejected on the basis of a ledger recording "accuracy contribution per millisecond." Items whose effect does not reproduce are recorded as rejected. See Appendix A for details.

OSS release and what it means for Co-MLOps members

The reference E2E AI model is available as open-source software. This includes the source code, trained model and training recipe (the procedure from data conversion and label derivation through training to in-vehicle implementation). Releasing it as OSS serves two main purposes. First, it presents a camera-only, HD-map-free E2E autonomous driving AI model in a format that can be reproduced, verified and modified. Second, it demonstrates the Co-MLOps development approach (autolabeling plus agentic-AI development) through a working model and code.

Companies participating in the Co-MLOps project can start from this released version and:

• Keep improving the model
  The reference E2E AI model continues to be updated with the data shared on Co-MLOps. Members receive the latest model and use it as the starting point for evaluation, fine-tuning and distillation on their own data.
  
  
• Add their own features
  On top of the released recipe, members can add functionality matched to their own vehicle configuration, regions and tasks. With the same eight-camera configuration as the DRS, collected data can be used directly for fine-tuning, straight through to in-vehicle evaluation.
  
  
• Share the development process
  The way of working itself, label generation by autolabeling and the agentic-AI development loop, is shared with members.

The benefit of using the reference E2E AI model is that it reduces the initial setup burden compared to starting E2E development from scratch.

Wrap-up: From driving data to an E2E autonomous driving AI model

This article introduced our reference E2E AI model, a camera-only, HD-map-free E2E autonomous driving AI model built by combining nationwide Japanese driving data collected through Co-MLOps, CoMET autolabeling, and an agentic-AI development loop.

• Data and labels: Nationwide data collected with the DRS, training labels derived from CoMET's general-purpose label assets, with noise suppressed by the two-source agreement check and related mechanisms.
• Model: A single network built around a depth-aware BEV transform that outputs everything from BEV perception to E2E trajectory generation at once and runs at a practical latency on NVIDIA Jetson Orin.
• Development: An agentic AI runs the train–evaluate–improve–deploy loop under the one-variable principle and pre-registered acceptance criteria.

The performance of E2E AI depends heavily on the volume and diversity of data. The reference E2E AI model is not a finished product but a model that keeps being updated as data grows. As data from new regions and conditions is added to Co-MLOps, the model provided to members improves. Through Co-MLOps and the reference E2E AI model, TIER IV aims to advance autonomous driving AI development together with the community. If you are interested, please explore the public repository and join the discussion. We also welcome inquiries about joining Co-MLOps or sharing data.

Appendix

Technical detail: Inside METEOR

Architecture: A single network built around a depth-aware BEV transform

METEOR is a single network built from efficient operators, designed from the outset for quantization and acceleration on an in-vehicle SoC. TensorRT compatibility is treated as a design constraint rather than a porting task after research: the operators are limited to a set that TensorRT executes efficiently, such as Conv, GridSample and Gather, and temporal memory is expressed not as a recurrent structure but as a static graph in which the previous frame's BEV features are passed as ordinary input and output tensors. As a result, the trained model can be converted to a TensorRT engine almost as-is via ONNX, and deployment to NVIDIA Jetson Orin needs no special porting work.

1. Image feature extraction: A backbone with shared weights (ResNet family + FPN) applied to the eight cameras.
2. Depth-aware BEV transform (lift): A distance probability distribution is estimated per pixel, and image features are projected onto the BEV grid weighted by that distribution. The core mechanism that builds the BEV space from what is visible, without a map.
3. Temporal memory fusion: BEV features from 0.4, 1.2 and 2.8 s earlier are warped by the ego motion and overlaid, giving robustness to moving-object velocity and to temporary occlusion.
4. Multi-task heads: From the shared BEV features, road-surface segmentation (nine classes including lane lines, stop lines and crosswalks), 3D detection of vehicles and pedestrians (with heading and velocity), the occupancy grid, future trajectory prediction, traffic-light state, and 2D segmentation and 2D detection are output simultaneously.
5. E2E trajectory head: The ego trajectory is generated as three hypotheses (structurally corresponding to straight, left and right); if a driving command is given, that hypothesis is selected, otherwise a selector estimates it. A penalty based on a distance field from the road surface keeps the trajectory within the drivable area from training time. Perturbation training that shifts the ego position laterally and uses a lane-returning trajectory as the target [14] is also applied.
6. Rule-based safety check (guardrail): Independently of the network output, the generated trajectory is checked against the occupancy grid and risk map, and an intervention decision is returned if approach to an obstacle or departure from the road is detected.

METEOR additionally has a PointPainting-style [15] feature fusion that adds 2D segmentation probabilities to the input features of the BEV transform. A distant pedestrian tends to collapse into a single cell and vanish on the BEV grid, while in the image it remains as a tall, thin shape. Carrying this 2D perception result through the lift into the BEV cells has been confirmed to improve the detection recall of distant pedestrians. The injection is done through a zero-initialized 1×1 projection, and it was verified numerically that the model output is unchanged at the moment of insertion (function-preserving) before training (Fig. 11).

Fig. 11. PointPainting: The probabilities of 2D segmentation are injected before the BEV transform through a zero-initialized 1×1 projection. BEV width, the lift plugin and the distribution layout are unchanged.

How the labels are made

• BEV road-surface labels: The results of 2D segmentation are associated with the LiDAR point cloud accumulated over the whole drive, projected into BEV space and integrated. Only pixels on which two independently generated label sources agree are kept as labels (disagreeing pixels are excluded from training), so that label noise is not transferred into the model.
• 3D bounding boxes and depth labels: LiDAR-derived 3D bounding boxes and per-pixel depth from LiDAR projection. Depth labels, including those from telephoto cameras, are checked for consistency against the 3D bounding boxes.
• Enforced hold-out: Evaluation-only data is rejected at the code level so that it cannot slip into training by mistake.

Edge optimization

The projection in the BEV transform is implemented as a dedicated CUDA plugin to secure compute efficiency. In the current configuration, the degradation of BEV segmentation from INT8 quantization stays within the noise band (the accuracy difference before and after quantization is measured following the evaluation procedure of [12]), and latency stays at a practical level on a realistic in-vehicle SoC. We do not regard this as sufficient, however, and plan to keep exploring improvements on both the model-structure and implementation sides. Members deploying to their own Jetson Orin-equipped vehicles can use the same path as-is.

References

[1] Q. Li et al., “HDMapNet: An Online HD Map Construction and Evaluation Framework,” ICRA 2022.
[2] B. Liao et al., “MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction,” ICLR 2023.
[3] J. Philion, S. Fidler, “Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D,” ECCV 2020.
[4] Y. Li et al., “BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection,” AAAI 2023.
[5] Z. Li et al., “BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers,” ECCV 2022.
[6] A. Hu et al., “FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras,” ICCV 2021.
[7] T. Yin, X. Zhou, P. Krähenbühl, “Center-based 3D Object Detection and Tracking,” CVPR 2021.
[8] Y. Hu et al., “Planning-oriented Autonomous Driving,” CVPR 2023.
[9] B. Jiang et al., “VAD: Vectorized Scene Representation for Efficient Autonomous Driving,” ICCV 2023.
[10] L. Chen et al., “End-to-end Autonomous Driving: Challenges and Frontiers,” IEEE TPAMI, 2024.
[11] B. Jacob et al., “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” CVPR 2018.
[12] H. Wu et al., “Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation,” arXiv:2004.09602, 2020.
[13] M. Nagel et al., “A White Paper on Neural Network Quantization,” arXiv:2106.08295, 2021.
[14] M. Bansal, A. Krizhevsky, A. Ogale, “ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst,” RSS 2019.
[15] S. Vora et al., “PointPainting: Sequential Fusion for 3D Object Detection,” CVPR 2020.

Dan Umeda | Data Scaling Department

Dan joined TIER IV in 2023 and currently serves as a principal AI engineer and team lead, driving the development of Co-MLOps.

TIER IV is always on the lookout for passionate individuals to join our journey. If you share our vision of making autonomous driving accessible to all, get in touch.

We’re currently hiring for the following related positions:

• Edge AI engineer (autonomous driving)
• MLOps engineer (Co-MLOps)
• Senior machine learning engineer (E2E autonomous driving model)

Visit our careers page to view all job openings.

If you’re unsure which roles fit your experience, or if the current job openings don’t quite match your preferences, you can register your interest here. We’ll contact you when a suitable role becomes available and arrange an informal interview.

Inquiries

• Recruitment: recruit@tier4.jp
• Media: pr@tier4.jp
• Business: sales@tier4.jp

Social Media

X (Japan/Global) | LinkedIn | Facebook | Instagram | YouTube

More

• TIER IV
• Autoware
• Autoware Foundation
• TIER IV Amazon Store