Building a dataset foundation for autonomous driving with NVIDIA Cosmos

The performance of autonomous driving AI is no longer determined by model architecture alone, but increasingly by how much diverse, high-quality data we can supply. This article describes how TIER IV integrated NVIDIA's world foundation model (WFM) platform, NVIDIA Cosmos, into the Co-MLOps data-sharing and machine learning operations platform to automate data search, edge-case generation, and data augmentation.

Why a data foundation is key to autonomous driving

The single biggest obstacle to safely deploying autonomous driving systems in society is the so-called long-tail problem. The vast majority of real-world driving data consists of ordinary, uneventful driving, and rarely contains the rare scenes – edge cases – that are directly tied to collision risk. Yet it is precisely these undiscovered edge cases that most strongly affect a system's safety, functional goals, and deployment success. Physical on-road testing is slow and expensive, and cannot cover the near-infinite possibilities of the real world.

This is why TIER IV leverages generative AI that simulates the world – WFMs – to fill data gaps through three approaches: search, create, and augment. NVIDIA Cosmos, the focus of this article, is a family of models for physical AI designed for exactly this purpose.

Co-MLOps: A platform for sharing data and AI

TIER IV is a deep-tech company that leads the development of Autoware, open-source software for autonomous driving. To date, the company has conducted autonomous driving demonstrations in more than 39 prefectures and 127 locations across Japan.

In 2024, TIER IV launched Co-MLOps, a collaborative framework that unifies data sharing and machine learning operations (MLOps) to develop ADAS/AD systems at scale. It has been designed to operate as a virtuous cycle (active learning loop): as more organizations participate, the shared data becomes richer and the reference AI continuously improves on large-scale data.

The active learning loop at the heart of Co-MLOps

At the center of Co-MLOps is an active learning loop that keeps data continuously flowing into training and improvement, rather than treating collection as the end goal. Starting from data gathered across diverse vehicle types, the following stages are cycled through and fed back into the next round of improvement:

Autolabeling: Large-scale automatic annotation
Data tagging: Tagging scene attributes to make data searchable
Data gap report: Identifying which scenes are missing
Data generation: Filling missing data with generative AI
Quality validation: Verifying the quality of labels and data
Similarity ranking: Organizing duplicate/similar data to preserve diversity
Uncertainty estimation: Extracting high-value data the model struggles with
Privacy anonymization: Anonymization processing for sharing

By linking these stages into a single loop, an improvement cycle runs autonomously: find missing data → generate/collect it → label it and assure quality → feed it into training. NVIDIA Cosmos plays the role of powerfully boosting the data search, generation, and augmentation stages of this loop in particular.

The complete loop is shown in Figure 1. Starting from data gathered across diverse vehicle types, it cycles through eight stages – autolabeling, data tagging, data gap report, data generation, quality validation, similarity ranking, uncertainty estimation, and privacy anonymization – to continuously improve data and AI.

Figure 1. The Co-MLOps Active Learning Loop: an eight-stage cycle to continuously improve data and AI

Collection infrastructure: Co-MLOps Data Recording System (DRS)

The collection infrastructure (Figure 2) is a sensor configuration combining four 120° LiDARs covering front, rear, left, and right with eight multi-FoV cameras. It is designed to capture the vehicle's surroundings without blind spots.

Figure 2. Co-MLOps Data Recording System (DRS): a sensor configuration of 4× LiDAR and 8× cameras

Examples of the data actually collected are shown in Figure 3. They include diverse and rare scenes of high training value – urban roads at dusk and night, tunnels, construction zones, rain and snow, railroad crossings, and farm roads – collected on roads across Japan.

Figure 3. Examples of diverse and rare scenes collected on roads across Japan

CoMET: The foundation of autolabeling in Co-MLOps

Even when large volumes of data are collected, labels (ground truth) are needed to use them for training. Manual annotation suffers from a triple constraint: throughput limits, linearly increasing cost, and a quality assurance bottleneck. To overcome these, Co-MLOps provides a high-performance, general-purpose autolabeling foundation.

At its core is the Collaborative Multi-stage Ensemble-based Teacher Model (CoMET), an ensemble-based teacher model combining 12 large-scale models. Taking both camera and LiDAR as input, it automatically generates the following advanced annotations:

The supported annotation types are shown in Figure 4. It illustrates how diverse sensor data – from 2D images to 3D point clouds, including panoptic segmentation, traffic-light recognition, and LiDAR 3D bounding boxes – can be processed efficiently within a unified, streamlined interface.

Figure 4. Autolabeling foundation support: multi-dimensional annotation (panoptic/traffic-light/LiDAR 3D BBox)

The configuration of the autolabeling foundation is shown in Figure 5. This is the processing flow of CoMET, which takes camera and LiDAR as input and integrates 12 large-scale models as a multi-stage ensemble to generate panoptic segmentation, traffic-light recognition, 3D object detection, and more in one pass.

Figure 5. CoMET: an ensemble-based autolabeling foundation integrating 12 large-scale models

As a result, the system can generate millions of labels beyond the limits of human throughput, processing everything – including automated judgment of label quality and confidence – within a single consistent pipeline.

Key technology for robust and efficient autolabeling

For CoMET to work in production, it must combine robustness across diverse sensors, vehicles, and data with the efficiency to support large-scale processing.

On the input side, we deliberately use a diverse, mixed dataset to improve generalization accuracy. Specifically, we train on data spanning different vehicle types – passenger cars, taxis, small buses, and large buses – together with narrow/wide/fisheye cameras and LiDAR of different mechanisms, from rotational to MEMS. This is to keep the system stable even as the vehicle, camera, and LiDAR configurations change; in addition, we also leverage the edge-case data created by the generative AI described later.

At the core of the training side is a mixture/ensemble of task-specific expert models – individual models each trained for a specific task such as object detection, various segmentation, traffic-light detection, and various classifications (an ensemble-based design, as the CoMET name implies). These run in a two-stage pipeline – a high-resolution first stage and a low-resolution second stage – optimizing accuracy per task. A major advantage is that each expert model can be trained independently and in parallel, making it easy to flexibly incorporate new changes: for example, to add a new task, you simply add one deep neural network that solves that subtask.

On the inference side, it achieves high throughput through batched optimization, TensorRT optimization, parallel execution, and automatic CPU/GPU affinity. Notably, despite combining many models, it supports operation across a wide range of hardware – from consumer-grade GPUs to high-end GPUs.

This technology stack is shown in Figure 6. Taking a diverse, mixed dataset as input, it ensures robustness through two-stage training built on a mixture/ensemble of task-specific expert models, while boosting efficiency through inference optimizations such as TensorRT optimization and parallel execution.

Figure 6. Key technology for robust and efficient autolabeling: a mixture/ensemble of task-specific experts and a two-stage pipeline

Co-MLOps autolabeling in action

To see this output in practice, the following demonstration showcases 3D object detection and panoptic segmentation in a complex urban environment. Across the front camera, surround cameras, and LiDAR point clouds, elements like vehicles, pedestrians, and road structures are consistently labeled with high precision. This autolabeling foundation plays a decisive role as the receiver for the Cosmos integration described below.

NVIDIA Cosmos: World foundation models for physical AI

NVIDIA Cosmos is a platform of world foundation models designed to simulate and reason about the physical world. Cosmos Predict 2.5 and Cosmos Transfer 2.5 were released in October 2025, significantly advancing synthetic-data generation for autonomous driving and robotics. Cosmos comprises three model families with distinct roles.

Cosmos Reason: A reasoning vision language model (VLM) for physical AI, responsible for semantic video understanding and scene search
Cosmos Predict: Generates future world states from text/image/video, responsible for creating edge cases
Cosmos Transfer: Generates diverse driving videos from structured inputs (HD maps, depth, segmentation, etc.), responsible for augmenting data

TIER IV connected these three models directly into the Co-MLOps automated pipeline, realizing a continuous improvement-and-validation loop: search (Reason) → create (Predict) → augment (Transfer) → autolabel (CoMET).

This can be seen as evolving the search (Reason), creation (Predict), and augmentation (Transfer) stages of the active learning loop described above into a form fully powered by NVIDIA Cosmos.

The division of roles among the three models is shown in Figure 7. It illustrates how Cosmos Reason (search), Predict (create), and Transfer (augment) each feed edge cases, augmented data, and search results into the Co-MLOps automated pipeline, contributing to continuous model improvement and validation.

Figure 7. Integrating Cosmos Predict/Transfer/Reason into the Co-MLOps pipeline

Cosmos Reason: Reasoning-based data search

Finding scenes within petabyte-scale video archives is itself a major challenge. Taking driving video as input, Cosmos Reason produces a strict, structured JSON output covering scene description, time of day, weather, location (urban/suburban/expressway/mountainous), and ego-vehicle behavior (lane following/turning/crossing railroads, etc.).

The generated tags and captions are ingested into the Co-MLOps console, enabling natural-language text search, uncertainty-based search, and tag search compliant with the international standard ISO 34504. ISO 34504:2024 is an international standard for categorizing test scenarios for automated driving systems (SAE Level 3–5) using tags for dynamic entities, environmental conditions, and more, defining a harmonized approach to scenario classification.

In a demo, we showed that high-uncertainty scenes such as “pedestrians with umbrellas at an intersection” can be retrieved instantly via natural-language queries, providing immediate access to high-value, rare data for training.

The captioning and tagging process is shown in Figure 8. It depicts the flow in which driving video is fed into Cosmos Reason, which outputs scene description, time of day, weather, location, and ego-vehicle behavior as strict, structured JSON.

Figure 8. Captioning and tagging by Cosmos Reason (structured JSON output)

The search demo in the clip below works as follows. First, we enter “umbrella” in natural language; next, we narrow the uncertainty score to the 10%-0.1% range (restricting the results to high-uncertainty samples). We then set the ISO 34504-based tags to “pedestrian” and “intersection.” By combining these three conditions – language, uncertainty, and tags – we can pinpoint the targeted scene, demonstrating immediate access to high-value, rare data for training.

Cosmos Predict: Creating edge cases

When dangerous scenes don't exist in real-world data, or are simply too difficult to collect, synthetic generation is the only option. Cosmos Predict 2.5 generates high-fidelity, physically consistent synthetic data from text, image, or action prompts, filling data gaps.

Cosmos Predict 2.5 is a flow-based model that unifies Text2World/Image2World/Video2World into a single model, incorporates Cosmos Reason 1 as its text encoder, and was trained on 200 million high-quality clips. It can generate spatially and temporally consistent video up to 30 seconds long, and is offered in two sizes (2B and 14B).

Edge-case generation pipeline in Co-MLOps

The edge-case generation flow is shown in Figure 9. The clear goal here is to improve detection performance for low-lying objects. Using Predict2.5-2B and Gemini Flash Image (Nano Banana), this pipeline generates synthetic video from synthetic images via image2world. The generated data is immediately annotated (semi-annotation) by CoMET autolabeling and fed into the training pipeline.

Figure 9. Cosmos Predict edge-case generation pipeline (image2world)

Examples of generated synthetic data are shown in the video below. These are rare scenes difficult to collect in reality – animals such as boars and deer, a crouching person with a dog, people lying on the road, and debris such as tires and ladders – to which automatic annotation has been applied.

As a result of this generation and training, detection performance on real data actually improved. Specifically, we confirmed that detection of previously difficult cases – people lying on the road and small animals – improved even on real driving data. This shows that supplementing rare scenes with synthetic data directly enhances detection capability on real-world scenes.

The performance improvement is shown in the video below. On real data with autolabeling, detection of a person lying on the road (left) and of a small animal (right) improved, demonstrating that adding synthetic data via Cosmos Predict 2.5 contributed to better detection performance on real data.

Cosmos Transfer: Data augmentation for Sim2Real

Reliable AI must be trained across all possible conditions. However, physically collecting data for conditions like snowstorms is difficult, expensive and does not scale. Cosmos Transfer is a world-to-world transfer model that algorithmically closes these data gaps using a simulation-to-reality (Sim2Real) approach based on a simple concept: If you can't collect it, synthesize it.

It adds weather (rain, snow, fog) and time of day (night, dusk) to existing data in a physically realistic way, increasing data diversity. While being 3.5x smaller than its predecessor at 2B parameters, Cosmos Transfer 2.5 achieves up to a 60% performance gain on autonomous driving 3D lane and cuboid detection tasks.

Post-training to close domain gap

The challenge is that the pre-trained Transfer-multiview model was trained on HD-map data, whereas Co-MLOps has a different sensor configuration and camera FoV (a domain gap). TIER IV performed post-training using panoptic-segmentation masks generated by Co-MLOps autolabeling, bridging the domain shift between the HD-map and Co-MLOps datasets. This produced Co-MLOps Transfer2.5-2B-multiview, which can consistently generate multi-view images from multi-view masks.

The data-augmentation process is shown in Figure 10. Using camera/LiDAR and autolabeling results (3D bounding boxes, panoptic segmentation, traffic-light recognition) as control inputs, it is a flow in which Cosmos Transfer 2.5 generates video of diverse environments reflecting the caption.

Figure 10. Cosmos Transfer data-augmentation pipeline (linked with autolabeling)

The result of post-training is shown in Figure 11. It is an example of generating multi-view images consistent across viewpoints from input multi-view segmentation masks – the output of Co-MLOps Transfer2.5-2B-multiview after closing the domain gap. In other words, without any original RGB image, it can now generate data for any scene solely from the mask data produced by autolabeling.

Figure 11. Co-MLOps and Cosmos Transfer 2.5: image generation from multi-view masks

Moreover, this multi-view generation can freely transform the environmental conditions of the same scene into backlight, rain, snow, and nighttime. Because day/night and weather variations can be synthesized consistently across multiple viewpoints from a single driving scene, we can efficiently augment data with diverse environmental conditions that cannot be obtained simultaneously in reality.

A demo of multi-view generation is shown in the video below. It is an example of freely transforming a single driving scene into diverse environmental conditions – backlight, rain, snow, and nighttime – generated as imagery consistent across multiple viewpoints.

Quantitative evaluation: Verifying effectiveness of synthetic data

Finally, we quantitatively verified whether the synthetic data generated this way actually improves model performance. We constructed rare scenes in a simulator (Unity) and converted them into realistic video footage with Cosmos Transfer 2.5 and Cosmos Reason 2-B to generate synthetic data. We added this to the training data and compared how much the accuracy (IoU) of “Dog” detection improved.

The data-generation flow used for verification is shown in the video below. It is a pipeline that converts Unity simulator output into realistic video with Cosmos Transfer 2.5 and adds captions with Cosmos Reason 2-B to generate synthetic data.

Training Data	IoU (Dog)
Real data only (baseline)	0.671
Real + Cosmos synthetic data	0.893 (+0.222)

Against the baseline IoU of 0.671 (trained on real data only), adding Cosmos synthetic data improved it substantially to 0.893 (+0.222). This demonstrates that leveraging synthetic data genuinely improves detection performance, quantitatively confirming that supplementing hard-to-collect rare scenes with synthetic data directly boosts real-task performance.

Next steps: Migrating to and evolving with Cosmos 3

The work described so far is built on Cosmos Predict 2.5/Transfer 2.5/Reason 2. However, TIER IV is already advancing the migration to the next-generation Cosmos 3. The basic policy is to fine-tune (post-train) each function to the Japanese driving domain before using it. Here is a look at our current progress and future plans.

Post-training Cosmos 3 Nano for Japan-specific edge-case generation

We are currently post-training Cosmos 3 Nano and focusing on generating edge-case data suited to Japan-specific road environments. In the generation pipeline, Cosmos-Reason2-32B generates captions, Gemma4-31B converts them into prompts, and the fine-tuned Cosmos 3-FT-Nano generates video via three paths – text-to-video, video-to-video, and image-to-video. Gemini (image inpainter) is also used for image inpainting.

The Cosmos 3-based generation pipeline is shown in Figure 12. Cosmos-Reason2-32B generates captions, Gemma4-31B converts them into prompts, and the fine-tuned Cosmos 3-FT-Nano generates video via three paths, connecting to CoMET autolabeling.

Figure 12. Data-generation pipeline via Cosmos 3 (Caption → Prompt → Video generation → Autolabeling)

Implementing automatic caption conversion

Captions strongly influence the quality of generation and search. We are implementing a caption creator/prompt creator function in which a fine-tuned Gemma4-31B automatically converts captions output by Cosmos-Reason2-32B into a form suitable for generation. The goal is to mass-produce high-quality prompts without human intervention and make the generation pipeline autonomous.

Strengthening autolabeling and AutoQA/data cleansing AI (in development)

To use the edge-case data generated by Cosmos 3 directly for training, it must be possible to autolabel the generated data with high precision. We are therefore strengthening CoMET autolabeling to support both training and semi-annotation use. We are also developing an AutoQA/data cleansing AI that automatically checks and corrects autolabeling results – automatically correcting low-criticality items and routing high-criticality ones to human review, aiming to balance quality and efficiency. As both are still under active development, we omit the details in this article.

Figure 13. Demo: seven-camera surround multi-view generation (Text2World) with post-trained Cosmos 3

Extending Cosmos 3 to multi-view generation

Going beyond single-perspective generation, we are also extending post-trained Cosmos 3 to 7-camera surround multi-view video generation tailored to the camera configuration of our data collection vehicles. Figure 13 shows an example of a snow scene generated via Text2Video using the post-trained Cosmos 3 for multi-view. By equipping Cosmos 3 with knowledge of Japanese snow scenes – acquired through the Data Recording System (DRS) during the winter season – we can generate arbitrary snow scene videos directly from text prompts.

Figure 14. Demo: multi-view Cosmos Transfer driven by segmentation autolabels – consistent night/rain/snow transformations across all camera views

In parallel, we are also advancing multi-view Cosmos Transfer, which leverages the auto-labels from CoMET-generated Panoptic Segmentation. Figure 14 shows the video generated from segmentation masks using the post-trained Cosmos 3 Transfer. We plan to share the details of both initiatives in the future.

Future steps: Full migration to Cosmos 3

Going forward, we plan to fully migrate all functions – Predict (edge-case generation), Transfer (data augmentation), and Reason (reasoning/tagging) – to Cosmos 3. Each will be fine-tuned to the Japanese driving domain before operation, integrating the entire workflow of search, generation, augmentation, and quality assurance into a single, consistent Cosmos 3-based pipeline. The goal is a next-generation data foundation that runs the full cycle – from edge-case generation through autolabeling to AutoQA-based quality assurance – fully autonomously.

Conclusion: Search, create, augment end-to-end, and on to Cosmos 3

This initiative automated the autonomous driving data lifecycle (search, generation, augmentation, labeling) end-to-end by integrating the three NVIDIA Cosmos models with the Co-MLOps autolabeling foundation.

Cosmos Reason: A VLM attaches tags and captions to petabyte-scale video, providing instant access to rare data via ISO 34504-compliant natural-language search
Cosmos Transfer + Autolabeling: A World-to-World transfer realizes Day→Night and Clear→Rain/Fog, closing the Sim2Real gap and strengthening generalization and robustness
Cosmos Predict + Autolabeling: A predictive world model synthesizes long-tail dangerous scenes in a physically consistent manner and rapidly integrates them into the training pipeline

TIER IV has also begun migrating these functions to the next-generation Cosmos 3. We are performing full fine-tuning of Cosmos 3 Nano for Japan-specific edge-case generation, and have begun extending the fine-tuned Cosmos 3 to seven-camera surround multi-view generation, with multi-view Transfer running in parallel for augmentation. Alongside these, we are implementing automatic caption conversion, strengthening autolabeling for Cosmos 3-generated data, and developing an AutoQA/data cleansing AI. Ultimately, Predict, Transfer, and Reason will all be fully migrated to Cosmos 3, each fine-tuned to the Japanese driving domain before operation.

We are complementing what cannot be achieved through data collection alone with the generative AI power to search, create and augment. By integrating these capabilities with AutoQA into a single, consistent Cosmos 3-based pipeline, we aim to build a next-generation data foundation. This foundation will run everything from edge-case generation and autolabeling to quality assurance fully autonomously. The evolution of Cosmos and Co-MLOps is already underway.

This article reconstructs content from NVIDIA GTC 2026 session S81897, “Building a Dataset Foundation for Autonomous Driving with NVIDIA Cosmos,” presented by TIER IV Principal Engineer Dan Umeda.

TIER IV is always on the lookout for passionate individuals to join our journey. If you share our vision of making autonomous driving accessible to all, get in touch.

We’re currently hiring for the following related positions:

Visit our careers page to view all job openings.

If you’re unsure which roles fit your experience, or if the current job openings don’t quite match your preferences, you can register your interest here. We’ll contact you when a suitable role becomes available and arrange an informal interview.

Inquiries

Media: pr@tier4.jp
Business: sales@tier4.jp

Social media

X (Japan/Global) | LinkedIn | Facebook | Instagram | YouTube

More