To build reliable autonomous driving systems that can handle real-world conditions, developers need access to a comprehensive range of traffic scenarios. However, finding specific scenarios within petabytes of unstructured video can be a major bottleneck in development. Mobility Observer (MOB) is TIER IV’s in-house observability platform, providing clear visibility into our autonomous driving systems through search and visualization of driving data. To accelerate data extraction, we have now added natural language scene search to MOB, powered by NVIDIA Cosmos Embed1. This post sets out why the feature was needed, then covers the design decisions and the implementation behind it: how we chose the search approach, how we integrated it into the existing search platform, and how we optimized inference cost.
Driving data accumulates every day as we develop and operate autonomous vehicles, and the total volume stored on TIER IV's platform is on the order of petabytes. In its raw form – rosbag, the log format used by Robot Operating System 2 (ROS 2) – this data is hard to work with. We therefore run it through preprocessing pipelines tailored to specific use cases. Preprocessing can involve extracting video frames as images or assigning the scene tags described later. The camera video extracted from TIER IV's driving data alone exceeds 9,000 hours.
Volume is not the only challenge. Even among logs recorded in the same urban area, factors like weather, time of day, traffic, and the behavior of surrounding vehicles and pedestrians all vary. Developing an autonomous driving system requires pulling one specific situation out of that variety, over and over. This is not only for perception: path planning and end-to-end models need it too, because a model that holds up across a wide range of driving scenes has to be trained and evaluated on them. A recent TIER IV tech blog describes one example of this kind of model development: an end-to-end model that runs on cameras alone, without a high-definition map, that we released as open source.
The scenes that matter most are the infrequent ones in the long tail, such as lane closures for road construction or debris on the road. No matter how many ordinary scenes are available, it is difficult to produce a model that can handle long-tail situations. We must intentionally extract these rare scenes for both training and evaluation. How quickly we can access the scenes we want sets the pace of development.
MOB already had ways to search driving data, including search by metadata and by scene tags. Metadata covers attributes such as date and time, vehicle, location, and the firmware version that was deployed. When driving data is registered, a multimodal large language model that accepts camera images as input assigns scene tags, which support filtering by attributes such as weather and road geometry. Our presentation at AWS Summit Japan 2026 (slides in Japanese) covers the pipeline from collecting driving data to building training datasets, including search by scene tags.
We designed and built natural language scene search as a new addition to these search functions. The feature converts the search query directly into a vector and compares it with vectors computed from the video to find the closest scenes. This means you can describe a context that involves several elements at once without assembling strict query conditions, and that tolerance for ambiguity is what makes the feature effective for extracting long-tail scenes.
In practice, you type an English description of the scene you are looking for into the search box, and the search returns driving scenes that match it. The matching is based on the semantic content of the video rather than on keywords. For example, you can find scenes like these:
Interactions with traffic participants: “a pedestrian crossing the road in front of the car,” “waiting behind a bus at a bus stop,” “driving behind a truck at night”
Events on the road: “workers and traffic cones blocking the lane,” “passing a construction site”
Environmental conditions: “intersection on a rainy day,” “driving on a wet road in the rain”
The desired situation simply needs to be described in English.
Figure 1: search UI showing a natural language query and its results
Two aspects of the search experience are worth highlighting.
First, natural language search combines with the existing search criteria. MOB's driving data search provides faceted search on the metadata described above, and you can specify a natural language query at the same time. In other words, you can narrow the data down by conditions such as time period, vehicle, and area, and then look within that set for scenes where the vehicle passes a construction site.
Second, results come back at the segment level. The feature divides driving data into 5-second intervals, which we call segments, and searches at that granularity. Rather than returning whole driving data files, it identifies which part of which file matched. So there is no need to review a long recording from the beginning. You can play the matching scene immediately.
Figure 2: playback of a matching scene from the search results
Whether you are collecting data to annotate for machine learning, pinpointing a situation during failure analysis, or looking for material for a test scenario, you can now find the scene you want quickly.
From here on, we look at how we designed the feature. When determining how to search video using natural language, we considered two main approaches.
The first is to generate captions or tags from the video in advance with a vision-language model and then search that text. It is strong at strict filtering with structured tags, and the generated captions can also serve other purposes such as annotation. The second is to vectorize the video directly with a video embedding model and search for nearest neighbors in the same vector space as the query text. Because it does not pass through text as an intermediate representation, inference at ingestion is lighter, and there is no need to fix the format of search conditions in advance.
MOB had two requirements: to make every registered driving data file searchable immediately after registration at minimal additional cost, and to allow users to search for complex, multi-element scenarios using natural language without fixed formatting for search queries. In other words, English sentences like the ones listed earlier should work directly as search conditions. These requirements led us to the second approach: searching embeddings directly.
Metadata and scene tags still handle strict filtering. Natural language search does not replace them; we designed it to coexist with them as a third axis. That division of roles lets us take full advantage of what direct embedding search offers.
Figure 3: two approaches – search through captions and tags, and direct embedding search
The model we adopted is NVIDIA's Cosmos Embed1. It is part of Cosmos, a family of models for physical AI, and has a dual-encoder architecture with one encoder for video and one for text. It was trained on approximately 8 million video-caption pairs covering domains that include robotics and autonomous driving. Contrastive learning aligns the outputs of both encoders in a single 768-dimensional space, so cross-modal search – finding video with text – reduces to a k-nearest neighbor (k-NN) search over vectors. The output vectors are normalized, so proximity corresponds directly to semantic similarity.
The Cosmos Embed1 video encoder expects 8 frames per clip as input. The model's suitability for the driving video domain, together with inference light enough to run in under 0.2 seconds per segment, as described shortly, settled the choice.
The feature consists of two data flows: the query side at search time, and the index side when driving data is ingested.
Figure 4: Architecture overview
The Cosmos Embed1 text encoder, running on AWS Lambda, converts the search text into a 768-dimensional vector. Text encoding is light enough that a CPU-based Lambda completes it in about 100 ms on a warm start. We configure provisioned concurrency so that cold starts never affect search response time.
We then run a k-NN search with that vector against an Amazon OpenSearch Serverless vector search collection. We apply facet conditions such as period and vehicle as filters on the vector search as well, so it looks for neighbors only within the narrowed set. We aggregate the resulting segments per driving data file and combine them with the results of the existing faceted search to assemble the final result.
Figure 5: query flow – text to vector, filtered k-NN, aggregation
We left the existing search collection untouched and created a dedicated vector search collection. This adds a new search axis while keeping the existing search feature stable.
MOB already had a pipeline that extracts video frames as images when driving data is registered. The new indexing pipeline for natural language search reuses those images, so it never decodes the video a second time just to generate embeddings. It divides the extracted frames into 5-second segments, and a graphics processing unit (GPU) worker generates a video embedding from the 8 frames of each segment.
We ingest the generated embeddings through Amazon SQS. Here we chose a design that tolerates duplicates rather than one that strictly prevents them. Message delivery is at-least-once, and an idempotent upsert keyed on the pair of driving data file and segment number absorbs the duplicates. A periodic cleanup removes duplicate documents that reach the search index. Taking the existence check out of the ingestion path keeps the implementation simple and throughput high.
The source of truth is Amazon DynamoDB. We pack the 768-dimensional float32 vector into binary and store it as a 3,072-byte item. Compared with storing it as a list of numbers, this reduced read cost by about 30% and read latency by about 90%. The search index is derived data that we can rebuild from DynamoDB at any time, and neither rebuilding the index nor recovering from a failure requires GPU inference again.
We store an embedding per segment to pinpoint where a scene occurs in the data, not just which file contains it. A single driving data file can include very different scenes, such as straight roads and intersections, or moments with pedestrians crossing and moments without. A granularity of 5 seconds is precise enough to capture these variations.
Figure 6: segmentation and frame sampling
Embedding models keep improving from one generation to the next. When we update the model, we plan to re-encode all segments into a new index and switch the reference on the search side. To keep old and new vectors from being searched together during the transition, every document carries a model version and every search filters on it, a defense we built in from the start.
The system runs on managed services – Lambda, SQS, DynamoDB, and OpenSearch Serverless – with only the GPU batch processing on Amazon EC2 through AWS Batch. This keeps operational effort low while scaling with changes in ingestion volume.
Generating video embeddings consumes the most compute on the index side. To handle an initial backfill of more than 9,000 hours and the new data arriving every day at a realistic cost, we optimized the inference based on performance metrics.
We measured inference time on three AWS GPU instance types.
|
Instance |
GPU |
Compute performance (BF16) |
Memory bandwidth |
Inference time per segment |
|
g6.xlarge |
NVIDIA L4 |
121 TFLOPS |
300 GBps |
1,190 ms |
|
g5.xlarge |
NVIDIA A10G |
70 TFLOPS |
600 GBps |
783 ms |
|
g6e.xlarge |
NVIDIA L40S |
362 TFLOPS |
864 GBps |
354 ms |
Note: Compute performance is the published peak bfloat16 (BF16) Tensor Core throughput of each GPU in teraflops (TFLOPS), without sparsity.
The compute performance of the GPU did not explain inference time. It was close to inversely proportional to memory bandwidth. The L4 and the A10G are the telling pair: the L4 has about 1.7 times the compute performance of the A10G, yet the A10G runs inference 1.5 times faster. This points to a memory-bandwidth-bound workload: the GPU spends most of its time moving data to and from memory rather than computing. Although g6e.xlarge carries the highest hourly price of the three, the difference in speed more than compensates, and its cost per inference is the lowest.
Figure 7: memory bandwidth, compute performance, and inference time
We also evaluated batching, the standard technique for raising GPU throughput by running several inputs together, and decided against it. At batch=2, throughput degrades by 4-9%. Memory capacity is not the constraint, since peak usage is about 8 GB against the 48 GB available. Throughput does not improve because memory bandwidth is already saturated: each additional input brings its own intermediate data to move, so processing two at once moves twice as much data, and the spare compute capacity stays idle. Batching therefore does not buy throughput for our video embedding workload.
By contrast, torch.compile paid off. With mode="reduce-overhead", inference time dropped from 354 ms to 189 ms, a reduction of 46.7%. Kernel fusion cuts memory traffic, which is the bottleneck itself, so it fits this workload particularly well. The input shape is completely fixed at 1x8x3x448x448, so no recompilation occurs. Compilation costs about 50 seconds at startup, which the worker design described next amortizes.
GPU inference itself takes just under 0.2 seconds per segment. Initializing a GPU environment from scratch, however, takes 6-7 minutes before the first inference, and the breakdown was not what we expected. Fetching and loading the model – downloading an artifact of about 1.8 GB from Amazon S3, extracting it, and loading it onto the GPU – accounts for only about 30 seconds. What dominates is acquiring the GPU instance and pulling the container image, which takes 4.5-6 minutes. About 50 seconds of torch.compile follows.
Figure 8: cold start breakdown – loading the model is not what dominates
These measurements led to two design decisions.
The first is to fetch the model from S3 as an artifact at startup instead of baking it into the container image. The lighter image shortens the pull, and cold start as a whole is about 30% faster. Swapping the model then means replacing a file on S3, with no image rebuild and no redistribution.
The second is to separate the scaler, which manages the work to be processed, from the worker, which runs the inference. The scaler is a lightweight Lambda that runs every 5 minutes and starts as many GPU workers on AWS Batch as the queued work requires. Workers are long-lived containers: each one loads and compiles the model once at startup, then pulls work from the queue and generates embeddings until the queue is empty, at which point it exits. The 6- to 7-minute initialization is amortized across the many driving data files processed in one run, and GPU time goes into inference itself.
The throughput calculation: one minute of driving data splits into 12 segments, which at 189 ms per segment takes about 2.3 seconds. Effective throughput per worker is about 1,500 files per hour, so a single worker keeps up with new data even at peak hours. Even the initial backfill of more than 9,000 hours, or several million segments, fits within a few days on a handful of GPUs running in parallel.
In short, two kinds of measures paid off. The first made inference itself faster: selecting the instance based on memory bandwidth cut inference time from 1,190 ms to 354 ms, and torch.compile cut it further to 189 ms. The second shortened the time before inference starts: delivering the model as an S3 artifact made cold starts about 30% faster, and long-lived workers reduced the initialization overhead per unit of data processed. Batching, the standard technique for GPU inference, did not help in our case, so we do not use it.
This post detailed our approach to bringing natural language scene search to MOB using NVIDIA Cosmos Embed1. Because we vectorize video directly instead of going through captions, every registered driving data file is searchable in free-form language at minimal additional cost. Three things support this: an index design that runs a second collection alongside the existing search, DynamoDB as a source of truth the index can always be rebuilt from, and GPU inference optimized based on performance metrics. We cut inference itself to 189 ms per segment by selecting the instance based on memory bandwidth and applying torch.compile. By delivering the model as an S3 artifact and keeping workers long-lived, we keep the overhead before inference starts small relative to the volume of data processed. As a result, we have reached the point where we can process the initial backfill of more than 9,000 hours within a few days at a realistic cost, and a single worker keeps up with each day's new data, even at peak hours.
Embedding models will continue to advance with each new iteration. This feature was designed to support model swapping, ensuring future advancements translate directly into better search quality. As we expand the data we cover and connect the feature more tightly to metadata, MOB will keep evolving as a data platform for autonomous driving development.
Yusuke Hino | ML & Data Infrastructure Department
Yusuke joined TIER IV in 2021, working on a cloud data platform that manages driving data recorded by autonomous vehicles. He currently oversees the development and operation of Mobility Observer, an in-house observability platform for autonomous driving systems. He led the design and development of the natural language search feature introduced in this post.
TIER IV is always on the lookout for passionate individuals to join our journey. If you share our vision of making autonomous driving accessible to all, get in touch.
We’re currently hiring for the following related positions:
Visit our careers page to view all job openings.
If you’re unsure which roles fit your experience, or if the current job openings don’t quite match your preferences, you can register your interest here. We’ll contact you when a suitable role becomes available and arrange an informal interview.
Inquiries
Media: pr@tier4.jp
Business: sales@tier4.jp
Social media
X (Japan/Global) | LinkedIn | Facebook | Instagram | YouTube
More