In this Edge.Auto series of blog posts, we’ve been focusing on TIER IV’s automotive cameras and the technology behind them. The second installment covered the evolution and structure of CMOS image sensors. In the third installment, we’ll explore the functions of image sensors and examine the differences between global and rolling shutters.
Most modern CMOS image sensors output RAW images. As the name suggests, RAW images are unprocessed data. In the context of digital cameras, as explained in the previous installment, RAW images comprise pixel data containing information for specific colors, such as red, green, and blue, filtered through RGB color filters. RAW data output from the CMOS image sensor is processed by an image signal processor (ISP) before being displayed or used as input for computer vision systems.
Visualization of RAW image data
For devices geared toward automotive applications, there is demand for CMOS image sensors that integrate ISPs onto the same chip – as described in the first installment of this blog series. These sensors process RAW signals and output them as YUV (YCbCr) signals. This architecture is particularly valued in specific applications, such as rear-view cameras, because it enables a direct connection between the camera and the display, reducing system component and development costs. Sensors with built-in ISPs, which provide a complete set of camera functions on a single chip, are sometimes referred to as image sensor system-on-chips.
In TIER IV’s automotive cameras, the ISX021 in the C1 outputs YUV images, while the IMX490 in the C2 outputs RAW images. However, because the C2 incorporates an external ISP, its final output is YUV images.
Signal processing flow with functional blocks
The diagram illustrates an example of typical functional blocks and the flow of signal processing. Although various functions exist, this section focuses on introducing some of the more common functions. Signal processing can be divided into two main categories: processing in the RAW domain and processing in the RGB/YUV domain. Following is an overview of each functional block.
The pixel circuitry is central to the functionality of an image sensor. It comprises components such as microlenses, color filters, photodiodes, and multiple readout transistors. Microlenses focus incoming light, color filters select the wavelength (color), and photodiodes convert the light into an electric charge. Readout transistors then convert the charge into voltage, amplify it, and enable signal readout.
The voltage signal read from the pixels is converted into a digital value by an A/D converter. The resulting digital values are typically 10 bits (1024 levels) or 12 bits (4096 levels). Performing the analog-to-digital conversion early in the process helps minimize signal degradation caused by external noise and allows for faster signal transfer.
Image processing in camera systems covers a wide range of functions. One common example is the defect correction function.
A certain percentage of random defects occur during the pixel manufacturing process. The defect correction function corrects these defects through signal processing. There are two main types of defect correction:
In static correction, the locations of defects to be corrected are recorded in the camera system’s non-volatile memory, typically during the inspection process. This information is then used to correct the defects in the specified addresses. On the other hand, dynamic correction determines whether a pixel is defective by comparing its output with the output of surrounding pixels. This method dynamically identifies defects and applies corrections accordingly.
In static correction, defects are recorded during the inspection process, ensuring stable correction. However, the number of defects that can be corrected is limited by the capacity of the available non-volatile memory. On the other hand, the dynamic correction method has no limit on the number of defects it can correct, but there is a possibility of incorrect corrections or missed corrections, depending on the scene or pattern being captured.
High dynamic range (HDR) is crucial in automotive image sensors. HDR is achieved by capturing multiple signals from the pixels and combining them. Typically, three or four images are combined to generate a single HDR image. The composition algorithm varies by company, serving as a key differentiator.
Demosaicing, also known as debayering, is the process of adding full color information to each pixel. Before demosaicing, each pixel holds information for only one color (red, green, or blue). Demosaicing adds the missing color information to each pixel, resulting in a full-color image.
A typical demosaicing algorithm involves interpolating pixel values from the surrounding pixels, such as a 5x5 or 7x7 grid of pixels. Here is a very simple example. To interpolate the red and blue information for the central pixel (green), the following formula is used.
In actual products, algorithms vary by company, with techniques such as weighting coefficients to account for subject edges or refining the filter coefficients.
Basic camera controls are sometimes referred to collectively as the 3 A’s:
As automotive cameras are equipped with fixed-focus lenses, autofocus is not covered here: We’ll only be looking at auto white balance and autoexposure.
The human eye has the ability to perceive the true colors of objects under varying light conditions – such as incandescent, fluorescent, or natural light – which is referred to as color constancy. However, the colors output by a camera system are affected by differences in environmental lighting. To compensate for this, the white balance function adjusts the red, green, and blue channels by applying specific coefficients, ensuring objects that appear white to the human eye are also rendered as white. Most camera systems are equipped with an auto white balance function that automatically adjusts for different lighting conditions, but users can also manually adjust the white balance based on the ambient light.
The algorithm for auto white balance is quite complex, so we’ll only touch on the basics here. The fundamental concept behind it is the gray-world hypothesis. This approach is based on the empirical observation that when you sum up all the pixel values in an image, the result should be gray, meaning that the red-to-green and blue-to-green ratios are equal. This allows the white balance to be adjusted so that white objects appear white. However, this method may not work well in certain scenes, such as a blue sky. In recent years, machine learning has been used for color temperature estimation, and this technology is now being implemented in cameras.
Autoexposure adjusts the camera’s exposure by controlling the lens aperture and shutter speed or exposure time. In automotive cameras, where the lens aperture is fixed and there is no mechanical shutter, the function typically adjusts the image sensor’s exposure time and gain. It calculates the brightness from the pixel values in specific regions, averages them, and compares the result with a target value to determine if the exposure time is appropriate. If the image is overexposed, the exposure time is shortened and the gain is reduced; if underexposed, the exposure time is lengthened and the gain is increased.
So far, we’ve looked at some of the functional blocks of the camera system, but in addition to the features mentioned here, camera systems include many other functions. To extract the full performance of a camera system, it is important to correctly understand the features and limitations of the functions and use them appropriately.
Now, let's shift gears and discuss the two types of electronic shutters for image sensors that are frequently inquired about by our customers: global shutters and rolling shutters.
An electronic shutter in an image sensor refers to the process of controlling the charging time by releasing the charge before the readout occurs.
As explained in the second installment of this series, in CMOS image sensors, the pixel array is selected using an XY method for shuttering and readout. Therefore, the timing of shuttering or readout differs depending on the position of each pixel in the array.
Here’s an example using an 8x8 pixel array.
Illustration of rolling shutter operation
In this case, the shutter for the first row is triggered at time = 0. From time = 1 to time = 4, the photodiodes accumulate charge through photoelectric conversion, and at time = 5, the readout occurs. However, for the last row, the shutter occurs at time = 7, and the readout takes place at time = 12. As a result, when capturing fast-moving objects, rolling shutter distortion may occur. It’s worth noting that in recent years, as the readout time per line has decreased, rolling shutter distortion has become less noticeable.
Example of rolling shutter distortion (Credit: Dicklyon; CC BY-SA 4.0 via Wikimedia Commons)
In CMOS image sensors with global shutters, the pixel circuitry is designed to enable simultaneous shutter operation across all pixels, along with a full-frame readout.
In commercially available products, "global readout" doesn’t mean that the output from all pixels is simultaneously A/D converted and read out. Instead, the signal from each pixel is first stored in a memory section within the pixel (global transfer) and then read out sequentially. This is the most common method used in global shutter CMOS image sensors.
In this example, the global shutter operation occurs at time = 0, the global transfer happens at time = 2, and from time = 3 onward, the signal is read out row by row.
Each shutter method has advantages and disadvantages, and as such they are suited to different applications.
Considering the advantages and disadvantages, selecting the appropriate shutter type based on the application is crucial. In TIER IV cameras, rolling shutter image sensors are used, as HDR and LFM are critical requirements for automotive applications.
In this installment, we covered the basic operations of image sensors and explained the differences between global and rolling shutters. The final installment will focus on the interfaces used in automotive cameras.
TIER IV's automotive cameras offer top-tier performance and reliability. With a high dynamic range of 120 dB, the C1 Camera (2.5 MP) and C2 Camera (5.4 MP) capture clear images in both bright and dark environments. Trusted by over 100 companies worldwide, these cameras are now available via the TIER IV store on Amazon.
TIER IV is always on the lookout for passionate individuals to join our journey. If you share our vision of making autonomous driving accessible to all, get in touch.
Visit our careers page to view all job openings.
If you’re uncertain about which roles align best with your experience, or if the current job openings don’t quite match your preferences, register your interest here. We’ll get in touch if a role that matches your experience becomes available, and schedule an informal interview.
Inquiries
Social Media
X (Japan/Global) | LinkedIn | Facebook | Instagram | YouTube
More