CamCyclopedia
Contents
1. What is ISP?
2. Basic structure of ISP
3. Major blocks of ISP
4. ISP in Galaxy Camera
5. Other ISP blocks
6. Reference
What is ISP (Image Signal Processor)?
ISP is a device that processes digital data obtained from an image sensor. It can enhance, restore, convert, and compress images and more. All cameras come equipped with a built-in ISP chip. They existed as a separate chip in mobile phones, but now most ISPs are embedded within an AP (Application Processor).
The image sensor sends signals to ISP within the AP in the Galaxy camera, and ISP is responsible for the final image you see.
Default structure of ISP
Every AP Chip manufacturer has their unique ISP structure. We will introduce the most basic structure here.
- Basic ISP block diagram
Main ISP blocks
Let’s look at the 8 basic and representative ISP blocks mentioned above.
1.Pedestal Cut
Owning to multiple physical properties of the image sensor (dark current, etc.), even when it is dark and has no data, a certain level of value is fed into the ISP. If this barely legible (distorted) sensor data is not removed, the image looks like its floating or the black does not look black.
For example, you can think of it as weighing something in a plastic bag with a precise scale. You weigh the bag separately and discount this weight from the weight that shows up on the scale.
2. LSC (Lens Shading Compensation/Correction)
Lens Shading is the reduced intensity of light or saturation in the image as you move from the centre towards the perimeter of an image, because of the optical property of the lens. LSC (Lens Shading Compensation or Correction) is a block that corrects lens shading.
The correction applies the gain at the centre and the corners differently, i.e. it uses more gain at the periphery to keep the overall balance.
Generally, Lens Shading is mostly talked about in terms of brightness. If we add ‘color’ in the front, "Color Shading" would refer to the difference in color at the centre and the sides.
How to correct/compensate lens shading
The gain at the centre and sides are applied differently (More gain at the sides. For example, if the gain at the centre is 1x, the gain at the corner is put to 5x).
3. WB (White Balance)[3]
This block helps the camera adjust the color balance and render them accurately. It ensures that an image has a uniform WB Gain and that the RGB of the grey (achromatic) areas are adjusted to the same value (to depict the grey as grey).
The sensor output fed to ISP has twice as many green channels as any other. This is the reason why the raw image before the application of WB Gain looks green overall. (Refer to 'De-mosaic' below)
4. De-mosaic (= Bayer Interpolation, Color Interpolation)
Most smartphone cameras use the Bayer type (or pattern) sensor. The Bayer sensor, as illustrated below [2] is 3 color filter combination sensor mounted on top of a photodiode array.
=>A single pixel is either a red, green or blue pixel.
This filter is called a ‘Color Filter Array’, and any image filtering through this color filter array has one color per pixel. These one-color pixels go through a color reconstruction process called ‘demosaic’ where they use the information of the neighbouring pixels to render a full 3-color image.
The block performing the ‘demosaic’ is usually known as Demosaic / Bayer Interpolation / Color Interpolation, etc.
5. CCM (Color Correction Matrix)[3]
The CCM block compensates for any inaccurate (short-range) image sensor-induced color representation. The color results produced by the sensor are quite different from how humans see colors. The CCM block is the one that converts these results into colors perceived by the human eye.
It corrects colors in real-time for RGB data input, by multiplying the increase/decrease of gain in red, green or blue.
6. Gamma
It is usually called ‘Gamma Correction’. It transforms the intensity of incident light, using a non-linear function (I/O is not 1:1). It is known that Braun tubes (cathode-ray tube or CRT) were used popularly in the past and gamma correction came to its aid in correcting its non-linear output. You can take a look at this non-linear character illustrated in graph no. 2 of the below image.
Even though technology developments have led to Braun tubes producing near linear outputs (diagram 1 in the image below), gamma correction continues to be used concomitantly with them. The main reason behind using gamma correction, in addition to the nature of the output device, is that human vision also responds non-linearly. According to Weber's law, human vision responds to the stimuli of light (brightness) in a non-linear way [4]. That is, human eyes become more sensitive to any change in light, in the absence of it (relative darkness), and become relatively impervious to any change in its intensity in a bright environment.
We need a ‘non-linear function’ or gamma correction to fix this sort of non-linearity of human vision (Graph no. 2). The third graph below (right) is how it is usually applied i.e. the third graph is the corrected version of the second non-linear function graph.
- After gamma fix
7. NR (Noise Reduction)
The NR Block helps reduce noise in images/videos. Below are the two most common and basic NR methods.
One way is to find the average of pixels around the noise and manage the noise pixel data using that average value, and another is to find the median of the pixels distributed around the noise and handle the noise pixel data with this median value (median filter).
But, it inevitably follows that too intense an NR will result in a loss of sharpness/detail. It is always a trade-off between NR intensity and sharpness.
The goal of NR is to reduce noise while preserving as much of the information on each pixel received from the image sensor.
Every AP chip manufacturer has its own technology/structure when it comes to NR algorithms and mechanisms. It can be broadly divided into two kinds below.
The first one is 2D-NR or Spatial-NR. It uses the NR mechanism mentioned above (handle it using the average or the median value of surrounding pixels) to remove noise in an image (one frame).
The second is 3D-NR or Time-Domain-NR/Multiframe-NR. This takes multiple shots (multiple frames) to better separate the noise from the signal and applies NR only to the noise pixels as much as possible. The basic concept is similar to the CamCyclopedia Multi-frame Processing technique. The only difference lies in where it is applied, the SW or the HW.
8. Sharpen (Edge Enhancement)
This block compensates for/optimizes the loss of detail/sharpness caused by the NR (Noise Reduction) block above or hardware performance limitations such as sensors/lenses.
Some of it is just for post-edit and the idea is to widen the disparity between this area and the surrounding pixels. So, some degree of increase in noise is imminent and you can also experience some unintended image distortions. In other words, it's a trade-off between Sharpen and NR as discussed in #7.
Each AP chip manufacturer has its own technology/structure for ‘sharpen’, and the most basic way is to increase the pixel difference at the edge (Figure 2 below) or decrease the gap in slope (Figure 3 below), as shown below.
The first image below is the original. The second image has only gone through step 3 (figure 3 above), and the third image has gone through both steps 2 and 3 above.
ISP in Galaxy camera
Now, we'll give you an illustrative overview of what each of the above ISP blocks in Galaxy Camera does.
Pedestal Cut
LSC (Lens Shading Correction)
Demosaic (Bayer Interpolation)
CCM (Color Correction Matrix)
Gamma
NR (Noise Reduction)
Sharpen
Other ISP blocks
Let’s look at other ISP blocks in addition to the main ISP blocks we saw above. They may vary by AP chip manufacturer but are provided by most.
FD (Face Detection)
This block detects faces in an image/video and passes their coordinates to the next block. The face coordinates can be used in other algorithms or solutions to implement different features.
[Example of application] Using FD information in AWB (Auto White Balance) and AE (Auto Exposure) algorithm blocks
- When a face is detected, it compares and corrects the difference between the face brightness and the targeted brightness of the whole image. It utilizes the face coordinates received, to adjust the face brightness, etc.
- When multiple sources are detected illuminating a face and the face is overly affected by these lights, this FD data can be used to correct the color on it. For example, if a face is detected in a low-color temperature light setup giving off a slightly reddish glow, the face coordinates are used to adjust the color of the face.
Contents Aware [4]
This feature allows you to define some specific items within an image/video and categorize them accordingly. Furthermore, this allows you to apply separate image quality settings to different segments of an image. For example, you can segment the image into hair/grass/sky and apply different IQ settings to each area.
CamCyclopedia Index - Samsung Members
You can also access CamCyclopedia anytime by going to Community -> Category (app) -> CamCyclopedia -> “CamCyclopedia Index”.
Reference
[1] Some of the pictures above come with graphic effects for demonstration purposes.
[2] Bayer Color Filter: https://en.wikipedia.org/wiki/Bayer_filter (Artwork by Cburnett CC BY 3.0)
[3] You can find more details in the article ‘Color in Camera’ from the CamCyclopedia index.
[4] https://namu.wiki/w/베버의 법칙
[5] We are in the process of applying this to some of the major models like Galaxy S22, etc.
5 Comments
