Deep-Learning driven image quality-enhancing AI Technology

[Admin]

Content

1. Concept of deep learning driven image quality-enhancing AI technology
2. Low light image quality-enhancing AI model: AI Nightography
3. High-resolution AI image processing algorithm: AI High-Resolution

Concept of deep learning driven image quality-enhancing AI technology

As deep learning AI technology advances, software deep learning IQ (image quality) -enhancing AI models are slowly taking over the role of hardware camera ISP (Image Signal Processor) processing driven by traditional ML (machine learning) signal processing technology. These models are increasingly used to improve image quality instead. Deep learning-based solutions, in particular, perform better for degraded or distorted images against low light or dense resolutions.

H/W ISP vs S/W AI-ISP

The image below shows how the Camera ISP converts the input signal from the sensor into the final RGB image visible to you.

The H/W camera ISP is made up of several blocks for traditional ML-based IQ processing and usually functions under daylight shooting. The configuration and performance vary depending on the AP (Application Processor) manufacturer.

S/W camera AI-ISP [1] works under low light and for high-definition captures. Input image pre-processing and output image post-processing is required to optimize the performance of deep learning image quality-enhancing AI models. The core image quality enhancement AI model can use various network architectures such as ANN (Artificial Neural Network), CNN (Convolutional Neural Network), ViT (Vision Transformer), etc.

Use of deep learning by Galaxy Camera

Galaxy Camera uses supervised and unsupervised ML AI models to deploy miscellaneous image quality enhancements such as demosaicing, noise removal, and increase in sharpness.

First, supervised learning uses low- and high-definition videos as training data for image quality-enhancing AI models. The quality of the training data impacts image quality significantly.

Second, unsupervised learning trains the image-enhancing AI model to enhance low-quality images without feeding it any high-quality images.

The trained image-enhancing AI models are optimized for application to Galaxy cameras and carry out actions quickly in association with the smartphone graphics processing unit (GPU) or neural processing unit (NPU).

Low light image quality-enhancing AI model: AI Nightography

In low-light environments, noises of different kinds in the camera output distort the image detail. In general, well-trained network-based image-enhancing AI models outperform non-network-based solutions in improving image quality. The night mode in Galaxy camera offers a Nightography experience that captures bright, vibrant colors and subject details using AI technology, even at night or in the dark.

Types of low-light noise

Photon shot noise
Unlike when there is plenty of light, in low light conditions, many photons bounce off the camera sensor's photodiode and are less likely to form a hole. As a result, the pixel-to-pixel calculations are unstable, resulting in photon shot noise. This amplifies with ISO amplification, i.e. analog gain amplification.

Readout noise

Readout noise is unique to the amplifier converting analog image sensor signals to digital in the readout process. Unlike photon shot noise, this is not significantly affected by ISO amplification.

There are many other forms of noise, including long-exposure dark current, sensor reset, quantization, black-and-white dots, etc.

The photo below shows the distribution of photon noise as per a camera ISO setting from Wikipedia. [2] You can see that the lower the lighting conditions, the more distorted the image becomes.

Galaxy camera’s use of IQ-enhancing AI model

How to use ‘night mode’ in Galaxy Camera
When you take a photo in low light with "Use Night mode" on in Auto mode or Portrait mode or "More > Select night" on your Galaxy Camera, as shown in the image below, the image-enhancing AI model kicks in and removes noise for cleaner and sharper output.

Low-light image-quality enhancing AI model input-output

The image-enhancing AI model applied in Night mode effectively removes any color and dot noise, and minimizes damage to original details providing consumers with bright and sharp results.

The figure below shows an example of a low-light image-enhancing AI model IO (input and output).

The left image is a low-light input with a lot of color noise.

On the right, the image-enhancing AI model reduces image grain/distortion.

High-resolution AI image processing algorithm: AI High-Resolution

It uses deep learning AI image-processing technology to overcome the physical limitations of small pixels (noise and loss of detail).

Smaller pixels mean less light reaches the pixel. They contribute to more noise.

AI image processing optimized for high-megapixel sensors dramatically improves perceived image quality and delivers high-resolution images that match the pixel count.

What is a high-megapixel sensor?

High-megapixel sensors are mainly used in the S22 Ultra and S22/S22+, which have 108MP and 50MP pixel counts, respectively.

Typical sensors use a Bayer Color Filter Array (CFA) structure, while high-megapixel sensors consist of Nona CFA (108 MP) and Tetra CFA (50 MP).

As the number of pixels on the same chipset size increases, the size per pixel decreases, and the physical light harvesting surface area also decreases.

AI Nona/Tetra Demosaic – Maximizing details using an improved demosaic AI algorithm

An image consists of three channels (red/green/blue), and an imaging sensor can typically only acquire one channel per pixel. Therefore, if the sensor obtains the green channel, it must guess the remaining channels (red/blue) to form the image. The process of converting the single-channel information obtained from the sensor into a three-channel image is called demosaicing.

A pixel can use a color filter to determine what color it will have. Imaging sensors are equipped with color filter arrays (CFAs) to distinguish colors, and all have a specific pattern.

Existing H/W ISPs take Bayer patterns as input, so the re-mosaic algorithm is required to convert non-Bayer patterns to Bayer patterns, which inevitably involves a loss of detail.

Meanwhile, the AI demosaic algorithm restores color information directly from Nona/Tetra structured RAW images through a trained neural network, thus maximizing detail and accurately restoring three colors from a single pixel.

Diversely patterned high-resolution images were used for training, to model the Nona/Tetra CFA patterns of the sensors and train the AI neural network.

AI Denoise – Overcoming the physical limitation of small pixels

The goal of a denoise algorithm is to remove noise from an image signal while preserving its details.

Noise removal is a double-edged sword. Removing is almost always accompanied by a loss of detail. So, over-removing noise does not bide well for image detail. It follows that denoising intensity must be within limits to preserve detail.

AI-powered denoising algorithms rely on neural networks to recognize patterns and apply appropriate filters. They can denoise and restore images with much more accuracy and detail retention than H/W ISPs' denoising methods.

It models the physical noise characteristics of the sensor, which in turn trains an AI neural network with thousand to tens of thousands of high-resolution images.

High-resolution mode in Galaxy camera - Detail Enhancer

The above-mentioned AI demosaic and AI denoise algorithms, combined with S/W ISP's pre- and post-processing algorithms, form an optimal pipeline, commercialized as a 'Detail Enhancer' in high-resolution mode.

A detail enhancer is implemented to run faster in a mobile environment by leveraging Galaxy's AI-specific H/W chipset, NPU, and the pipeline is optimized by the distribution of GPU/CPU resources.

■ Related Content

High-Resolution Camera - Samsung Members

CamCyclopedia Index - Samsung Members

You can also access CamCyclopedia anytime by going to Community -> Category (app) -> CamCyclopedia -> “CamCyclopedia Index”.

Reference

[1] S/W AI-ISP used for low light modifications are deployed in some models like S22, Z Fold4, Z Flip4, etc.

[2] https://en.wikipedia.org/wiki/Image_noise (captured by Mdf CC BY-SA 3.0)