CamCyclopedia
Content
1.There is a KPI for the camera too?
2. Key metrics and our efforts
3. An endless homework
4. Concluding remarks
5. Notes
There’s a KPI index for the camera too?
KPI stands for ‘key performance indicator’ and is a quantifiable metric used to measure the performance of a smartphone camera.
There are a variety of ways to quantify camera performance, as shown below. But, let's start with the most common metrics you might come across in a typical situation.
- Time taken to open the camera application
This is the time taken to launch the camera application and for the preview screen to appear.
- Time taken to flip between cameras
This is the time taken to wake a camera up from a standby state, and switch between front/rear cameras or camera lenses depending on the zoom.
- Shutter response time
The shutter response time refers to the time it takes from the moment you press the shutter button to the time a sound or screen effect notifies you of the finished shot.
- Time taken to capture
This is the time taken from pressing the shutter button till the saved result in the gallery. The capture time includes time taken to retrieve photos from the camera sensor, process different frames, and post-process activities like storing the results and any additional information.
- Frame drop status for a video
This refers to confirming if the frames in video footage have been saved at regular intervals. For example, information for a video recorded at UHD/FHD 30fps should be saved at a consistent interval of 33.33ms per frame. But, an unexpected issue may result in a loss of information in the middle (frame drop), causing the video to lose resolution and appear choppy.
Key metrics and our efforts
1. Reducing camera launch and lens switching time
- Hardware standby
Turning on the camera requires a lot of work, like controlling the power, loading image/coordinate calibration data, setting the camera sensor to operation mode, etc.
However, going through this every time the camera is turned on can tire out the user and make the camera launch seem slow. To circumvent this, the camera hardware is activated when the phone boots up and is kept on standby to enable the camera preview as soon as the camera application is launched.
- Early resource allocation
All data flow happens via memory read and write operations. The larger the video resolution, the larger the data size and the more time is required to allocate a buffer in memory to hold the data. The required memory is secured and allocated in advance to shorten this time.
- Clock boost
Clocks are a metric to measure how fast particular hardware components in a product work. This means that the same amount of data can be processed in less time at a higher clock speed. Boosting the clock speed of a relevant hardware component at the launch of the camera application will reduce the overall processing time. However, a higher clock speed draws a higher current. So, finding the right clock usage level and retention time is of the utmost importance to optimize/minimize battery consumption.
2. Taking pictures without any delay in the data processing
- Early shutter notification
When we say ‘notification’ here, we are referring to the signal that lets the user know that the photo is taken. If the signal has a significant lag after the shutter is pressed, or if there is no signal at all, the user won't know when the process is complete. They may end up moving the camera during image processing, blurring the image, or staying in the shooting position longer than necessary.
This begs the question, but what is a good time to send this signal? The answer is, it is best to send the signal once the phone has secured the photo to save, even if it's yet to be saved. That is the ‘early shutter notification’, and we're implementing the signal to be delivered as early in the processing path of the shot as possible.
- Pipeline optimization and parallel execution
The number of steps the camera sensor image goes through before it is displayed on the screen or stored is called a pipeline (image processing path). The more steps the pipeline has, the longer it takes for the data to be processed, and the longer it takes to click a picture.
Pipeline optimization is used as a remedy to reduce execution time by processing independent tasks simultaneously, or by dividing a single video into multiple pieces and processing them in parallel, eliminating unnecessary waiting time, data copying, and repetitive tasks that exist in the continuous data processing.
3. Filming videos without any frame drop
- Resource balancing
Video recording is happening at a constant frame per second (FPS), and the processing for each frame is mostly consistent throughout the video recording. So, the necessary hardware resources, such as CPU, GPU, NPU [1], etc. must work reliably without bottlenecks. If certain frames are not processed in time, some frames may drop.
For example, it is very important to allocate resources efficiently so that multiple functions that can operate simultaneously are not spent on any one hardware, such as NPU for facial recognition, ISP for image quality improvement, and GPU for image stabilization (VDIS).
- Exception/failure handling in time
When it comes to video recording, it's not so much about creating the perfect shot as it is about completing the required actions within a fixed time to ensure stable recording.
So, if the required data is not fed in time to the pipeline, or if certain hardware is overloaded and processing is temporarily delayed, instead of all processes coming to a halt, the camera has the flexibility to utilize as much existing data as possible to continue recording the video.
An endless homework
How to optimize for a real-world user environment?
A lot of testing and rigorous checking in a variety of environments is involved before releasing a product. But, with the number of different users and user environments, it is challenging to rise to everyone’s expectations of performance. The best way is to simulate optimization for each environment as much as possible. This includes repeatedly reading and writing large amounts of data, installing hundreds of applications, and running them continuously for 4 to 72 hours.
We evaluate how well the camera application performs in these situations and make improvements, such as adjusting priorities, based on the analysis results for each situation.
Speed VS Image quality, the polarity and negotiation
While it would be great to take photos quickly and get the best quality results every time, it is often difficult to achieve both going by the shooting backdrop.
A prime example of this is when you want good-quality photos while shooting in the dark, and have minimal light to work with. Your best bet is processing sufficiently exposed multi-frames to create a bright and clear picture. But, this inadvertently means a longer shooting time. [2]
We can only try to make the best choice depending on the ambient light, type of scene, and movement of the subject. Here's an example to help you understand.
- Multi frames (3-7) clicked against a well-lit background are processed and saved and help quick shoots
- For low-light settings, the camera adjusts the exposure to improve the quality of frame inputs (8 to 12) by changing the exposure and combining them
Concluding Remarks
Further, there are more techniques in use to improve camera performance than the ones mentioned. We're not stopping here. We will continue to put our minds to researching more ways to provide a better user experience.
CamCyclopedia Index - Samsung Members
You can also access CamCyclopedia anytime by going to Community -> Category (app) -> CamCyclopedia -> “CamCyclopedia Index”.
Reference
[1] NPU(Neural Processing Unit): Deep learning algorithm computation optimized processor that can quickly and effectively process bigdata like the neural network of humans.
[2] Multi-frame processing technology: Multi-frame Processing Technology - Samsung Members
3 Comments
