CamCyclopedia
Outline
The Samsung Galaxy Camera packs powerful hardware and intelligent software to deliver the ultimate camera experience.
The default Samsung camera app preloaded on Galaxy demonstrates the hardware and software prowess of the Galaxy camera to the utmost. In addition to the default photo modes, there are tons of other modes and options for taking photos, including Night Mode, Portrait Mode, Pro Mode, Slow Motion, Single Take, and Director's View. Night mode lets you capture the best photos and videos in low-light, while single-take lets you shoot a video and save the result in a variety of AI-synthesized formats.
We've also released the Expert RAW app in the Galaxy Store to further provide a RAW capture mode for pros. You can use any lens to capture high-quality HDR shots, and you have fine-tuned control over options like shutter speed, exposure, and white balance.
The Samsung Camera development team is working hard to bring the power of the Samsung Galaxy Camera to the Android camera app developers, beyond the Samsung Camera app. Our goal is to provide app developers with an enhanced camera development ecosystem to bring the best camera experience to even non Samsung camera users. We've decided to call this whole ecosystem the ‘Samsung Galaxy Camera Ecosystem’.
Let’s introduce you to the Samsung Galaxy Camera ecosystem, which will bring the Samsung Galaxy Camera experience to 3rd party cameras.
Contents
- Android Camera API & Framework
- Android 13 Collaboration with Google
- Camera Extensions
- Samsung Camera SDK
- Video Call Effects
Android Camera API & Framework
(Source: https://source.android.com/docs/core/camera#architecture)
Android app developers can use the Camera2 API in the Android SDK to implement camera functionality. (Since Android 5.0 started supporting the Camera2 API, the original Camera API has deprecated. We recommend using the Camera2 API for enhanced camera functionality in complex scenarios.) Refer to the Android Developer's Guide to find out what features are supported by each camera.
Different Camera Software Layers
- Application
- Framework
- Hardware Abstraction Layer (HAL)
- Image Quality
- Solution
Normal camera operation on Android devices requires the interaction of many software layers and hardware modules. Not only the default Samsung Camera app on Samsung Galaxy, but also all camera apps developed with Android SDK must operate normally. To ensure the compatibility and stability of the Android Framework and Hardware Abstraction Layer (HAL) on devices, the Compatibility Definition Document (CDD) conditions must be met, and it should also pass various automated verifications such as CTS, VTS, and ITS and get approved by Google before the Android devices can be released. Of course, these basic procedures are the minimum qualifications, and Samsung is also conducting high-quality automated verification and user verification.
- Compatibility Definition Document (CDD)
- Compatibility Test Suite (CTS)
- Vendor Test Suite (VTS)
- Camera Image Test Suite (ITS)
- Compatibility Definition Document (CDD)
First, there is a compatibility definition document called a CDD. The CDD serves to stipulate and clarify specific requirements and eliminate ambiguity. It is not a complete specification document as it is not comprehensive document on all specifications. It is compatible with Android code, SDK API documentation, CTS, etc. It is deployed for each Android version.
Ex)
[C-1-3] MUST NOT use a front-facing camera as the default for the Camera API and MUST NOT configure the API to treat a front-facing camera as the default rear-facing camera, even if it is the only camera on the device.
You cannot use the default Camera API camera as the front camera. Even if there is only one camera on the device, the front camera should not act as the default rear camera.
This is kind of thing is hard to understand unless it is documented.
- Compatibility Test Suite (CTS)
CTS tests compatibility. It runs directly on a device or emulator connected to a PC to detect incompatibilities early and maintain compatibility. It tests compatibility at various layers, from the platform API to Data, Intents, Permissions, Resources, Signature, and Dalvik tests. It is the most fundamental and largest scope of testing in Android OS development.
- Vendor Test Suite (VTS)
VTS tests the vendor's deployment of the kernel, HAL, etc. with gtest, JUnit, kernel test, etc.
(Source: https://source.android.com/docs/core/architecture#hidl)
In Android 8.0, we unveiled Treble architecture, which separates the Android OS framework from hardware-dependent vendor deployments. This architecture allows vendor images to be reused when updating the OS. The vendor interface in the figure above should be compatible with both the old and new OS. The VTS includes testing implementation with this vendor interface.
- Camera Image Test Suite (Camera ITS)
Cameras work with sensitive image sensors and behave differently in different environments, such as exposure conditions when shooting. While automated testing often utilizes a virtual, automated environment, there is a need to verify image results in a real-world environment. ITS consists of a suite to set up and test real-world camera environments utilizing defined structures such as boxes.
(Source: https://source.android.com/docs/compatibility/cts/camera-wfov-box-assembly)
This is one of the ITS tests. It is a kit for testing the field of wide-angle camera view. You can install an Android device in it and test it while changing the position of the device. There is a specification for this kit on the Android developer’s site, you can build your own based on that, or purchase the official kit directly.
Within the scope of the above standardization conditions, Samsung Camera has implemented and used its own path rather than the basic API and HAL Interface of the Android Framework for features not officially provided by Android.
While our policy direction was to deliver the best camera app possible, there are many features that we had to limit due to the limitations imposed by our specialisation (distinct characteristics).
However, we've been working on integrating our own code into AOSP for a while now. This is because the less custom code a device manufacturer has to write and the more it follows the AOSP code, the easier it is to maintain, and the better it is for the entire Android ecosystem.
In Android 13, we've collaborated with Google to make some of the features we've developed and used in-house, available as public APIs. Features that were only available on Samsung cameras can now be implemented and used by general Android applications. We collaborated with Google from early stages of development, making plans together, developing and testing together.
Android 13 Collaboration with Google
- Torch Flash Strength Control
- Stream Use Case
- Preview Mirror Mode
- Camera Preview Stabilization
- Camera-Display Sync
Features that were implemented internally by Samsung until the previous OS version have been released as Android public APIs in collaboration with Google. Due to the OS update policy, many of the features are mainly supported in models released with Android 13. Naturally there are some regrets in terms of backward compatibility but we are taking the long-term perspective.
- Torch Flash Strength Control
- Developer document
On Android, you can control the flash with the Camera2 API by default without any camera permissions. Previously, there was only an on/off function, but as of Android 13, functions and callbacks have been added to adjust the level of flash light. The native flashlight feature on the Galaxy has its own API to adjust the brightness, which is now public.
Since API level 23
val cameraId = "0"
val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
val torchCallback = object : TorchCallback() {
override fun onTorchModeChanged(cameraId: String, enabled: Boolean) {
// get enabled
}
}
// register
cameraManager.registerTorchCallback(torchCallback, handler)
// Turn on flash
cameraManager.setTorchMode(cameraId, true)
Since API level 33 (Android 13)
val cameraId = "0"
val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
val torchCallback = object : TorchCallback() {
override fun onTorchStrengthLevelChanged(cameraId: String, newStrengthLevel: Int) {
// get newStrengthLevel
}
}
// register
cameraManager.registerTorchCallback(torchCallback, handler))
// Read default level and max level value
val cameraCharacteristics = cameraManager.getCameraCharacteristics(cameraId)
val defaultLevel = cameraCharacteristics.get(CameraCharacteristics.FLASH_INFO_STRENGTH_DEFAULT_LEVEL)
val maxLevel = cameraCharacteristics.get(CameraCharacteristics.FLASH_INFO_STRENGTH_MAXIMUM_LEVEL)
// Turn up flash to a specific level
cameraManager.turnOnTorchWithStrengthLevel(cameraId, 3)
// Read current flash level
val currentLevel = cameraManager.getTorchStrengthLevel(id) // return 3
- Stream Use Case
- Developer document
Stream in Camera2 API, is a buffer that consists of information like image format, size, etc. You can create streams with respective image formats and sizes according to scenarios like Preview, Capture, Video, etc. and also use multiple streams simultaneously for image analysis and much more.
Features that autonomously optimised types of stream by declaring them as options have been implemented as Stream Use Cases, allowing you to clearly state and use values such as PREVIEW, RECORD, VIDEO_SNAPSHOT, SNAPSHOT, VIDEO_CALL, etc. We are optimising image quality and performance for each stream use case, such as maintaining high quality in the case of SNAPSHOT and optimising current consumption for long-term use in the case of VIDEO_CALL.
- Preview Mirror
- Developer document
The reason the screen looks like a mirror on the front camera preview screen is because the framework is mirroring the camera preview screen based on the camera's mounting angle (0, 90, 180, 270) and position (front, back).
The mirror mode option got added to the camera OutputConfiguration from Android 13 onwards, allowing you to switch the flip function and flip axis from VERTICAL to HORIZONTAL and vice versa. It is not a big deal for the default value of front/rear camera in existing bar type models to be set to MIRROR_MODE_AUTO. As for foldable models such as Galaxy Z Fold and Galaxy Z Flip, the mirror function works via the VERTICAL and HORIZONTAL in Rear Cam Selfie and Quick Shot modes that use the rear camera.
The Quick Shot mode, uses the MIRROR_MODE_V value to flip up and down, and the Rear Cam Selfie mode uses the MIRROR_MODE_H value to flip left and right, which causes the screen to resemble a mirror.
- Camera Preview Stabilization
Video stabilisation can be achieved in several ways: optical, electronic, and digital. With the Camera2 API, you can call up and use digital video stabilisation values. Digital video stabilisation typically utilises an extra margin area cut out of the screen to create a smooth, steady video using gyroscopic sensors and video analytics.
Sometimes internal edits while filming may result in a difference between the preview and the final product.
The new CONTROL_VIDEO_STABILIZATION_MODE_PREVIEW_STABILIZATION setting, you can get a preview of upto 10% correction margin (in S22), and shoot with stability correction in real-time after the angle of view narrows down. The advantage of VIDEO_STABILIZATION is that you can use this feature directly from the Preview screen.
- Camera-Display Sync
The reworked camera display code is incorporated in the Android framework.
The screen output is usually at even intervals to match the VSYNC cycle. While the actual camera imaging happens at regular intervals, but it is sometimes processed and delivered to the Display or Codec at irregular intervals. This untimely delivery of camera output can cause frame drops.
Samsung has added a code to optimise the camera screen output to match the Display's refresh cycle, to minimise this Display frame drop. We also shared the idea with Google and collaborated to optimise it. Now, it is also a part of common Android Code to minimise Display frame drops.
Camera Extensions
CameraX is a Jetpack Library for Camera released by Google in 2019. Like other Jetpack libraries, it serves as a support library for Android SDK. CameraX is backward compatible up to Android 5.0 (API Level 21) and APIs such as Preview, Image Analysis, Image Capture, Video Capture, etc. are easier to use than the existing Camera2 APIs.
CameraX provides camera-specific features provided by device manufacturers in a separate library called CameraX Extensions. The CameraX Extensions library allows you to use unique shooting modes supported by each vendor.
Samsung has been collaborating with Google to develop the Extensions feature since the early stages of CameraX development, and the CameraX Extensions feature is now available on many Galaxy models. Once each handset company deploys the OEM Vendor Library, the CameraX Extensions library will be available on an app. Currently, AUTO, Bokeh, Face Retouch, HDR, and Night modes are supported.
Videos of Google I/O and SDC introducing CameraX.
- Android Jetpack: Understand the CameraX camera-support library (Google I/O'19)
- Introducing the CameraX with Samsung (SDC19)
(Source: https://source.android.com/docs/core/camera/camerax-vendor-extensions#architecture)
Starting with Android 12, the Camera2 Extensions API was added to allow the CameraX Extensions library to be used with the Camera2 API. If your device already has an OEM Vendor Library that is used by the previous CameraX Extensions, you can use the same with the Extensions API in Camera2. Apps using the Camera2 API can now implement and use special features similar to the Camera2 API without using CameraX. Since CameraX and Camera2 APIs eventually use the same library under the hood, we have integrated the two recently and call them Camera Extensions.
Samsung Camera SDK
The Samsung Camera SDK is available for select vendors to enable Samsung camera features that cannot be installed with the current Android SDK and Camera Extensions.
The Samsung Camera SDK is based on the Camera2 API and works by calling the API of the Camera SDK Client and running the Samsung Camera solution on the connected SDK Service.
Supported Features
- HDR
- Super Night
- Smooth Zoom
- Super Resolution
- Video HDR
- VDIS
Snapchat is a popular app that uses the Samsung Camera SDK. As announced at the Galaxy S22 Unpack event and SDC, Snapchat supports HDR shooting, Super Night, Smooth Zoom, and Super Resolution features of the Samsung Camera SDK on the Galaxy S22. Snapchat's night mode allows you to take bright, high-quality shots in low-light. You can also use any camera lens for smooth transitions and high quality Tele Camera shots. In addition, certain modes in Snapchat use the Video HDR and Preview Stabilization of Samsung Camera SDK.
Video Call Effects
We've gone beyond the previous approach, to build a new concept platform that allows you to apply Samsung Camera solutions to 3rd party apps without requiring a separate SDK or any modifications. With the ‘video call effects’ feature above, you can add Samsung camera solutions to 3rd party video call apps. You can set the background you want in the video call effects settings and apply the effect in real time (while you are on a call) via the floating button.
You can apply the options by tapping the floating button you see while using the video calling app. You can blur, change color, or replace the background with an image of your choice. You can also use the Auto Framing Mode option to automatically adjust the shooting angle and zoom based on the number and composition of the people detected.
When you lower the value of the mode you want to apply in the video call effects settings, this is how the results set through the Samsung Camera Framework and Samsung Camera Solution are further applied to the video call app.
Concluding Remarks
So far, we've covered five topics to introduce you to the Samsung Galaxy Camera ecosystem. We're working hard to ensure compatibility and stability of the underlying Camera APIs, while also opening up our own internal features to new APIs. We are providing specialised features for the Galaxy Camera through extensions and SDKs for CameraX and Camera2, and providing additional features for video calling apps with our own platform. We are building a camera development ecosystem for app developers, in this way.
We're just getting started with the Samsung Galaxy Camera ecosystem, and we'll be bringing new features to new models and the next Android OS. We're still working on a lot of things, and will continue to lead the mobile camera ecosystem towards better image and video quality, new visual experiences, and ease of implementation. Stay tuned.
5 Comments
