Presentation of SoftISP at fosdem 2024
During the FOSDEM 2024 conference, Hans de Goede, a Fedora developer working with Red Hat, I talk about the current problem that have been identified in many of the computerss recent Intel laptops as they have replaced the UVC USB camera module standard with a raw MIPI camera sensor connected to the IPU6.
Mention that MIPI interface is being adopted instead of video transmission via USB bus, which was common in devices supporting the UVC (USB Video Class) standard. MIPI provides access to the camera sensor through a CSI (Camera Serial Interface) receiver and an Image Signal Processor (ISP) integrated into the CPU.
This approach enables image capture based on raw data directly from the sensor. Intel provides a set of proprietary drivers to work with MIPI cameras on Linux via IPU6 on Intel Tiger Lake, Alder Lake, Raptor Lake and Meteor Lake processors.
Both the hardware interface of the ISP part of IPU6 and the image processing algorithms used are considered a trade secret and so far the only Linux support for IPU6 is based on an out-of-tree kernel driver with a userspace stack proprietary at the top which is currently available on rpmfusion.
Both Linaro and Red Hat have identified the lack of ISP support for various ARM and X86 chips as an issue. Linaro has started a project to add a SoftwareISP component to libcamera to allow these cameras to work without the need for proprietary software and Red Hat has teamed up with Linaro to work on this.
Hans, mention that the main difficulty in the development of open drivers for MIPI cameras it lies in the fact that hardware interface ISP processor and image processing algorithms generally implemented in it are not disclosed by CPU manufacturers and are kept as a trade secret. To address this problem, Linaro and Red Hat have collaborated to develop a software implementation of an image processor called SoftISP, which allows working with MIPI cameras without relying on proprietary components.
The implementation of SoftISP has been proposed for inclusion in the libcamera project, which provides a software stack to work with camcorders, cameras and TV tuners on Linux, Android and ChromeOS systems. In addition to SoftISP, the stack for working with MIPI cameras includes a driver for ov2740 sensors that operates at the kernel level, as well as code to support the CSI receiver in the Linux kernel, which is part of the IPU6 of Intel processors.
Currently, the SoftISP stack has not yet been accepted in the Linux kernel or in the libcamera project, but development is considered to have now reached a stage suitable for testing by a wide range of enthusiasts. The functionality of this stack has been tested with MIPI cameras based on the various sensors of Lenovo, Dell and HP laptops.
For interested in the project, You should know that the Linux kernel and libcamera packages, which include the project changes, are now available in the repository COPR for installation on Fedora 39. The Pipewire media server can be used to capture video from MIPI cameras and support for cameras via Pipewire has already been integrated into the libwebrtc library.
In addition to this, it is worth mentioning that in Firefox, the ability to work with cameras through Pipewire has reached a state suitable for use with WebRTC as of version 122. By default, the functionality to work with cameras through Pipewire in Firefox it is disabled and requires the parameter «media.webrtc.camera.allow-pipewire» in about:config.
Finally, if you are interested in being able learn more about it, you can check the details in the following link.