savant framework
Few weeks ago we share here on the blog a little information about Savant which is video analysis framework that makes it easy to use NVIDIA DeepStream to solve machine learning problems. The reason for talking about it is that recently announced the release of the new version of «Savant 0.2.4“, in which new features have been integrated and the use cases of this great framework have been expanded.
For those unfamiliar with the framework, you should know that this takes care of all the work with GStreamer or FFmpeg, allowing you to focus on creating optimized output pipelines using declarative (YAML) syntax and Python functions.
Learned hides all internals of Gstreamer from the developer and provides practical tools for implementing streaming AI applications in real life. Uses the standard Nvidia PeopleNet model to detect people and their faces and especially in places where privacy regulations apply, the framework allows tracking and blurring of faces.
What's new in Savant 0.2.4?
As we mentioned at the beginning, this new release comes with new features which have expanded the use cases of the framework and of which it is mentioned that for example now Savant can be used in age/gender prediction, demonstrating the use of YoloV5-Face, working with a custom attribute model that predicts age and gender, and GPU-affine transformations based on facial features using OpenCV-CUDA and Python.
Another use case is conditional video encoding, which demonstrates a pipeline that is frame-based and encodes the video stream only on demand (in the example, only when the model detects objects); shows how to avoid wasting computing resources when a certain external condition requires frames.
Also now it is possible to use Savant to handle multiple RTSP streams, Demonstrating a simple pipeline that processes two RTSP streams, and because Savant is very different from what users expect in terms of dynamic thread processing, we implement a simple pipeline that processes multiple threads at the same time to show how it works.
On the part of the new features added, it stands out the conditional rendering and coding to reduce traffic and efficiently use CPU/GPU resources, as well as a new FFmpeg based RTSP source adapter which works much better than GStreamer when streams include B-frames and a new FFmpeg based universal adapter that can work with all FFmpeg compatible inputs.
In addition to this, it is highlighted that an error was identified in the implementation of the NVENC functionality on Jetson:NVENC devices that incorrectly streamed encoded frames if the actual frame rate in the stream is not equal to the configured one, which often happens when working with RTSP streams or when frames are dropped due to certain conditions . In Savant, the problem is solved by a workaround by reordering the frames when necessary. The bug appears in DeepStream 6.2 and has been reported to NVIDIA, which has acknowledged the bug and will fix it in the next DeepStream release.
Of the other changes that stand out of this new version:
- During development, the practice of tracking possible performance regressions by merging each ticket was introduced.
- Migration of Savant internals from Python to Rust continued: Savant-rs core function library implemented with extensively tested code.
- Python-based components are gradually being replaced with Rust-based components to ensure that Savant works without a global lock (GIL) whenever possible and that the code is of high quality.
- Documented adapters for working with data.
Added examples of using image preprocessing.
Created a section on how to set up a development environment in VS Code.
Finally, it should be mentioned that the developers plan in the next release of Savant 0.2.5 to integrate more Rust code so that pipelines depend less on GIL. It is also planned to deliver new features related to dynamic pipeline configuration and edge development, and add three to four new samples covering basic and advanced features.
If you are interested in knowing more about it, you can consult the details in the following link