Nvds_obj_enc_process for whole frame leaves aritfacts over time

syther666 · July 12, 2022, 7:04am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

This is a component based on deepstream 6.0’s multi stream example. The pipeline structure is the same as examples, approximately following streammux → pgie → tracker → tiler → nvvideiconvert → osd

I added a buffer probe to pgie’s src pad and enable NvDsObjEncUsrArgs. In this function I set the following metadata:
NvDsObjEncUsrArgs userData = {0};
/* To be set by user */
userData.saveImg = true;
userData.attachUsrMeta = true;

Immediately after metadata creation I make sure process is encoded and call the enc_finish function.

                nvds_obj_enc_process(objEncContext, &userData, ipSurf, objMeta, frameMeta);
                // Seems to be an offset error when nvds_obj_enc_finish is called outside the for loop
                nvds_obj_enc_finish(objEncContext);

Then I keep the deepstream pipeline running and eventually I see the files getting written locally. (Default filename indicating frame # is correct as documented in NVIDIA DeepStream SDK API Reference: _NvDsObjEncUsrArgs Struct Reference) The contents of the few pictures look ok, but over time the images written gets blurry.

126_0_-1_Person_1920x1080.jpg

48_0_-1_Person_1920x1080.jpg

129_0_-1_Person_1920x1080.jpg

1_0_-1_Person_1920x1080.jpg

yuweiw · July 12, 2022, 11:04am

Hi @syther666 , Which sample app are you using for your test? Is it blurry if you save as a video file instead of saving as an image?

syther666 · July 13, 2022, 2:48am

Hi @yuweiw, it’s the same app as this except being Deepstream 6.0 version:
https://docs.nvidia.com/metropolis/deepstream/5.0/dev-guide/index.html#page/DeepStream_Development_Guide/deepstream_ref_app_deepstream_app.html

How do I save as a video file?

syther666 · July 13, 2022, 2:54am

Source code for app from deepstream 6.0:

github.com

NVIDIA-AI-IOT/deepstream_reference_apps/blob/master/runtime_source_add_delete/deepstream_test_rt_src_add_del.c

/*
 * Copyright (c) 2020-2021, NVIDIA CORPORATION.  All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

#include <stdlib.h>
#include <gst/gst.h>
#include <glib.h>
#include <math.h>

This file has been truncated. show original

yuweiw · July 13, 2022, 11:05am

Hi @syther666 , So what is your source(rtsp or file)? How do you build a rtsp server If it is rtsp src?
Maybe your source has been blurred already. You can try to set all the sources to file source. Thanks

syther666 · July 13, 2022, 1:54pm

I’m using file source, which is an mp4 file from nvidia samples.

yuweiw · July 14, 2022, 2:30am

Ok, Could you try to test our demo app deepstream-image-meta-test?

/opt/nvidia/deepstream/deepstream-6.1/sources/apps/sample_apps/deepstream-image-meta-test

If you just add simple code in our demo source, you can attach all your patches. Thanks

syther666 · July 14, 2022, 7:13am

deepstream_image_meta_test.c (21.3 KB)

Modified 4 lines around obj_meta->rect_params.height to have the same configurations as our app. Running ./deepstream-image-meta-test file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h265.mp4

Running the binary after building /opt/nvidia/deepstream/deepstream-6.1/sources/apps/sample_apps/deepstream-image-meta-test
from the docker image nvcr.io/nvidia/deepstream:6.1-devel gets a segmentation fault after -4 not-negotiated is thrown. The latest version of deepstream version also contains this seg-fault?

yuweiw · July 14, 2022, 7:32am

Do you confirm your display system in your docker env is OK? Cause this demo need display env.
Or you can change the code by yourself. Add the plugin below instead of nveglglessink and link them.

  nvvidconv2 =
      gst_element_factory_make ("nvvideoconvert", "nvvideo-converter2");
  nvenc = gst_element_factory_make ("nvv4l2h264enc", "nvh264enc");
  h264parse = gst_element_factory_make ("h264parse", "h264-parse");
  qtmux = gst_element_factory_make ("qtmux", "qt-mux");
  filesink = gst_element_factory_make ("filesink", "file-sink");

syther666 · July 14, 2022, 8:00am

Just replaced with fakesink
deepstream_image_meta_test.c (21.4 KB)

My laptop had some previous issues with OSD with our own deepstream derivation also.

Sample result as below:
1440_0_1_Person_1920x1080.jpg

,

yuweiw · July 15, 2022, 2:18am

Hi @syther666 , I use your code and deepstream 6.1 docker with the sample_1080p_h265.mp4, it have no blurry picture. I get the same picture that you attached, it isn’t blurry either. So could you try run more times to confirm few questions?
1.What’s the proportion of the blurry pictures? (20%, 30%…)
2. Are the blurry pictures the same all the time?
3.What’s the specific brand and driver version of your GPU?
Thanks

syther666 · July 15, 2022, 2:30am

Hi @yuweiw ,

Sometimes it starts earlier sometimes later; the example I gave begins on frame 7 and other times it starts maybe at picture 30. Frame 7 showing the first mismatch
Green squares and regularly colored squares appear seemingly randomly

Driver Version : 510.73.05
CUDA Version : 11.6
Product Name : NVIDIA RTX A3000 Laptop GPU

Thank you

yuweiw · July 15, 2022, 3:21am

What env do you run your docker? (linux on ubuntu or wsl on windows or vmvare on windows)

syther666 · July 15, 2022, 3:41am

Linux on ubuntu

yuweiw · July 15, 2022, 5:50am

Ok, the env have no problem. Could you try to save the video to a mp4 file.
You can just modify few codes from the config file below, Thanks

/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/source1_usb_dec_infer_resnet_int8.txt

1.change the source0 type to 2 and set the file uri(you can refer source2_1080p_dec_infer-resnet_demux_int8.txt)
2.disable the sink0 and enable the sink1 
3.run the cli: deepstream-app -c source1_usb_dec_infer_resnet_int8.txt

If your modify is ok, you can see a mp4 file in current path. Just play it and see if the picture is blurry.

syther666 · July 15, 2022, 6:07am

Hi @yuweiw ,
The video output from this binary is valid. The video quality output looks around 360-720p, but there is no distortion.

yuweiw · July 15, 2022, 8:05am

Let’s reconfirm it. We use the same docker, the same demo code you attached(deepstream_image_meta_test.c). In my env, it works well. Of cause, our gpu is different.
So, could you have a try about modify the quality of the encode by set the para below.

userData.quality = 80;

And if you have another gpu env, you can also try to test if the pictures are blurry in a new env.

syther666 · July 15, 2022, 8:39am

@yuweiw
Our GPU is different, that’s correct.

The version of the deepstream-app, is working though deepstream_image_meta_test.c is not. If deepstream-app is working, one way is you could point us the source for this instead.

In the c file I already have quality=80. Looking from frame 7, the bug found in deepstream_image_meta_test.c is originating from some sort of alignment offset inconsistencies or decoding errors instead of the actual image_quality parameter.

Observing that A3000 is Ampere architecture. What architecture/GPU are you working with? Or for other Nvidia GPU environments you have access to?

Tested on a server T4 it seems to be ok; A3000 is the environment of my development computer.

yuweiw · July 15, 2022, 9:20am

I have tried your code in diffrent GPUs or jetson platforms, it have no problem either.
So what I mean is change the quality to 90 , 95, or other value. Also you can change other parameters like saveImg and so on. You can try to confirm if there is a group parameters that is ok in your env. Thanks

syther666 · July 15, 2022, 9:29am

Have you tried it on a A3000 GPU? Wouldn’t be effective if you aren’t able to reproduce the issue on your side.