Image Distortion and Inference Issues in Deepstream 6 Pipeline with YOLO v4 and UNet

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

**• Jetson Xaiver NX **
• DeepStream Version 6.2
**• JetPack Version 5.1 **
**• TensorRT Version 8.5.2.2 **
**• Image Distortion and Inference Issues **

I am writing to report two issues encountered while running a Deepstream 6.2 pipeline with YOLO v4 and UNet inferences on JP 5.1. Previously, this pipeline functioned correctly on JP 4.6.

Problem 1: Image Distortion

We are observing image distortion in the lower portion of the image when viewing the results at the probe. This distortion is consistently present regardless of the frame rate (1, 5, 15, or 30 fps). We suspect the distortion might be caused by remnants from the previous frame.

An example image showcasing the distortion is attached to this message.

05_05_2024___12_50_08_103_rgb

crop_half
crop_half900×753 110 KB

Problem 2: Inference Discrepancy with File Input

When saving the data and reading images from a file using cv2.imread, YOLO detections are no longer present, even though the same images yielded detections in real-time. However, UNet inference remains functional.

Furthermore, the following observations were made:

  • Replacing RGB2BGR in the pipeline results in no detections for both YOLO and UNet.
  • Reading images with cv2.imread(IMREAD_UNCHANGED) also leads to the absence of YOLO detections in the same pipeline.

It is important to note that the source remains appsrc receiving a numpy array in both scenarios.

We would greatly appreciate any insights or suggestions on troubleshooting these issues and achieving the desired results in our Deepstream pipeline.

Thank you for your time and assistance.

Sincerely,
Daphna, Nanovel

3

As to your description, you are using your own customized application. We don’t know how did you use the DeepStream or Jetson Multimedia components. Can you share your application or some simplified code which can reproduce the issue? The configuration files are also important.

Can you try with the latest JetPack 6.0 version?

4

Hi, thanks for your reply. I’ve attached some code samples and configuration files that might help diagnose the issue. Unfortunately, we’re unable to test with JetPack 6.0 as we’re using a Jetson Xavier NX, which isn’t currently supported.

Thank you for your time and assistance.
Sincerely,
Daphna, Nanovel

create_pipe_and_plugin.txt (11.7 KB)
inject_data.txt (4.6 KB)
pipe_wrapper.txt (7.3 KB)
config_yolo_tao_april.txt (2.5 KB)
config_unet_tao_glare.txt (3.8 KB)

5

  1. Can you confirm with Intel whether your RealSense camera can be used with the realsense gstreamer plugin? Intel RS D415 Camera access with gStreamer in windows – Intel RealSense Help Center

  2. Why do you use “cv2.cvtColor(color_image, cv2.COLOR_RGB2BGR)” in the “read_image_from_file_and_inject_into_pipeline(self)” function? Do you want to convert the image data to BGR format? Seems your pipeline needs “RGB” data as you set “video/x-raw,format=RGB,width=640,height=480,framerate=30/1” caps after appsrc in your create_deepstream_pipeline_with_tee() function.

For problem 2, I don’t find any issue with DeepStream. Please google opencv and GStreamer document s and references by yourself.

6

For the distortion issue, have you enabled max power mode with your board?

Is there hardware loading log for your board when you run the pipeline? You can use “tegrastats” to monitor the hardware loading when running your cases.

7

Thanks for your response and insights! Let’s address each point:

RealSense GStreamer Plugin:

The RealSense GStreamer plugin link wasn’t relevant to our appsrc setup. We’re directly feeding numpy arrays from the camera’s color image.

Color Format (BGR vs. RGB):

I appreciate the clarification. We confirmed our models require BGR format, so we adjusted the pipeline accordingly. Unfortunately, switching from RGB to BGR within the pipeline didn’t resolve the missing YOLO detections on file input.

Problem 1: Image Distortion

Max Power Mode: Yes, we’re running on a 20W 6 Core board, which is the maximum power configuration.
Tegrastats Log: Thanks for reviewing the log. We noticed a warning about “system throttled due to Over-current” which might be related. Additionally, the screen recording showcases the distortion on the UNet mask output – the object duplication seems to support our theory of “remnants” from previous frames.
tegrastats_3.txt (13.0 KB)

Problem 2: YOLO Detection Discrepancy

We observed that removing the tracker from the pipeline allows YOLO detections when using file input. However, this also worsens the distortion, with almost every frame from the file being severely damaged and showing prominent straight lines. This suggests the tracker might be interacting with the distortion issue in some way.

Additional Information:

Here’s the screen recording showcasing the UNet mask distortion for your reference

  • We’d appreciate any suggestions on how to mitigate the “over-current” issue and potential impact on distortion.
  • Considering the tracker’s influence on distortion and detections, could there be a configuration or interaction causing unexpected behavior?

Thanks again for your continued support!

Sincerely,
Daphna

8

How did you switch from RGB to BGR within the pipeline?

This is Jetson issue, please raise topic in Jetson forum. Latest Autonomous Machines/Jetson & Embedded Systems topics - NVIDIA Developer Forums

Seems sometimes the GPU loading is very high with the tegrastats log you provided.

05-12-2024 10:53:00 RAM 4772/6849MB (lfb 4x1MB) SWAP 112/3425MB (cached 0MB) CPU [15%@1420,5%@1420,18%@1420,28%@1420,14%@1420,15%@1420] EMC_FREQ 0% GR3D_FREQ 99% AUX@50.5C CPU@51.5C thermal@51.55C AO@48.5C GPU@54.5C PMIC@50C

Can you measure the performance of your YOLOV and UNET model with trtexec tool?

9

Hi Fiona,

Thanks for suggesting the trtexec tool! I’m trying it out to measure the performance of my YOLOv and UNET models.

However, trtexec typically works with models in Caffe, UFF, or ONNX format. Since my models are currently in .etlt format and converted to engine files using tao-converter, the tool doesn’t recognize them directly.

Do you have any recommendations or guidance on how to use trtexec for performance measurement with models converted from etltusing tao-converter? Any pointers to relevant documentation or examples would be greatly appreciated.

This way, I can effectively leverage trtexec for my performance analysis.

Thanks!

10

You can use tao-converter to generate the TensorRT engine file, the trtexec can measure the TensorRT engine file peformance, the DeepStream also runs with TensorRT engine file, you can also get the DeepStream generated tensorRT engine file for performance measurement.

For more details about etlt performance measurement, please raise topic in TAO forum. Latest Intelligent Video Analytics/TAO Toolkit topics - NVIDIA Developer Forums

11

trtexec_test.txt (5.0 KB)

Hi Fiona,
please see the trtexec results after running the command:
./trtexec --onnx=/home/mark/grc_robot_parameters/models/yolo_v4/tao/tao_april_2023/yolov4_resnet18_epoch_090.etlt_b1_gpu0_fp16.engine

12

In the log:

Could not open file /home/mark/grc_robot_parameters/models/yolo_v4/tao/tao_april_2023/yolov4_resnet18_epoch_090.etlt_b1_gpu0_fp16.engine
Could not open file /home/mark/grc_robot_parameters/models/yolo_v4/tao/tao_april_2023/yolov4_resnet18_epoch_090.etlt_b1_gpu0_fp16.engine
[05/22/2024-11:11:37] [E] [TRT] ModelImporter.cpp:688: Failed to parse ONNX model from file: /home/mark/grc_robot_parameters/models/yolo_v4/tao/tao_april_2023/yolov4_resnet18_epoch_090.etlt_b1_gpu0_fp16.engine
[05/22/2024-11:11:37] [E] Failed to parse onnx file

It failed. The engine file is not onnx file. Please use “–loadEngine=” option.

For more details about etlt performance measurement, please raise topic in TAO forum. Latest Intelligent Video Analytics/TAO Toolkit topics - NVIDIA Developer Forums

13

trtexec_yolo.txt (4.0 KB)

./trtexec --loadEngine=/home/mark/grc_robot_parameters/models/yolo_v4/tao/tao_april_2023/yolov4_resnet18_epoch_090.etlt_b1_gpu0_fp16.engine >> /home/mark/trtexec_yolo.txt
[05/22/2024-15:43:14] [E] Error opening engine file: /home/mark/grc_robot_parameters/models/yolo_v4/tao/tao_april_2023/yolov4_resnet18_epoch_090.etlt_b1_gpu0_fp16.engine
[05/22/2024-15:43:14] [E] Failed to create engine from model or file.
[05/22/2024-15:43:14] [E] Engine set up failed
14

Does the engine file exist?

15

yolo results:

[05/22/2024-15:57:31] [W] [TRT] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[05/22/2024-15:57:37] [W] * GPU compute time is unstable, with coefficient of variance = 30.0907%.
[05/22/2024-15:57:37] [W]   If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.

trtexec_yolo_2.txt (9.3 KB)

unet results:


[05/22/2024-16:00:22] [W] [TRT] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[05/22/2024-16:00:26] [W] * GPU compute time is unstable, with coefficient of variance = 86.0385%.
[05/22/2024-16:00:26] [W]   If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.

trtexec_unet.txt (7.5 KB)

16

The performance of unet shows the mean compute time of a batch is more than 32ms, the max compute time is 240+ms.

Even the yolo modle mean time is 32ms, but the max time is 86~ms.

That means the GPU is overloaded when you run the two models with target 30fps.

17

Hi Fiona,
Thanks for analyzing the performance results of my Unet and Yolo models.
1. Image Distortion:
Can this high inference time explain the distortion we’re observing in the image pipeline? I’m wondering if the dropped frames or delayed processing could be causing the visual artifacts.
2. Performance Improvement:
I’d appreciate your advice on improving the performance. Since Unet seems to be the bottleneck, would retraining the model be a good approach? Are there other optimization techniques you’d recommend for running both models on the Jetson at a smooth 30fps/15fps?

18

We don’t know. From your previous description, after removing tracker, the distortion will be more heavy. That is why I asked you to measure the GPU and CPU loading. It may be related, but we are not sure.

Can you replace the Yolo and unet models with other smaller models?

Or you can refer to the TAO unet model. You may raise topic for how to optimize model with TAO toolkit in the TAO forum. Latest Intelligent Video Analytics/TAO Toolkit topics - NVIDIA Developer Forums

19

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

20

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.