Image Distortion and Inference Issues in Deepstream 6 Pipeline with YOLO v4 and UNet

Please provide complete information as applicable to your setup.

**• Jetson Xaiver NX **
• DeepStream Version 6.2
**• JetPack Version 5.1 **
**• TensorRT Version 8.5.2.2 **
**• Image Distortion and Inference Issues **

I am writing to report two issues encountered while running a Deepstream 6.2 pipeline with YOLO v4 and UNet inferences on JP 5.1. Previously, this pipeline functioned correctly on JP 4.6.

Problem 1: Image Distortion

We are observing image distortion in the lower portion of the image when viewing the results at the probe. This distortion is consistently present regardless of the frame rate (1, 5, 15, or 30 fps). We suspect the distortion might be caused by remnants from the previous frame.

An example image showcasing the distortion is attached to this message.

05_05_2024___12_50_08_103_rgb

Problem 2: Inference Discrepancy with File Input

When saving the data and reading images from a file using cv2.imread, YOLO detections are no longer present, even though the same images yielded detections in real-time. However, UNet inference remains functional.

Furthermore, the following observations were made:

  • Replacing RGB2BGR in the pipeline results in no detections for both YOLO and UNet.
  • Reading images with cv2.imread(IMREAD_UNCHANGED) also leads to the absence of YOLO detections in the same pipeline.

It is important to note that the source remains appsrc receiving a numpy array in both scenarios.

We would greatly appreciate any insights or suggestions on troubleshooting these issues and achieving the desired results in our Deepstream pipeline.

Thank you for your time and assistance.

Sincerely,
Daphna, Nanovel

As to your description, you are using your own customized application. We don’t know how did you use the DeepStream or Jetson Multimedia components. Can you share your application or some simplified code which can reproduce the issue? The configuration files are also important.

Can you try with the latest JetPack 6.0 version?

Hi, thanks for your reply. I’ve attached some code samples and configuration files that might help diagnose the issue. Unfortunately, we’re unable to test with JetPack 6.0 as we’re using a Jetson Xavier NX, which isn’t currently supported.

Thank you for your time and assistance.
Sincerely,
Daphna, Nanovel

create_pipe_and_plugin.txt (11.7 KB)
inject_data.txt (4.6 KB)
pipe_wrapper.txt (7.3 KB)
config_yolo_tao_april.txt (2.5 KB)
config_unet_tao_glare.txt (3.8 KB)

  1. Can you confirm with Intel whether your RealSense camera can be used with the realsense gstreamer plugin? Intel RS D415 Camera access with gStreamer in windows – Intel RealSense Help Center

  2. Why do you use “cv2.cvtColor(color_image, cv2.COLOR_RGB2BGR)” in the “read_image_from_file_and_inject_into_pipeline(self)” function? Do you want to convert the image data to BGR format? Seems your pipeline needs “RGB” data as you set “video/x-raw,format=RGB,width=640,height=480,framerate=30/1” caps after appsrc in your create_deepstream_pipeline_with_tee() function.

For problem 2, I don’t find any issue with DeepStream. Please google opencv and GStreamer document s and references by yourself.

For the distortion issue, have you enabled max power mode with your board?

Is there hardware loading log for your board when you run the pipeline? You can use “tegrastats” to monitor the hardware loading when running your cases.

Thanks for your response and insights! Let’s address each point:

RealSense GStreamer Plugin:

The RealSense GStreamer plugin link wasn’t relevant to our appsrc setup. We’re directly feeding numpy arrays from the camera’s color image.

Color Format (BGR vs. RGB):

I appreciate the clarification. We confirmed our models require BGR format, so we adjusted the pipeline accordingly. Unfortunately, switching from RGB to BGR within the pipeline didn’t resolve the missing YOLO detections on file input.

Problem 1: Image Distortion

Max Power Mode: Yes, we’re running on a 20W 6 Core board, which is the maximum power configuration.
Tegrastats Log: Thanks for reviewing the log. We noticed a warning about “system throttled due to Over-current” which might be related. Additionally, the screen recording showcases the distortion on the UNet mask output – the object duplication seems to support our theory of “remnants” from previous frames.
tegrastats_3.txt (13.0 KB)

Problem 2: YOLO Detection Discrepancy

We observed that removing the tracker from the pipeline allows YOLO detections when using file input. However, this also worsens the distortion, with almost every frame from the file being severely damaged and showing prominent straight lines. This suggests the tracker might be interacting with the distortion issue in some way.

Additional Information:

Here’s the screen recording showcasing the UNet mask distortion for your reference

  • We’d appreciate any suggestions on how to mitigate the “over-current” issue and potential impact on distortion.
  • Considering the tracker’s influence on distortion and detections, could there be a configuration or interaction causing unexpected behavior?

Thanks again for your continued support!

Sincerely,
Daphna

How did you switch from RGB to BGR within the pipeline?

This is Jetson issue, please raise topic in Jetson forum. Latest Autonomous Machines/Jetson & Embedded Systems topics - NVIDIA Developer Forums

Seems sometimes the GPU loading is very high with the tegrastats log you provided.

05-12-2024 10:53:00 RAM 4772/6849MB (lfb 4x1MB) SWAP 112/3425MB (cached 0MB) CPU [15%@1420,5%@1420,18%@1420,28%@1420,14%@1420,15%@1420] EMC_FREQ 0% GR3D_FREQ 99% AUX@50.5C CPU@51.5C thermal@51.55C AO@48.5C GPU@54.5C PMIC@50C

Can you measure the performance of your YOLOV and UNET model with trtexec tool?

Hi Fiona,

Thanks for suggesting the trtexec tool! I’m trying it out to measure the performance of my YOLOv and UNET models.

However, trtexec typically works with models in Caffe, UFF, or ONNX format. Since my models are currently in .etlt format and converted to engine files using tao-converter, the tool doesn’t recognize them directly.

Do you have any recommendations or guidance on how to use trtexec for performance measurement with models converted from etltusing tao-converter? Any pointers to relevant documentation or examples would be greatly appreciated.

This way, I can effectively leverage trtexec for my performance analysis.

Thanks!

You can use tao-converter to generate the TensorRT engine file, the trtexec can measure the TensorRT engine file peformance, the DeepStream also runs with TensorRT engine file, you can also get the DeepStream generated tensorRT engine file for performance measurement.

For more details about etlt performance measurement, please raise topic in TAO forum. Latest Intelligent Video Analytics/TAO Toolkit topics - NVIDIA Developer Forums