Different segmentation mask between Jetson Nano and Desktop GPU

Hello, I’m building an ADAS system with Jetson Nano and DeepStream. I have a lane segmentation model based on Unet. Everything runs fine but I get different segmentation masks when running on Jetson Nano and my desktop system GPU (RTX 2060). The mask generated by Jetson Nano is considerably better than the mask my desktop GPU. I would like to know why, or if is there anything that I can do to make inference on my desktop GPU better. They are using the same code, same config files and same model which makes me wonder why this happens.
Here on the left is the mask generated by Jetson Nano, on the right the mask generated by RTX 2060 and the original frame on the bottom.


As can be seen the mask generated by the same model on both systems has a lot better output in Jetson Nano. Is there any reason why this happens ? Or anything that I can do to improve the mask on my desktop GPU ?

Thanks a lot in advance!

A quick idea is, maybe it’s related to the scaling in nvinfer. you could try to set the “scaling-compute-hw” to 1 (GPU) on NANO to check the result.

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html

Hello @mchi , thank you for the reply. I changed the setting you mentioned but the output seems to be the same. But what you said makes a lot of sense it seems like a scale problem, because looking at more mask samples, it seems like only the right lane is being computed in the mask, with the left one being omitted. So do you have any other ideas that might help?

Anyone can help with this ?
I’ve been trying different configs accordingly to the documentation (Gst-nvinfer — DeepStream 5.1 Release documentation), but nothing seems to work.
This is the config I’m using:



################################################################################
# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

# Following properties are mandatory when engine files are not specified:
#   int8-calib-file(Only in INT8), model-file-format
#   Caffemodel mandatory properties: model-file, proto-file, output-blob-names
#   UFF: uff-file, input-dims, uff-input-blob-name, output-blob-names
#   ONNX: onnx-file
#
# Mandatory properties for detectors:
#   num-detected-classes
#
# Optional properties for detectors:
#   cluster-mode(Default=Group Rectangles), interval(Primary mode only, Default=0)
#   custom-lib-path,
#   parse-bbox-func-name
#
# Mandatory properties for classifiers:
#   classifier-threshold, is-classifier
#
# Optional properties for classifiers:
#   classifier-async-mode(Secondary mode only, Default=false)
#
# Optional properties in secondary mode:
#   operate-on-gie-id(Default=0), operate-on-class-ids(Defaults to all classes),
#   input-object-min-width, input-object-min-height, input-object-max-width,
#   input-object-max-height
#
# Following properties are always recommended:
#   batch-size(Default=1)
#
# Other optional properties:
#   net-scale-factor(Default=1), network-mode(Default=0 i.e FP32),
#   model-color-format(Default=0 i.e. RGB) model-engine-file, labelfile-path,
#   mean-file, gie-unique-id(Default=0), offsets, process-mode (Default=1 i.e. primary),
#   custom-lib-path, network-mode(Default=0 i.e FP32)
#
# The values in the config file are overridden by values set through GObject
# properties.

[property]
gpu-id=0
net-scale-factor=0.8
model-color-format=0
uff-file=../models/lane_segmentationLowerLoss-resnet34_384x384.uff
model-engine-file=../models/lane_segmentationLowerLoss-resnet34_384x384.uff_b1_gpu0_fp16.engine
infer-dims=3;384;384
uff-input-order=0
uff-input-blob-name=data
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=1
interval=0
gie-unique-id=1
network-type=2
output-blob-names=final_conv/BiasAdd
# output-blob-names=sigmoid/Sigmoid
segmentation-threshold=0.0
# scaling-filter=1
scaling-compute-hw=1

Hi @fredericolms ,
Sorry for delay!
Is it possible for you to apply the change in " 2. [DS5.0GA_Jetson_GPU_Plugin] Dump the Inference Input" to check the inputs of NANO and dGPU respectively?

Thanks!

Hello, @mchi I rebuilt my libnvds_infer.so with the code changes you mentioned, but now when I run deepstream applications nvinfer plugin fails to be added to GST pipeline. Do you have any idea where I could have made a mistake?

Thanks in advance!

I actually just used the change again on DS5.1 last week, it should work.
What error do you see?

I tested on my application and on deepstream-test2 and the same error happens. The error output is as follows:



(deepstream_test2:13994): GStreamer-WARNING **: 14:41:34.683: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so': libopencv_objdetect.so.4.4: cannot open shared object file: No such file or directory

(deepstream_test2:13994): GStreamer-WARNING **: 14:41:34.705: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so': libopencv_objdetect.so.4.4: cannot open shared object file: No such file or directory

(deepstream_test2:13994): GStreamer-WARNING **: 14:41:34.706: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so': libopencv_objdetect.so.4.4: cannot open shared object file: No such file or directory

(deepstream_test2:13994): GStreamer-WARNING **: 14:41:34.707: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so': libopencv_objdetect.so.4.4: cannot open shared object file: No such file or directory
One element could not be created. Exiting.


I was able to get rid of those gst-warnings by changing the makefile of gst-nvinfer plugin I only added the location of where my OpenCV is installed (I had to do the same to compile the libnvds_infer.so correctly), but this error happens anyway. It seems like it cant create the plugin on the pipeline.

Can you share your setup info as other topics, like below? This helps to understand the backgound of the issue

Hardware Platform (Jetson / GPU) - GPU GT 1030 and OS Ubuntu-18.04.4 LTS

DeepStream Version - DeepStream 5.0

TensorRT Version - 7.2.2.3

CUDA - 11.2

NVIDIA GPU Driver Version (valid for GPU only) - 460.32.03

Hardware Platform (Jetson / GPU) - GPU RTX 2060 and OS Ubuntu-18.04.4 LTS

DeepStream Version - DeepStream 5.0

TensorRT Version - 7.0.0.1

CUDA - 10.2

NVIDIA GPU Driver Version (valid for GPU only) - 450.57

Did you install opecv4 in your setup?
The opencv in DS5.0 is OpenCV3.2.

root@b5d1dea9544a:/opt/nvidia/deepstream/deepstream-5.0# dpkg -l | grep libopencv
ii libopencv-calib3d-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-calib3d3.2
ii libopencv-calib3d3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Camera Calibration library
ii libopencv-contrib-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-contrib3.2
ii libopencv-contrib3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision contrlib library
ii libopencv-core-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-core3.2
ii libopencv-core3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision core library
ii libopencv-dev 3.2.0+dfsg-4ubuntu0.1 amd64 development files for opencv
ii libopencv-features2d-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-features2d3.2
ii libopencv-features2d3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Feature Detection and Descriptor Extraction library
ii libopencv-flann-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-flann3.2
ii libopencv-flann3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Clustering and Search in Multi-Dimensional spaces library
ii libopencv-highgui-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-highgui3.2
ii libopencv-highgui3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision High-level GUI and Media I/O library
ii libopencv-imgcodecs-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-imgcodecs3.2
ii libopencv-imgcodecs3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Image Codecs library
ii libopencv-imgproc-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-imgproc3.2
ii libopencv-imgproc3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Image Processing library
ii libopencv-ml-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-ml3.2
ii libopencv-ml3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Machine Learning library
ii libopencv-objdetect-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-objdetect3.2
ii libopencv-objdetect3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Object Detection library
ii libopencv-photo-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-photo3.2
ii libopencv-photo3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision computational photography library
ii libopencv-shape-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-shape3.2
ii libopencv-shape3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision shape descriptors and matchers library
ii libopencv-stitching-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-stitching3.2
ii libopencv-stitching3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision image stitching library
ii libopencv-superres-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-superres3.2
ii libopencv-superres3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Super Resolution library
ii libopencv-ts-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for TS library of OpenCV (Open Computer Vision)
ii libopencv-video-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-video3.2
ii libopencv-video3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Video analysis library
ii libopencv-videoio-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-videoio3.2
ii libopencv-videoio3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision Video I/O library
ii libopencv-videostab-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-videostab3.2
ii libopencv-videostab3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision video stabilization library
ii libopencv-viz-dev:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 development files for libopencv-viz3.2
ii libopencv-viz3.2:amd64 3.2.0+dfsg-4ubuntu0.1 amd64 computer vision 3D data visualization library
ii libopencv3.2-java 3.2.0+dfsg-4ubuntu0.1 all Java bindings for the computer vision library
ii libopencv3.2-jni 3.2.0+dfsg-4ubuntu0.1 amd64 Java jni library for the computer vision library

Sorry, I didn’t know that DS OpenCV version was 3.2, I had OpenCV 4 installed in my system. But ok, now I got libnvds_infer.so running with the changes you mentioned. Where do I find the inference dumps to show you? I can see some prints in console now, is there anything else?

when you run your application, it should be able to dump some jpg images. did you see them?

Yes, I can see them, they’re the following images:
gie-1_input-0_batch-0_frame-11

gie-1_input-0_batch-0_frame-12

gie-2_input-0_batch-0_frame-11

gie-2_input-0_batch-0_frame-12

And these are the outputs in console:



batchDims.batchSize = 1, scale = 0.003922, format = 0
batchDims.batchSize = 1, scale = 0.003922, format = 0
batchDims.batchSize = 1, scale = 0.800000, format = 0
batchDims.batchSize = 1, scale = 0.800000, format = 0
batchDims.batchSize = 1, scale = 1.000000, format = 1
batchDims.batchSize = 1, scale = 1.000000, format = 1


Ok, could you compare the images between NANO and dGPU to check if the image quality is the same?

The images from Jetson Nano look the same, but those two last images are not saved. The output on Jetson, follows:

gie-1_input-0_batch-0_frame-11 gie-1_input-0_batch-0_frame-12

And the console output:



batchDims.batchSize = 1, scale = 0.003922, format = 0
batchDims.batchSize = 1, scale = 0.003922, format = 0
batchDims.batchSize = 1, scale = 1.000000, format = 0
batchDims.batchSize = 1, scale = 1.000000, format = 0


did you compare the image b/w NANO and dGPU?

Well, I posted here the images for comparison…

These are the imagens generated on the dGPU system with an RTX 2060:

And here it is the Jetson Nano images:

Both are looking the same to me… so I don’t know where is the problem.

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi @fredericolms ,
Sorry for delay!

I saw the value of “scale =” are different on NANO and RTX gpu, do you know why ?
I saw it’s 0.8 in your config file, but it changed in the log