Issue while Feeding Pytorch TRT 2D-U-net model into Deepstream 6 having same TRT version

nitinp14920914 · July 7, 2022, 9:12am

Hello,

We have trained our Pytorch Unet model and converted that into TRT using the torch2trt but while feeding that into the Deepstream- 6 we are facing the errors logged below we have converted our torch2trt model on the same machine having deepstream having similar TRT version

(pytorch) nkumar@u117282:~/deepstream_trt$ deepstream-segmentation-app -t infer deepstream_tao_apps/configs/unet_tao/pgie_unet_tao_config.txt deepstream_tao_apps/***********67.11704647123835-81.03094573969791.h264 

(deepstream-segmentation-app:4023651): GStreamer-WARNING **: 10:20:01.186: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.

(deepstream-segmentation-app:4023651): GStreamer-WARNING **: 10:20:01.186: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

(deepstream-segmentation-app:4023651): GStreamer-WARNING **: 10:20:01.187: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvblender.so': libgstbadvideo-1.0.so.0: cannot open shared object file: No such file or directory

(deepstream-segmentation-app:4023651): GStreamer-WARNING **: 10:20:01.187: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
One element could not be created. Exiting.
Failed to create source bin. Exiting.

My configuration file looks like :

[property]

labelfile-path=unet_labels.txt
model-engine-file=../../models/unet/**_**_ICG_100_2switched.trt_ew
#CD_CD_ICG_100_swiched.trt
#CA_CD_ICGGrayscale.trt
#engine-create-func-name=NvDsInferYoloCudaEngineGet


#tlt-encoded-model=../../models/unet/CA_CD.etlt

net-scale-factor=0.00784313725490196
offsets=127.5;127.5;127.5
infer-dims=3;512;512
#tlt-model-key=CA_CD_ICG
network-type=2
num-detected-classes=3
model-color-format=1
segmentation-threshold=0.0
output-blob-names=softmax_1
segmentation-output-order=1
gie-unique-id=1

Environment

GPU Type: NVIDIA RTX A6000
Nvidia Driver Version: 510.60.02
Operating System : Ubuntu 21:
Python Version : 3.7

deepstream-app version 6.0.1, 
DeepStreamSDK 6.0.1, 
CUDA Driver Version: 11.6, 
CUDA Runtime Version: 11.4, 
TensorRT Version: 8.4, 
cuDNN Version: 8.4, 
libNVWarp360 Version: 2.0.1d3

NVES · July 7, 2022, 9:38am

Hi,

This looks like a Deepstream related issue. We will move this post to the Deepstream forum.

Thanks!

Amycao · July 8, 2022, 8:41am

Ubuntu 18.04
GStreamer 1.14.5
NVIDIA driver 470.63.01
CUDA 11.4
TensorRT 8.0.1

yingliu · July 26, 2022, 7:25am

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

nitinp14920914 · August 1, 2022, 2:31pm

Still waiting for a valuable response

yingliu · August 2, 2022, 1:34am

Sorry for the confusion, our team will do the investigation and provide suggestions soon. Thanks

mchi · August 2, 2022, 1:48am

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi @nitinp14920914
Can you try this in DeepStream triton docker - nvcr.io/nvidia/deepstream:6.1-triton ?
If you are already using nvcr.io/nvidia/deepstream:6.1-triton, please run below command to check if all the dependent libraries of libnvdsgst_inferserver.so all exist,

$ ldd /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so

nitinp14920914 · August 29, 2022, 12:38pm

Here’s my output for the Command mentioned i think i see all the libraries.

(base) nkxmxxar@u1172xxx:~/deep$ ldd /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so
	linux-vdso.so.1 (0x00007ffc549e3000)
	libcudart.so.11.0 => /usr/local/cuda-11.4/lib64/libcudart.so.11.0 (0x00007fe357c53000)
	libnvds_infer_server.so => ///opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_infer_server.so (0x00007fe356a7f000)
	libnvds_meta.so => ///opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_meta.so (0x00007fe356875000)
	libnvdsgst_meta.so => ///opt/nvidia/deepstream/deepstream-6.0/lib/libnvdsgst_meta.so (0x00007fe356670000)
	libnvdsgst_helper.so => ///opt/nvidia/deepstream/deepstream-6.0/lib/libnvdsgst_helper.so (0x00007fe356469000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe35643d000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe35641a000)
	libgstreamer-1.0.so.0 => /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (0x00007fe3562d3000)
	libgstbase-1.0.so.0 => /lib/x86_64-linux-gnu/libgstbase-1.0.so.0 (0x00007fe356255000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fe35612c000)
	libgobject-2.0.so.0 => /lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007fe3560cc000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe355ee8000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe355ecd000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe355cdb000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe358200000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe355cd1000)
	libtritonserver.so => not found
	libnvbufsurftransform.so => ///opt/nvidia/deepstream/deepstream-6.0/lib/libnvbufsurftransform.so (0x00007fe352291000)
	libnvbufsurface.so => ///opt/nvidia/deepstream/deepstream-6.0/lib/libnvbufsurface.so (0x00007fe352088000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe35206c000)
	libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fe351d96000)
	libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fe351d03000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe351bb4000)
	libgstrtp-1.0.so.0 => /lib/x86_64-linux-gnu/libgstrtp-1.0.so.0 (0x00007fe351b8c000)
	libgmodule-2.0.so.0 => /lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007fe351b86000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fe351b13000)
	libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007fe351b07000)
	libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1 (0x00007fe3502c4000)
	libnppc.so.11 => /usr/local/cuda-11.4/lib64/libnppc.so.11 (0x00007fe34ff3d000)
	libnppig.so.11 => /usr/local/cuda-11.4/lib64/libnppig.so.11 (0x00007fe34de8f000)
	libnppidei.so.11 => /usr/local/cuda-11.4/lib64/libnppidei.so.11 (0x00007fe34d336000)
	libnvbuf_fdmap.so => ///opt/nvidia/deepstream/deepstream-6.0/lib/libnvbuf_fdmap.so (0x00007fe34d128000)

mchi · August 30, 2022, 4:52am

this lib is missed

nitinp14920914 · September 8, 2022, 11:17am

Is this one for Jetson platfom or DGPU ?

mchi · September 8, 2022, 4:09pm

both platform have this lib for nvinferserver/Triton

nitinp14920914 · September 9, 2022, 4:20pm

Can you pl help me with any reference Installation Guide for same without using Container/Docker on the Host OS Ubuntu20/18.04 running platform having Dgpu Dual A6000

nitinp14920914 · September 12, 2022, 9:44am

Waiting for Response