Deepstream_lpr_app for DS6 - libcufft.so.10: cannot open shared object

user95782 · May 10, 2022, 7:42pm

Please provide the following information when requesting support.

• Hardware (T4)
• Network Type (deepstream_lpr_app sample)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

attcloud@deepstream-dev-02-vm:~/deepstream_lpr_app/deepstream-lpr-app$ sudo ./deepstream-lpr-app 1 2 0 Driving.mp4 output.264
Request sink_0 pad from streammux
Warning: ‘input-dims’ parameter has been deprecated. Use ‘infer-dims’ instead.
Warning: ‘input-dims’ parameter has been deprecated. Use ‘infer-dims’ instead.
Now playing: 1
WARNING: [TRT]: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.3.0
WARNING: [TRT]: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.3.0
0:00:01.338099280 15393 0x55b68b87e0c0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 3]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 3]: deserialized trt engine from :/home/attcloud/deepstream_lpr_app/models/LP/LPR/us_lprnet_baseline18_deployable.etlt_b16_gpu0_fp16.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [FullDims Engine Info]: layers num: 3
0 INPUT kFLOAT image_input 3x48x96 min: 1x3x48x96 opt: 4x3x48x96 Max: 16x3x48x96
1 OUTPUT kINT32 tf_op_layer_ArgMax 24 min: 0 opt: 0 Max: 0
2 OUTPUT kFLOAT tf_op_layer_Max 24 min: 0 opt: 0 Max: 0

ERROR: [TRT]: 3: Cannot find binding of given name: output_bbox/BiasAdd
0:00:01.338219042 15393 0x55b68b87e0c0 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger: NvDsInferContext[UID 3]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1868> [UID = 3]: Could not find output layer ‘output_bbox/BiasAdd’ in engine
ERROR: [TRT]: 3: Cannot find binding of given name: output_cov/Sigmoid
0:00:01.338237647 15393 0x55b68b87e0c0 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger: NvDsInferContext[UID 3]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1868> [UID = 3]: Could not find output layer ‘output_cov/Sigmoid’ in engine
0:00:01.338248488 15393 0x55b68b87e0c0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 3]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 3]: Use deserialized engine model: /home/attcloud/deepstream_lpr_app/models/LP/LPR/us_lprnet_baseline18_deployable.etlt_b16_gpu0_fp16.engine
0:00:01.339328939 15393 0x55b68b87e0c0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus: [UID 3]: Load new model:lpr_config_sgie_us.txt sucessfully
WARNING: [TRT]: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.3.0
WARNING: [TRT]: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 11.3.0
0:00:01.355299185 15393 0x55b68b87e0c0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 2]: deserialized trt engine from :/home/attcloud/deepstream_lpr_app/models/LP/LPD/usa_pruned.etlt_b16_gpu0_int8.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x480x640
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x30x40
2 OUTPUT kFLOAT output_cov/Sigmoid 1x30x40

0:00:01.355360764 15393 0x55b68b87e0c0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger: NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 2]: Use deserialized engine model: /home/attcloud/deepstream_lpr_app/models/LP/LPD/usa_pruned.etlt_b16_gpu0_int8.engine
0:00:01.356877468 15393 0x55b68b87e0c0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus: [UID 2]: Load new model:lpd_us_config.txt sucessfully
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Failed to open low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
dlopen error: libcufft.so.10: cannot open shared object file: No such file or directory
gstnvtracker: Failed to initilaize low level lib.
Running…

Morganh · May 11, 2022, 6:53am

Could you double check the CUDA installation? Thanks.
For example, can you find any file as below?
$ sudo find /usr/local -name libcufft.so*

Maybe something mismatching on your machine.

user95782 · May 11, 2022, 7:17am

Hi Morganh,

The result came back empty… Is this mean the Cuda installation was bad?

attcloud@deepstream-dev-02-vm:~/deepstream_lpr_app/deepstream-lpr-app$ sudo find /usr/local -name libcufft.so*
attcloud@deepstream-dev-02-vm:~/deepstream_lpr_app/deepstream-lpr-app$

thanks,
Chris

Morganh · May 11, 2022, 7:25am

At least libcufft.so* should be found. Suggest to check CUDA installation.

user95782 · May 11, 2022, 7:30am

Are you suggesting to reinstall these 2 sections of the install guide? NVIDIA DeepStream SDK Developer Guide — DeepStream 6.1.1 Release documentation

Install NVIDIA driver 470.63.01

Install CUDA Toolkit 11.4.1 (CUDA 11.4 Update 1)

Morganh · May 11, 2022, 7:36am

Could you share the link for above?

user95782 · May 11, 2022, 7:42am

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Quickstart.html#install-nvidia-driver-470-63-01
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Quickstart.html#install-cuda-toolkit-11-4-1-cuda-11-4-update-1

basically, I would like to know which section of the installation guide I should redo.

I am using T4 VM:

$ cat /etc/os-release
NAME=“Ubuntu”
VERSION=“18.04.6 LTS (Bionic Beaver)”

$ nvidia-smi
CUDA Version: 11.4

$ dpkg -l | grep TensorRT
libnvinfer-bin 8.0.1-1+cuda11.3 amd64 TensorRT binaries

Morganh · May 11, 2022, 7:50am

You can redo “Install CUDA Toolkit 11.4.1 (CUDA 11.4 Update 1)” . From nvidia-smi, you can check if “NVIDIA driver 470.63.01” is available.

user95782 · May 11, 2022, 7:54am

I got Driver Version: 470.103.01

±----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |

user95782 · May 11, 2022, 8:13am

I reran the following instruction; But still not resolved.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pinsudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pubsudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"sudo apt-get updatesudo apt-get -y install cuda

user95782 · May 11, 2022, 8:14am

https://developer.nvidia.com/cuda-11-4-1-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=18.04&target_type=deb_network

user95782 · May 11, 2022, 8:17am

I got a lot of error messages during the install like:

W: GPG error: Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository ‘Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease’ is no longer signed.
N:

Err:11 Index of /compute/cuda/repos/ubuntu1804/x86_64 InRelease
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
Reading package lists… Done

Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/./cuda_11.6.2-1_amd64.deb Hash Sum mismatch
Hashes of expected file:
- SHA512:1ba4915fa40fdc21a5bdbadcf1abd0cb9df5ce7b40b5654c2b9690e688761160706cad46f2fbbd1b31fd083eeaee61099a9aa41f8279bccc9dc95e6617b616ad
- SHA256:5215130d80e0002c08d2daaf6de53c345132ecdb1af493dc51373b6c92808731
- SHA1:46661f399bf43ab575d2a66636230d539e6910f2 [weak]
- MD5Sum:8867883a511e226111f843dd0c3118ee [weak]
- Filesize:2392 [weak]
Hashes of received file:
- SHA512:5d5aac4ba50da9f37927a0bb72c3a16bd8b64abb8784fd50105fbf595d4d3f12cef041c27b5f8d320bdbc485aefdf40a3d0b79bff09e5e96a8bc7c05e0a36e08
- SHA256:634d8318e0612bf6654d4c75831f88da491411056500caab68e1146d2295020e

Morganh · May 11, 2022, 8:23am

Refer to GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 - #3 by pablete

Try:

$ sudo apt-key del 7fa2af80
$ sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub

More error or question, you can search in that CUDA forum or create a topic on it. Thanks.

user95782 · May 11, 2022, 8:55am

Thank you Morganh. The good news is that we resolved the missing libcufft.so issue. But look like there is still Cuda error. Below is the output from the app:

attcloud@deepstream-dev-02-vm:~/deepstream_lpr_app/deepstream-lpr-app$ ./deepstream-lpr-app 1 2 0 /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_qHD.mp4 output.264

Request sink_0 pad from streammux

Warning: ‘input-dims’ parameter has been deprecated. Use ‘infer-dims’ instead.

Now playing: 1

WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.

WARNING: [TRT]: TensorRT was linked against cuBLAS/cuBLAS LT 11.5.1 but loaded cuBLAS/cuBLAS LT 110.9.2
…

Running…

qtdemux pad video/x-h264

Cuda failure: status=1 in cuResData at line 316

Cuda failure: status=1 in cuResData at line 348

Cuda failure: status=1 in cuResData at line 316

Cuda failure: status=1 in cuResData at line 348

Frame Number = 1 Vehicle Count = 0 Person Count = 0 License Plate Count = 0

Frame Number = 2 Vehicle Count = 0 Person Count = 0 License Plate Count = 0

Cuda failure: status=1 in cuResData at line 316

Cuda failure: status=1 in cuResData at line 337

Frame Number = 3 Vehicle Count = 0 Person Count = 0 License Plate Count = 0

Cuda failure: status=1 in cuResData at line 316

user95782 · May 11, 2022, 8:59am

This is the new Cuda info:

Morganh · May 11, 2022, 9:23am

I am afraid there is still something mismatching/wrong in GPU driver and CUDA.
See Quickstart Guide — DeepStream 6.3 Release documentation

Could you please try to use a docker image provided by deepstream?
See Docker Containers — DeepStream 6.3 Release documentation

You can pull nvcr.io/nvidia/deepstream:6.0.1-devel

user95782 · May 11, 2022, 1:30pm

The VM install is easier for development and integration. Let me try to fix the install some more.

system · May 25, 2022, 1:30pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.