I am trying to run the GazeNet TAO sample app on Jetson with following configuration:
NVIDIA Jetson AGX Xavier [16GB]
L4T 32.6.1 [ JetPack 4.6 ]
Ubuntu 18.04.5 LTS
Kernel Version: 4.9.253-tegra
CUDA 10.2.300
CUDA Architecture: 7.2
OpenCV version: 4.1.1
OpenCV Cuda: NO
CUDNN: 8.2.1.32
TensorRT: 8.0.1.6
Vision Works: 1.6.0.501
VPI: ii libnvvpi1 1.1.12 arm64 NVIDIA Vision Programming Interface library
Vulcan: 1.2.70
When I run the app with the command as mentioned in ReadME (with default sample config files)
./deepstream-gaze-app 2 /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/configs/facial_tao/sample_faciallandmarks_config.txt file:///home/nvidia/Pictures/face.jpg ./gazenet
It crashes with the following error:
Request sink_0 pad from streammux
Now playing: file:///home/nvidia/Pictures/face.jpg
Library Opened Successfully
Setting custom lib properties # 1
Adding Prop: config-file : ../../../configs/gaze_tao/sample_gazenet_model_config.txt
Inside Custom Lib : Setting Prop Key=config-file Value=../../../configs/gaze_tao/sample_gazenet_model_config.txt
0:00:03.325956146 22185 0x55a517f8f0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 2]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/faciallandmarks.etlt_b32_gpu0_int8.engine
INFO: [FullDims Engine Info]: layers num: 3
0 INPUT kFLOAT input_face_images:0 1x80x80 min: 1x1x80x80 opt: 32x1x80x80 Max: 32x1x80x80
1 OUTPUT kFLOAT softargmax/strided_slice_1:0 80 min: 0 opt: 0 Max: 0
2 OUTPUT kFLOAT softargmax/strided_slice:0 80x2 min: 0 opt: 0 Max: 0
0:00:03.326175740 22185 0x55a517f8f0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 2]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/faciallandmarks.etlt_b32_gpu0_int8.engine
0:00:03.348156671 22185 0x55a517f8f0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<second-infer-engine1> [UID 2]: Load new model:../../../configs/facial_tao/faciallandmark_sgie_config.txt sucessfully
0:00:03.348498992 22185 0x55a517f8f0 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1161> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
0:00:03.384695424 22185 0x55a517f8f0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/facenet.etlt_b1_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x416x736
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x26x46
2 OUTPUT kFLOAT output_cov/Sigmoid 1x26x46
0:00:03.384832039 22185 0x55a517f8f0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/facenet.etlt_b1_gpu0_int8.engine
0:00:03.387896538 22185 0x55a517f8f0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-infer-engine1> [UID 1]: Load new model:../../../configs/facial_tao/config_infer_primary_facenet.txt sucessfully
Decodebin child added: source
Decodebin child added: decodebin0
Running...
Decodebin child added: nvjpegdec0
In cb_newpad
###Decodebin pick nvidia decoder plugin.
0:00:03.417211615 22185 0x55a5158d40 WARN nvvideoconvert gstnvvideoconvert.c:3098:gst_nvvideoconvert_transform:<source_nvvidconv> error: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
0:00:03.417257217 22185 0x55a5158d40 WARN nvvideoconvert gstnvvideoconvert.c:3098:gst_nvvideoconvert_transform:<source_nvvidconv> error: surface-gpu-id=604095472,source_nvvidconv-gpu-id=0
0:00:03.417366983 22185 0x55a5158d40 ERROR nvvideoconvert gstnvvideoconvert.c:3484:gst_nvvideoconvert_transform: buffer transform failed
ERROR from element source_nvvidconv: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvvideoconvert/gstnvvideoconvert.c(3098): gst_nvvideoconvert_transform (): /GstPipeline:pipeline/GstBin:source-bin-00/Gstnvvideoconvert:source_nvvidconv:
surface-gpu-id=604095472,source_nvvidconv-gpu-id=0
Returned, stopping playback
Deserializing engine from: ./gazeinfer_impl/../../../../models/gazenet/gazenet_facegrid.etlt_b8_gpu0_fp16.engineThe logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. TensorRT maintains only a single logger pointer at any given time, so the existing value, which can be retrieved with getLogger(), will be used instead. In order to use a new logger, first destroy all existing builder, runner or refitter objects.
Average fps 0.000233
Totally 0 faces are inferred
Deleting pipeline
The same problem occurs when running the deepstream-faciallandmark-app. As clearly depicted, the application is trying to allocate default memory type as dGPU in Jetson devices.
ERROR from element source_nvvidconv: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvvideoconvert/gstnvvideoconvert.c(3098): gst_nvvideoconvert_transform (): /GstPipeline:pipeline/GstBin:source-bin-00/Gstnvvideoconvert:source_nvvidconv:
surface-gpu-id=604095472,source_nvvidconv-gpu-id=0
I have commented gpu-id=0 in every config files related to the app. Usually there is #ifndef PLATFORM TEGRA flag inside the codes but still somewhat the nvvidconv element is allocating dGPU memory. Please help me resolving the issue.
I can not reproduce the error on my board. Have you tried deepstream-test1 sample?
Yes I can run the deepstream-test1 sample successfully. But the facial landmark app along with the gazenet app returns the error.
~/Documents/deepstream-6.0/sources/deepstream_tao_apps/apps/tao_others/deepstream-faciallandmark-app$ ./deepstream-faciallandmark-app 2 ../../../configs/facial_tao/sample_faciallandmarks_config.txt file:///home/nvidia/Pictures/face.jpg ./landmarks
(gst-plugin-scanner:11601): GStreamer-WARNING **: 18:05:38.282: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libcustom2d_preprocess.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libcustom2d_preprocess.so: undefined symbol: NvBufSurfTransformAsync
(gst-plugin-scanner:11601): GStreamer-WARNING **: 18:05:38.284: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
(gst-plugin-scanner:11601): GStreamer-WARNING **: 18:05:38.286: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory
(gst-plugin-scanner:11601): GStreamer-WARNING **: 18:05:38.378: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_preprocess.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_preprocess.so: undefined symbol: NvBufSurfTransformAsync
Request sink_0 pad from streammux
Now playing: file:///home/sigmind/Pictures/face.jpg
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:04.461341173 11600 0x55a159d0d0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 2]: deserialized trt engine from :/home/sigmind/Documents/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/faciallandmarks.etlt_b32_gpu0_int8.engine
INFO: [FullDims Engine Info]: layers num: 3
0 INPUT kFLOAT input_face_images:0 1x80x80 min: 1x1x80x80 opt: 32x1x80x80 Max: 32x1x80x80
1 OUTPUT kFLOAT softargmax/strided_slice_1:0 80 min: 0 opt: 0 Max: 0
2 OUTPUT kFLOAT softargmax/strided_slice:0 80x2 min: 0 opt: 0 Max: 0
0:00:04.461586945 11600 0x55a159d0d0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 2]: Use deserialized engine model: /home/sigmind/Documents/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/faciallandmarks.etlt_b32_gpu0_int8.engine
0:00:04.630281416 11600 0x55a159d0d0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<second-infer-engine1> [UID 2]: Load new model:../../../configs/facial_tao/faciallandmark_sgie_config.txt sucessfully
0:00:04.630575767 11600 0x55a159d0d0 WARN nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1161> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:04.658072852 11600 0x55a159d0d0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/home/sigmind/Documents/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/facenet.etlt_b1_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x416x736
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x26x46
2 OUTPUT kFLOAT output_cov/Sigmoid 1x26x46
0:00:04.658223164 11600 0x55a159d0d0 INFO nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /home/sigmind/Documents/deepstream-6.0/sources/deepstream_tao_apps/models/faciallandmark/facenet.etlt_b1_gpu0_int8.engine
0:00:04.660025878 11600 0x55a159d0d0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-infer-engine1> [UID 1]: Load new model:../../../configs/facial_tao/config_infer_primary_facenet.txt sucessfully
Decodebin child added: source
Decodebin child added: decodebin0
Running...
Decodebin child added: nvjpegdec0
In cb_newpad
###Decodebin pick nvidia decoder plugin.
0:00:04.688422111 11600 0x55a1769190 WARN nvvideoconvert gstnvvideoconvert.c:3098:gst_nvvideoconvert_transform:<source_nvvidconv> error: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
0:00:04.688468066 11600 0x55a1769190 WARN nvvideoconvert gstnvvideoconvert.c:3098:gst_nvvideoconvert_transform:<source_nvvidconv> error: surface-gpu-id=671220208,source_nvvidconv-gpu-id=0
0:00:04.688572199 11600 0x55a1769190 ERROR nvvideoconvert gstnvvideoconvert.c:3484:gst_nvvideoconvert_transform: buffer transform failed
ERROR from element source_nvvidconv: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvvideoconvert/gstnvvideoconvert.c(3098): gst_nvvideoconvert_transform (): /GstPipeline:pipeline/GstBin:source-bin-00/Gstnvvideoconvert:source_nvvidconv:
surface-gpu-id=671220208,source_nvvidconv-gpu-id=0
Returned, stopping playback
Average fps 0.000233
Totally 0 faces are inferred
Deleting pipeline
Any clue from my debug log?
I have done exactly the step-by-step tutorial to the point. But still the error persists.
Btw, I am confused on the step:
For Jetson
Copy the gst-nvdsvideotemplate plugin source code from DeepStream for servers and workstations package and copy from the following folder:
/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsvideotemplate
cd /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsvideotemplate
make
cp libnvdsgst_videotemplate.so /opt/nvidia/deepstream/deepstream/lib/gst-plugins/
rm -rf ~/.cache/gstreamer-1.0/
As mentioned, should I download the DeepStream for servers and workstations package tar file, untar and copy the unzipped_file/sources/gst-plugins/gst-nvdsvideotemplate folder to my Jetson systems /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsvideotemplate folder?
There is already a folder in my Jetson’s /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdsvideotemplate path. Why should I copy the from DeepStream for servers and workstations package?
It seems ambiguous and makes no sense.
For a sanity check, I have flashed a Jetson Nano with JP 4.6 with Ds 6 and tried to reproduce the error. And the error shows up there too in the facialLandmark app:
In cb_newpad
###Decodebin pick nvidia decoder plugin.
0:02:10.708442007 31433 0x5599347320 WARN nvvideoconvert gstnvvideoconvert.c:3098:gst_nvvideoconvert_transform:<source_nvvidconv> error: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
0:02:10.708494042 31433 0x5599347320 WARN nvvideoconvert gstnvvideoconvert.c:3098:gst_nvvideoconvert_transform:<source_nvvidconv> error: surface-gpu-id=202084192,source_nvvidconv-gpu-id=0
0:02:10.708586462 31433 0x5599347320 ERROR nvvideoconvert gstnvvideoconvert.c:3484:gst_nvvideoconvert_transform: buffer transform failed
ERROR from element source_nvvidconv: Memory Compatibility Error:Input surface gpu-id doesnt match with configured gpu-id for element, please allocate input using unified memory, or use same gpu-ids OR, if same gpu-ids are used ensure appropriate Cuda memories are used
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvvideoconvert/gstnvvideoconvert.c(3098): gst_nvvideoconvert_transform (): /GstPipeline:pipeline/GstBin:source-bin-00/Gstnvvideoconvert:source_nvvidconv:
surface-gpu-id=202084192,source_nvvidconv-gpu-id=0
Returned, stopping playback
Average fps 0.000233
Totally 0 faces are inferred