Cuda failure: status=801 DeepStream 6.1 o Azure

[w.viscomi]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)

GPU A10 (NV6ads_A10_v5 on Azure)

• DeepStream Version

6.1

• JetPack Version (valid for Jetson only)
• TensorRT Version

8.2.5-1+cuda11.4

• NVIDIA GPU Driver Version (valid for GPU only)

| NVIDIA-SMI 510.73.08 Driver Version: 510.73.08 CUDA Version: 11.6 |

• Issue Type( questions, new requirements, bugs)

bugs

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m width=1920 height=1080 batch-size=1 batched-push-timeout=4000000 ! nvinfer config-file-path=dstest1_pgie_config.txt ! nvvideoconvert ! nvdsosd ! fakesink

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

/opt/nvidia/deepstream/deepstream-6.1/sources/deepstream_python_apps/apps/deepstream-test1-rtsp-out$ gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m width=1920 height=1080 batch-size=1 batched-push-timeout=4000000 ! nvinfer config-file-path=dstest1_pgie_config.txt ! nvvideoconvert ! nvdsosd ! fakesink
Setting pipeline to PAUSED …
0:00:00.293733737 256968 0x55a4aede3410 WARN nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1161> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
WARNING: …/nvdsinfer/nvdsinfer_model_builder.cpp:1482 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.1/sources/deepstream_python_apps/apps/deepstream-test1-rtsp-out/…/…/…/…/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine open error
0:00:02.098307152 256968 0x55a4aede3410 WARN nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1888> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.1/sources/deepstream_python_apps/apps/deepstream-test1-rtsp-out/…/…/…/…/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine failed
0:00:02.131737550 256968 0x55a4aede3410 WARN nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1993> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.1/sources/deepstream_python_apps/apps/deepstream-test1-rtsp-out/…/…/…/…/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine failed, try rebuild
0:00:02.131762617 256968 0x55a4aede3410 INFO nvinfer gstnvinfer.cpp:646:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files

WARNING: …/nvdsinfer/nvdsinfer_model_builder.cpp:1454 Serialize engine failed because of file path: /opt/nvidia/deepstream/deepstream-6.1/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine opened error
0:01:36.619951667 256968 0x55a4aede3410 WARN nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1941> [UID = 1]: failed to serialize cude engine to file: /opt/nvidia/deepstream/deepstream-6.1/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x368x640
1 OUTPUT kFLOAT conv2d_bbox 16x23x40
2 OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40

0:01:36.651069135 256968 0x55a4aede3410 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus: [UID 1]: Load new model:dstest1_pgie_config.txt sucessfully
Pipeline is PREROLLING …
Cuda failure: status=801

[fanzh]

cudaErrorNotSupported = 801
This error indicates the attempted operation is not supported on the current system or device.

please check if all lib versions meet DeepStream 6.1’s demand. please refer to this link: Quickstart Guide — DeepStream 6.2 Release documentation
here are the commands:
CUDA version nvcc -V
dpkg -l |grep TensorRT
dpkg -l | grep gstreamer
dpkg -l|grep cudnn
if all lib versions meet demand, to get more logs, could you use gst-launch-1.0 --gst-debug=*:5… to run again?

[fanzh]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

there are some prerequisites before using deepstream docker, please check the environment on your host according to this: deepstream docker
Ensure these prerequisites are available on your system:

1. nvidia-docker We recommend using Docker 20.10.13 along with the latest nvidia-container-toolkit as described in the installation steps. Usage of nvidia-docker2 packages in conjunction with prior docker versions are now deprecated.
2. NVIDIA display driver version