Deepstream Test 3 - nvinfer error:NVDSINFER_CUDA_ERROR Failed in mem copy (Segmentation fault)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): Jetson AGX Orin
• DeepStream Version: 7.1
• JetPack Version (valid for Jetson only): Jetpack L4T 36.4.7 (6.2.1)
• TensorRT Version: 10.3.0.26 (Provided via DS-7.1 Docker Image)
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs): Bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

./user_additional_install.sh && ./user_deepstream_python_apps_install.sh -v 1.2.0
pip3 install "cuda-python==12.6"

mkdir -p /opt/nvidia/deepstream/deepstream/samples/models/peoplenet && \
cd /opt/nvidia/deepstream/deepstream/samples/models/peoplenet && \
wget --content-disposition 'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/tao/peoplenet/pruned_quantized_decrypted_v2.3.4/files?redirect=true&path=resnet34_peoplenet_int8.onnx' -O resnet34_peoplenet_int8.onnx && \
wget --content-disposition 'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/tao/peoplenet/pruned_quantized_decrypted_v2.3.4/files?redirect=true&path=resnet34_peoplenet_int8.txt' -O resnet34_peoplenet_int8.txt && \
wget --content-disposition 'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/tao/peoplenet/pruned_quantized_decrypted_v2.3.4/files?redirect=true&path=labels.txt' -O labels.txt


python3 deepstream_test_3.py -i rtsp://127.0.0.1:8554/ds-test --pgie nvinfer -c config_infer_primary_peoplenet.txt --no-display


python3 deepstream_test_3.py -i file:///opt/nvidia/deepstream/deepstream-7.1/samples/streams/sample_1080p_h264.mp4 --pgie nvinfer -c config_infer_primary_peoplenet.txt --no-display

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I am experiencing an issue with deepstream_python_apps/deepstream_test3.py when launching with either an RTSP stream or source file with a duration longer than a few minutes.

My error is as follows:

Frame Number= 13342 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
ERROR: Failed to add cudaStream callback for returning input buffers, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
ERROR: Failed to make stream wait on event, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
Frame Number= 13343 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
Error: gst-stream-error-quark: Failed to queue input batch for inferencing (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1420): gst_nvinfer_input_queue_loop (): /GstPipeline:pipeline0/GstNvInfer:primary-inference
Exiting app

/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform_copy.cpp:552: => Failed in mem copy

I have tried increasing the number of extra surfaces(num-extra-surfaces) on the decoder to 4 and 16, but the issue still persists.

My power mode is 0: MAXN

My attempt at migrating to DS-8.0 yields the following error:

Traceback (most recent call last):
  File "/opt/nvidia/deepstream/deepstream-8.0/sources/deepstream_python_apps/apps/deepstream-test3/deepstream_test_3.py", line 38, in <module>
    import pyds
ImportError: /opt/nvidia/deepstream/deepstream/lib/libnvds_batch_jpegenc.so: undefined symbol: NvBufSurfaceGetDeviceInfo

where the binding are installed via the following command. I suspect this might be due to CUDA version mismatch with Jetson. My Jetson uses 12.6. Please inform me if this is incorrect.

./user_additional_install.sh && ./user_deepstream_python_apps_install.sh -v 1.2.2
pip3 install "cuda-python==12.8"

I am also experiencing the same issue with another pipeline that uses the nvinferserver element and is derived from ds-test3.py.

Possible duplicates:

3

This is a known issue in JP-6.2, and the workaround you mentioned in the Failed in mem copy with DeepStream 7.1

Try to set value of compute-hw property to 1

nvvidconv = Gst.ElementFactory.make("nvvideoconvert", "convertor")
    if not nvvidconv:
        sys.stderr.write(" Unable to create nvvidconv \n")
    nvvidconv.set_property("compute-hw", 1)

DS-8.0 only support thor with Jetpack-7.0 currently, so please use DS-7.1 on Orin.

4

Understood. Thank you. I’ll continue with DS-7.1

Are there any suggestions on how I can resolve my issue? Were you able to recreate it on your end?

I was able to get my rtsp stream and file sources to run a little longer by adding the following properties to the nvvideoconvert element

...
nvvidconv.set_property("compute-hw", 2)
nvvidconv.set_property("copy-hw", 2)
...

But the pipeline fails regardless.

Using nvinferserver yields the same error with a segmentation fault

ERROR: Failed to record cuda network preprocess event, cuda err_no:700, err_str:cudaErrorIllegalAddress
Segmentation fault (core dumped)
5

Please add only this property for nvvideoconvert, some previous users said it could be resolved.

6

Pardon me. I didn’t see your edit.

I have applied your suggestion and removed nvvidconv.set_property("copy-hw", 2). The pipeline works and is stable for a file source with a 10min duration.

I am currently running the pipeline on a single RTSP stream. I’ll update with my discovery soon

7

The recommendation works and the pipeline is stable for 1+ rtsp stream(s).

Are there any reasons as to why this is the recommended solution? The docs mentions the following:

~# gst-inspect-1.0 nvvideoconvert

...
  compute-hw          : Compute Scaling HW
                        flags: readable, writable, controllable
                        Enum "GstNvComputeHWType" Default: 0, "Default"
                           (0): Default          - Default, GPU for Tesla, VIC for Jetson
                           (1): GPU              - GPU
                           (2): VIC              - VIC
...

Setting other properties such as below does not seem to impact the pipeline:

streammux.set_property('compute-hw', 2)
nvvidconv.set_property("copy-hw", 2)