Failed in mem Copy nvcr.io/nvidia/deepstream-l4t:7.1-samples-multiarch

[enea01]

Hi I wanted to mention a problem with the deepstream Docker image

Environment

**• Jetson Orin Nano
**• Deepstream 7.1
**• JetPack 6.1 and 6.2
*• Docker Build image nvcr.io/nvidia/deepstream-l4t:-triton-multiarch
**• Docker run image nvcr.io/nvidia/deepstream-l4t:7.1-samples-multiarch

Continuing the discussion from Failed in mem copy:

I had a similar issue within the Deepstream Docker image. I tried running it both on a board with Jetpack6.2

GPUassert: an illegal memory access was encountered /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvmultiobjecttracker/src/modules/cuDCFv2/cuDCFFrameTransformTexture.cu 611
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform_copy.cpp:341: => Failed in mem copy

0:14:51.580888591     1 0xaaab18233040 WARN                 nvinfer gstnvinfer.cpp:2423:gst_nvinfer_output_loop:<nvinfer1> error: Internal data stream error.
0:14:51.580918928     1 0xaaab18233040 WARN                 nvinfer gstnvinfer.cpp:2423:gst_nvinfer_output_loop:<nvinfer1> error: streaming stopped, reason error (-5)

!![Exception] GPUassert failed
An exception occurred. GPUassert failed

Here is my pipeline

gst-launch-1.0  v4l2src device="/dev/video0"  !  capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1"  !  jpegdec  !  videoconvert  !  nvvideoconvert  !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1"  !  mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0   !  nvinfer name="primary-inference" config-file-path="/app/cfg/inference/YOLOV8S.yml"  !  nvtracker name="tracker" tracker-width=640 tracker-height=384 gpu-id=0 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ll-config-file="/app/cfg/tracking/config_tracker_NvDCF_perf.yml"  ! nvdsanalytics name="analytics" config-file="/app/cfg/analytics/analytics.txt"  !  nvdsosd name="onscreendisplay"  !  nvvideoconvert  !  videoconvert  !  x264enc speed-preset="ultrafast"  !  h264parse  !  rtph264pay  !  udpsink host="127.0.0.1" port=8554 

In the previous forum, the fix proposed was to downgrade to Jetpack 6.1 but the bug still happens in the docker image.

I had to downgrade the board to Jetpack 6.0 rev 2 and deepstrem to 7.0
After that the pipeline is stable in the docker image nvcr.io/nvidia/deepstream-l4t:7.0-samples-multiarch

I think nvcr.io/nvidia/deepstream-l4t:7.1 images need an update to be fixed

[fanzh]

1. did the crash happen at he beginning or after a while?
2. regarding “on a board with Jetpack6.2”, from the compatibilty table, DeepStream7.1 requires Jetpack 6.1 GA.
3. regardng “the bug still happens in the docker image”, do you mean using Jetpack6.1, DS7.1 and the same pipeline, the issue still persist?

[enea01]

The crash happened every time around 5 minutes after launching the pipeline.
The bug happens when running the docker image http://nvcr.io/nvidia/deepstream-l4t:7.1 on both Jetpack 6.2 and Jetpack 6.1. I thought downgrading to 6.1 would solve the issue but it did not. My guess is that the docker image has the issue embedded in it as it comes with all the CUDA and deepstream libraires.

I ran the same pipeline for 24h on Jetpack 6.0 with http://nvcr.io/nvidia/deepstream-l4t:7.0-samples-multiarch without issue

[fanzh]

As written in the topic mentioned in the issue description, please try the following workaround.

nvvideoconvert compute-hw=1 nvbuf-memory-type=3

[enea01]

I tried adding your fix.

gst-launch-1.0  v4l2src device="/dev/video0"  !  capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1"  !  jpegdec  !  videoconvert  !  nvvideoconvert  compute-hw=1 nvbuf-memory-type=3 !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1"  !  mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0   !  nvinfer name="primary-inference" config-file-path="/app/cfg/inference/YOLOV8S.yml"  !  nvtracker name="tracker" tracker-width=640 tracker-height=384 gpu-id=0 ll-lib-file="/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so" ll-config-file="/app/cfg/tracking/config_tracker_NvDCF_perf.yml"  ! nvdsanalytics name="analytics" config-file="/app/cfg/analytics/analytics.txt"  !  nvdsosd name="onscreendisplay"  !  nvvideoconvert compute-hw=1 nvbuf-memory-type=3  !  videoconvert  !  x264enc speed-preset="ultrafast"  !  h264parse  !  rtph264pay  !  udpsink host="127.0.0.1" port=8554 

but I get this error .

 nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /app/model_b1_gpu0_fp16.engine
0:00:00.862655353   105 0xaaaafa142fc0 INFO                 nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:/app/cfg/inference/YOLOV8S.yml sucessfully
0:00:00.863637949   105 0xffff74000da0 FIXME           videodecoder gstvideodecoder.c:1193:gst_video_decoder_drain_out:<jpegdec0> Sub-class should implement drain()
0:00:01.143369775   105 0xffff74000da0 WARN          v4l2bufferpool gstv4l2bufferpool.c:842:gst_v4l2_buffer_pool_start:<v4l2src0:pool0:src> Uncertain or not enough buffers, enabling copy threshold
0:00:01.317014955   105 0xffff74000da0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:01.324609572   105 0xffff74000da0 WARN              video-info video-info.c:760:gst_video_info_to_caps: invalid matrix 4 for RGB format, using RGB
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:4543: => Surface type not supported for transformation NVBUF_MEM_CUDA_UNIFIED

0:00:01.571709406   105 0xffff74000da0 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<v4l2src0> error: Internal data stream error.
0:00:01.571745759   105 0xffff74000da0 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<v4l2src0> error: streaming stopped, reason error (-5)
nvstreammux: Successfully handled EOS for source_id=0
ERROR from element v4l2src0: Internal data stream error.
Error details: ../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason error (-5)

[fanzh]

please use the following steps to narrow down this issue.

1. If using the following cmd including only streammux, will the issue persist?

gst-launch-1.0  v4l2src device="/dev/video0"  !  capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1"  !  jpegdec  !  videoconvert  !  nvvideoconvert  compute-hw=1 nvbuf-memory-type=3 !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1"  !  mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0 ! fakesink

2. If using the following cmd including only “streammux+nviner”, will the issue persist? if so, please add “scaling-compute-hw=1” in the configuration file of nvinfer.

gst-launch-1.0  v4l2src device="/dev/video0"  !  capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1"  !  jpegdec  !  videoconvert  !  nvvideoconvert  compute-hw=1 nvbuf-memory-type=3 !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1"  !  mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0   !  nvinfer name="primary-inference" config-file-path="/app/cfg/inference/YOLOV8S.yml" ! fakesink

[enea01]

I tried running both commands.

1. gst-launch-1.0 v4l2src device="/dev/video0" ! capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1" ! jpegdec ! videoconvert ! nvvideoconvert compute-hw=1 nvbuf-memory-type=3 ! capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1" ! mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0 ! fakesink

The issue persisted with this error message.

0:00:01.302070748    63 0xffff6c000b90 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:01.306008708    63 0xffff6c000b90 WARN              video-info video-info.c:760:gst_video_info_to_caps: invalid matrix 4 for RGB format, using RGB
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:4543: => Surface type not supported for transformation NVBUF_MEM_CUDA_UNIFIED

0:00:01.520400774    63 0xffff6c000b90 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<v4l2src0> error: Internal data stream error.
0:00:01.520426598    63 0xffff6c000b90 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<v4l2src0> error: streaming stopped, reason error (-5)
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:

2. I added the scaling-compute-hw=1 to my nvinfer configuration file and ran

gst-launch-1.0  v4l2src device="/dev/video0"  !  capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1"  !  jpegdec  !  videoconvert  !  nvvideoconvert  compute-hw=1 nvbuf-memory-type=3 !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1"  !  mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0   !  nvinfer name="primary-inference" config-file-path="/app/cfg/inference/YOLOV8S.yml" ! fakesink

I got the same error.

New clock: GstSystemClock
0:00:01.232589964   101 0xffff50000d80 WARN          v4l2bufferpool gstv4l2bufferpool.c:842:gst_v4l2_buffer_pool_start:<v4l2src0:pool0:src> Uncertain or not enough buffers, enabling copy threshold
0:00:01.409233995   101 0xffff50000d80 WARN              video-info video-info.c:760:gst_video_info_to_caps: invalid matrix 4 for RGB format, using RGB
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:4543: => Surface type not supported for transformation NVBUF_MEM_CUDA_UNIFIED

0:00:01.636623595   101 0xffff50000d80 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<v4l2src0> error: Internal data stream error.
0:00:01.636650603   101 0xffff50000d80 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<v4l2src0> error: streaming stopped, reason error (-5)
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.

I retried outside of docker and I get the same errors with both commands.

[fanzh]

the first pipeline in my last comment is a part of your complete command-line. if using this pipeline the issue still persists, please continue to narrow down this issue with the following methods.

1. wondering the negotiated format, please share the 1.log after running the following pipeline. running with “-v” will print the formats.

gst-launch-1.0  v4l2src device="/dev/video0"  !  capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1"  !  jpegdec  !  videoconvert  !  nvvideoconvert  compute-hw=1 nvbuf-memory-type=3 !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1"  !  mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0 ! fakesink >1.log  2>1.log

2. wondering if the issue is related to the source, please share the 2.log after running the following cmd.

gst-launch-1.0 -v videotestsrc ! video/x-raw,format=RGBA,width=1920,height=1080 ! videoconvert  !  nvvideoconvert  compute-hw=1 nvbuf-memory-type=3 !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1" ! fakesink >2.log  2>2.log

3. to continue to simplify the pipeline, please share the 3.log after running the following pipeline.

gst-launch-1.0  v4l2src device="/dev/video0"  !  capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1"  !  jpegdec  !  videoconvert  !  nvvideoconvert  compute-hw=1 nvbuf-memory-type=3 !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1"  !   fakesink  >3.log  2>3.log

[enea01]

@fanzh

gst-launch-1.0 v4l2src device="/dev/video0" ! capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1" ! jpegdec ! videoconvert ! nvvideoconvert compute-hw=1 nvbuf-memory-type=3 ! capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1" ! mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0 ! fakesink >1.log 2>1.log

I ran the first command with “-v” as you asked and also ran commands 2 and 3. Here are the logs
1.log (6.1 KB)
2.log (3.3 KB)
3.log (129.4 KB)

[fanzh]

1. Here is a workaround to the pipeline 2 in my last comment. I only added " compute-hw=1" for nvstreammux.

 gst-launch-1.0  v4l2src device="/dev/video0"  !  capsfilter caps="image/jpeg, width=1920, height=1080, framerate=30/1"  !  jpegdec  !  videoconvert  !  nvvideoconvert  compute-hw=1 nvbuf-memory-type=3 !  capsfilter caps="video/x-raw(memory:NVMM), format=RGBA, width=1920, height=1080, framerate=30/1"  !  mux.sink_0 nvstreammux name="mux" batch-size=1 width=1920 height=1080 batched-push-timeout=4000000 live-source=1 num-surfaces-per-frame=1 sync-inputs=0 max-latency=0  compute-hw=1  ! fakesink  

2. if the pipeline in the step1 works well. you can continue to add other plugins to the pipeline, and add “compute-hw=1” for other plugins which have “compute-hw” property.

[yingliu]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks.