Deepstream-app error when set gpu-id=2

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Hello, I have encountered some issues and may need your assistance.

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.3
• JetPack Version (valid for Jetson only)
• TensorRT Version cuda12.1 and tensorrt8.5.3.1
• NVIDIA GPU Driver Version (valid for GPU only) 535.104.05
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Docker image:nvcr.io/nvidia/deepstream:6.3-triton-multiarch.
*GPU:rtx3070 （*4）
cd /opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app
deepstream-app -c source2_1080p_dec_infer-resnet_demux_int8.txt

If I run with gpu-id=0, everything works perfectly.An error occurred when I used gpu-id=2.
The error information is as follows:

0:00:05.676919932   423 0x5631b7727230 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b2_gpu0_int8.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x368x640       
1   OUTPUT kFLOAT conv2d_bbox     16x23x40        
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40         

0:00:05.802166675   423 0x5631b7727230 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b2_gpu0_int8.engine
		0:00:05.806755501   423 0x5631b7727230 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_primary_gpu2.txt sucessfully

		Runtime commands:
				h: Print this help
				q: Quit

				p: Pause
				r: Resume

		** INFO: <bus_callback:239>: Pipeline ready

WARNING from src_elem: No decoder available for type 'audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, level=(string)2, base-profile=(string)lc, profile=(string)lc, codec_data=(buffer)119056e500, rate=(int)48000, channels=(int)2'.
Debug info: gsturidecodebin.c(920): unknown_type_cb (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstURIDecodeBin:src_elem
WARNING from src_elem: No decoder available for type 'audio/mpeg, mpegversion=(int)4, framed=(boolean)true, stream-format=(string)raw, level=(string)2, base-profile=(string)lc, profile=(string)lc, codec_data=(buffer)119056e500, rate=(int)48000, channels=(int)2'.
Debug info: gsturidecodebin.c(920): unknown_type_cb (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin1/GstURIDecodeBin:src_elem
		** INFO: <bus_callback:225>: Pipeline running

0:00:06.740833171   423 0x5631b69f40c0 ERROR         nvvideoconvert gstnvvideoconvert.c:4095:gst_nvvideoconvert_transform: buffer transform failed
0:00:06.741847070   423 0x5631b67d35e0 ERROR         nvvideoconvert gstnvvideoconvert.c:4095:gst_nvvideoconvert_transform: buffer transform failed
0:00:06.753960905   423 0x5631b69f4300 WARN                 nvinfer gstnvinfer.cpp:2397:gst_nvinfer_output_loop:<primary_gie> error: Internal data stream error.
0:00:06.753997411   423 0x5631b69f4300 WARN                 nvinfer gstnvinfer.cpp:2397:gst_nvinfer_output_loop:<primary_gie> error: streaming stopped, reason error (-5)
ERROR from primary_gie: Internal data stream error.
Debug info: gstnvinfer.cpp(2397): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie:
streaming stopped, reason error (-5)
ERROR from qtdemux0: Internal data stream error.
Debug info: qtdemux.c(6619): gst_qtdemux_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin1/GstURIDecodeBin:src_elem/GstDecodeBin:decodebin1/GstQTDemux:qtdemux0:
streaming stopped, reason error (-5)
ERROR from qtdemux1: Internal data stream error.
Debug info: qtdemux.c(6619): gst_qtdemux_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstURIDecodeBin:src_elem/GstDecodeBin:decodebin0/GstQTDemux:qtdemux1:
streaming stopped, reason error (-5)
Quitting
ERROR from osd_queue: Internal data stream error.
Debug info: gstqueue.c(988): gst_queue_handle_sink_event (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:osd_bin/GstQueue:osd_queue:
streaming stopped, reason error (-5)
ERROR from osd_queue: Internal data stream error.
Debug info: gstqueue.c(988): gst_queue_handle_sink_event (): /GstPipeline:pipeline/GstBin:processing_bin_1/GstBin:osd_bin/GstQueue:osd_queue:
streaming stopped, reason error (-5)
App run failed

I found that it does not throw an error when I do not use the encoding function, but as soon as I export the resulting video or generate an RTSP stream, it throws an error.

Looking forward to your response.

3

This is a known issue as video memory cannot be shared between multiple GPUs.

You also need to modify the value of nvbuf-memory-type

nvbuf-memory-type=3

Or directly specify the GPU_ID where CUDA is running

export CUDA_VISIBLE_DEVICES="gpu-id"
4

Hello, I tried it and it does work. However, after the modification, both GPU 0 and GPU 2 are using 384MB of memory simultaneously, even though I only set gpu-id=2. Is this behavior normal?

5

This is a known issue.

If you want to run on only one GPU, you can export CUDA_VISIBLE_DEVICES=gpu-id before running the program.

6

Thank you for your response.I want to launch multiple pipelines in one program, with each pipeline running on a different GPU (0, 1, 2, 3). If I set export CUDA_VISIBLE_DEVICES=gpu-id, can I still achieve this effect?

7

If I want to launch multiple pipelines simultaneously in one program to encode output streams or videos on different GPUs, how should I configure the parameters?

8

If each pipeline is an independent process, it should be effective. Otherwise, I don’t think it will be effective.

The above configuration is valid, except that decoding will run on GPU 0, and inference and encoding will run on other GPUs.

9

Can I still encode output on multiple GPUs if I launch multiple pipelines in the same thread? During testing, I found that launching multiple pipelines in different threads within the same process consumes a significant amount of resources. Therefore, I launched multiple pipelines within the same thread.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.