DS 6.0 Shared memory multiple GPU issue

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU 2XNvidia T4
• DeepStream Version nvcr.io/nvidia/deepstream:6.0-triton
• TensorRT Version TensorRT 8.0.1
• NVIDIA GPU Driver Version (valid for GPU only) 470.103.01
• Issue Type( questions, new requirements, bugs) Bug, the issue appears to be using multi GPU shared memory in DS 6. The same app and config works fine with one GPU on DS 6.0 and the below config works with DS 5.1 with 2 GPUs. The app on DS 6.0 2 GPU’s starts without error but it doesn’t run, freezes, shows green in the rtsp output and flickers ever 5-6 seconds.
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Using deepstream test app 5, and the following config:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=600
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=5
columns=6
width=1920
height=1080
gpu-id=1
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=3

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
#dock1_1
uri=rtsp://10.0.137.146/axis-media/media.amp?streamprofile=ddi
#num-sources=1
gpu-id=1
rtsp-reconnect-interval-sec=60
nvbuf-memory-type=3
cudadec-memtype=2

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.147/axis-media/media.amp?streamprofile=ddi
gpu-id=1
rtsp-reconnect-interval-sec=60
nvbuf-memory-type=3
cudadec-memtype=2

[source2]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.24/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source3]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.149/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source4]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.150/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source5]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.150/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source6]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.89/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

#POLE 7 FRONT
[source7]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.89/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source8]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.90/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source9]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.23/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source10]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.22/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source11]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.134.141/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source12]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
#dock2_2A 50-54
uri=rtsp://10.0.137.142/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source13]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
#dock2_2B
uri=rtsp://10.0.137.142/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source14]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.143/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source15]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.143/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source16]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.144/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source17]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.145/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source18]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.57/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source19]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.21/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source20]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.114/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source21]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.134.140/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source22]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.99/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source23]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.30/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=600
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source24]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.33/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source25]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.122/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source26]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.134.140/axis-media/media.amp?streamprofile=ddi&camera=2
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=3
#1=mp4 2=mkv
#container=1
#1=h264 2=h265
#codec=1
#output-file=yolov4.mp4

#TESTING WITH CLOUD KAFKA
[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=naming_conv_iotedge.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM   - Custom schema payload
msg-conv-payload-type=1
#(0): Create payload using NvdsEventMsgMeta
#(1): New Api to create payload using NvDsFrameMeta
msg-conv-msg2p-new-api=0
#Frame interval at which payload is generated
msg-conv-frame-interval=5
#frame-interval=100
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_kafka_proto.so
#Provide your msg-broker-conn-str here
msg-broker-conn-str=localhost;9094
topic=Helium
#new-api=0
#(0) Use message adapter library api's
#(1) Use new msgbroker library api's
sync=0

#RTSP output to see analytics
[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=4
sync=0
source-id=0
gpu-id=1
nvbuf-memory-type=3
#1=h264 2=h265
codec=1
bitrate=4000000
iframeinterval=10
rtsp-port=8554
udp-port=5400
profile=0

[osd]
enable=1
gpu-id=1
border-width=2
text-size=12
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=3

[streammux]
gpu-id=1
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=27
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=3

[primary-gie]
enable=1
gpu-id=0
labelfile-path=labels.txt
batch-size=27
#force-implicit-batch-dim=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=5
gie-unique-id=1
operate-on-class-ids=0
nvbuf-memory-type=3
#config-file=yolov4/config_infer_primary.txt
#config-file=yolov4-csp/config_infer_primary.txt
#config-file=yolor-csp/config_infer_primary.txt
config-file=yolov4-tiny/config_infer_primary.txt
#config-file=yolor-csp-x-star/config_infer_primary.txt
#config-file=yolov4x-mish/config_infer_primary.txt

[tracker]
enable=1
# For NvDCF and DeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
nvbuf-memory-type=3
ll-lib-file=/opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_nvmultiobjecttracker.so
# ll-config-file required to set different tracker types
# ll-config-file=config_tracker_IOU.yml
#ll-config-file=config_tracker_NvDCF_perf.yml
ll-config-file=config_tracker_NvDCF_accuracy.yml
# ll-config-file=config_tracker_DeepSORT.yml
gpu-id=1
enable-batch-process=1
enable-past-frame=1
display-tracking-id=1

[nvds-analytics]
enable=0
config-file=analytics_line.txt

The setup is everything on GPU 1 except primary gie which is GPU 0. Did anything change with DS 6.0 compared to DS 5.1 in relation to shared memory?

5

Hi @gabe_ddi
Could you try attached two configs which doES inference on 27 streams and only pgie running on gpu#0, the others are on GPU#1

config_infer_primary_yolov4.txt (2.3 KB)
deepstream_app_source1_detection_models.txt (3.8 KB)

Test steps:

  1. Follow deepstream_reference_apps/README.md at master · NVIDIA-AI-IOT/deepstream_reference_apps · GitHub to install TAO in nvcr.io/nvidia/deepstream:6.0-devel
  2. replace deepstream_app_source1_detection_models.txt and config_infer_primary_yolov4.txt with attached two files
  3. run
    # deepstream-app -c deepstream_app_source1_detection_models.txt

This works on my side, please check if it works as well on your side.

6

Thanks, will try this and report back soon

8

Same issue, it shows the following logs though:

** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src
Warning: Color primaries 5 not present and will be treated BT.601
Warning: Color primaries 5 not present and will be treated BT.601
** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src
Warning: Color primaries 5 not present and will be treated BT.601
Warning: Color primaries 5 not present and will be treated BT.601
** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src
** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src
** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src

9

Hi @gabe_ddi ,
But my script does not use rtsp, did you use the scripts I attached?

10

The machine is a live customer site, it is hard to use a different setup as it is currently using iotedge from azure in docker containers. Let me get back to you

11

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.