DS 6.0 Shared memory multiple GPU issue

gabe_ddi · March 10, 2022, 9:59pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU 2XNvidia T4
• DeepStream Version nvcr.io/nvidia/deepstream:6.0-triton
• TensorRT Version TensorRT 8.0.1
• NVIDIA GPU Driver Version (valid for GPU only) 470.103.01
• Issue Type( questions, new requirements, bugs) Bug, the issue appears to be using multi GPU shared memory in DS 6. The same app and config works fine with one GPU on DS 6.0 and the below config works with DS 5.1 with 2 GPUs. The app on DS 6.0 2 GPU’s starts without error but it doesn’t run, freezes, shows green in the rtsp output and flickers ever 5-6 seconds.
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Using deepstream test app 5, and the following config:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=600
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=5
columns=6
width=1920
height=1080
gpu-id=1
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=3

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
#dock1_1
uri=rtsp://10.0.137.146/axis-media/media.amp?streamprofile=ddi
#num-sources=1
gpu-id=1
rtsp-reconnect-interval-sec=60
nvbuf-memory-type=3
cudadec-memtype=2

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.147/axis-media/media.amp?streamprofile=ddi
gpu-id=1
rtsp-reconnect-interval-sec=60
nvbuf-memory-type=3
cudadec-memtype=2

[source2]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.24/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source3]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.149/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source4]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.150/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source5]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.150/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source6]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.89/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

#POLE 7 FRONT
[source7]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.89/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source8]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.90/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source9]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.23/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source10]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.22/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source11]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.134.141/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source12]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
#dock2_2A 50-54
uri=rtsp://10.0.137.142/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source13]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
#dock2_2B
uri=rtsp://10.0.137.142/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source14]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.143/axis-media/media.amp?camera=5
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source15]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.143/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source16]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.144/axis-media/media.amp?camera=7
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source17]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.145/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source18]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.57/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source19]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.21/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source20]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.114/axis-media/media.amp
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source21]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.134.140/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source22]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.99/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source23]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.30/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=600
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source24]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.33/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source25]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.137.122/axis-media/media.amp?streamprofile=ddi
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[source26]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://10.0.134.140/axis-media/media.amp?streamprofile=ddi&camera=2
rtsp-reconnect-interval-sec=60
gpu-id=1
nvbuf-memory-type=3
cudadec-memtype=2

[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=3
#1=mp4 2=mkv
#container=1
#1=h264 2=h265
#codec=1
#output-file=yolov4.mp4

#TESTING WITH CLOUD KAFKA
[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=naming_conv_iotedge.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM   - Custom schema payload
msg-conv-payload-type=1
#(0): Create payload using NvdsEventMsgMeta
#(1): New Api to create payload using NvDsFrameMeta
msg-conv-msg2p-new-api=0
#Frame interval at which payload is generated
msg-conv-frame-interval=5
#frame-interval=100
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_kafka_proto.so
#Provide your msg-broker-conn-str here
msg-broker-conn-str=localhost;9094
topic=Helium
#new-api=0
#(0) Use message adapter library api's
#(1) Use new msgbroker library api's
sync=0

#RTSP output to see analytics
[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=4
sync=0
source-id=0
gpu-id=1
nvbuf-memory-type=3
#1=h264 2=h265
codec=1
bitrate=4000000
iframeinterval=10
rtsp-port=8554
udp-port=5400
profile=0

[osd]
enable=1
gpu-id=1
border-width=2
text-size=12
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=3

[streammux]
gpu-id=1
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=27
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=3

[primary-gie]
enable=1
gpu-id=0
labelfile-path=labels.txt
batch-size=27
#force-implicit-batch-dim=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=5
gie-unique-id=1
operate-on-class-ids=0
nvbuf-memory-type=3
#config-file=yolov4/config_infer_primary.txt
#config-file=yolov4-csp/config_infer_primary.txt
#config-file=yolor-csp/config_infer_primary.txt
config-file=yolov4-tiny/config_infer_primary.txt
#config-file=yolor-csp-x-star/config_infer_primary.txt
#config-file=yolov4x-mish/config_infer_primary.txt

[tracker]
enable=1
# For NvDCF and DeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
nvbuf-memory-type=3
ll-lib-file=/opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_nvmultiobjecttracker.so
# ll-config-file required to set different tracker types
# ll-config-file=config_tracker_IOU.yml
#ll-config-file=config_tracker_NvDCF_perf.yml
ll-config-file=config_tracker_NvDCF_accuracy.yml
# ll-config-file=config_tracker_DeepSORT.yml
gpu-id=1
enable-batch-process=1
enable-past-frame=1
display-tracking-id=1

[nvds-analytics]
enable=0
config-file=analytics_line.txt

The setup is everything on GPU 1 except primary gie which is GPU 0. Did anything change with DS 6.0 compared to DS 5.1 in relation to shared memory?

mchi · March 16, 2022, 3:15pm

Hi @gabe_ddi
Could you try attached two configs which doES inference on 27 streams and only pgie running on gpu#0, the others are on GPU#1

config_infer_primary_yolov4.txt (2.3 KB)
deepstream_app_source1_detection_models.txt (3.8 KB)

Test steps:

Follow deepstream_reference_apps/README.md at master · NVIDIA-AI-IOT/deepstream_reference_apps · GitHub to install TAO in nvcr.io/nvidia/deepstream:6.0-devel
replace deepstream_app_source1_detection_models.txt and config_infer_primary_yolov4.txt with attached two files
run
# deepstream-app -c deepstream_app_source1_detection_models.txt

This works on my side, please check if it works as well on your side.

gabe_ddi · March 17, 2022, 12:34am

Thanks, will try this and report back soon

gabe_ddi · March 18, 2022, 5:14am

Same issue, it shows the following logs though:

** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src
Warning: Color primaries 5 not present and will be treated BT.601
Warning: Color primaries 5 not present and will be treated BT.601
** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src
Warning: Color primaries 5 not present and will be treated BT.601
Warning: Color primaries 5 not present and will be treated BT.601
** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src
** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src
** ERROR: <cb_newpad3:498>: Failed to link depay loader to rtsp src

mchi · March 18, 2022, 6:38am

Hi @gabe_ddi ,
But my script does not use rtsp, did you use the scripts I attached?

gabe_ddi · March 20, 2022, 9:30pm

The machine is a live customer site, it is hard to use a different setup as it is currently using iotedge from azure in docker containers. Let me get back to you

yingliu · June 15, 2022, 7:30am

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

system · June 29, 2022, 8:26am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.