Segmentation fault in running pipeline with yolov3 AGX

•Hardware Platform : Jetson
•DeepStream Version 5.0
•JetPack Version 4.4 [L4T 32.4.3]
•TensorRT Version 7.1.3
•Issue Type(question/bugs)
Using deepstream-test5 with yolov3 that is bundled with DS over 5 RTSP cameras is throwing a segmentation fault after pipeline ran for some time. It goes something like:

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***

Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
[NvDCF] Initialized
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Deserialize yoloLayerV3 plugin: yolo_83
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
INFO: [Implicit Engine Info]: layers num: 4
0   INPUT  kFLOAT data            3x608x608       
1   OUTPUT kFLOAT yolo_83         255x19x19       
2   OUTPUT kFLOAT yolo_95         255x38x38       
3   OUTPUT kFLOAT yolo_107        255x76x76       


Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:181>: Pipeline ready
Thu Apr  8 03:40:21 2021
**PERF:  0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	

Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
** INFO: <bus_callback:167>: Pipeline running

WARNING; playback mode used with URI [rtsp://****/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://****/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://****/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://****/cam0_0] not conforming to timestamp format; check README; using system-time
Thu Apr  8 03:40:26 2021
**PERF:  9.56 (9.29)	10.38 (10.22)	10.07 (9.80)	10.06 (9.78)	9.26 (9.09)	
.
After few hours
.
Thu Apr  8 04:26:31 2021
**PERF:  10.11 (9.69)	10.21 (10.08)	10.28 (10.02)	10.11 (9.90)	10.37 (9.69)	
Quitting
Segmentation fault

What can be the possible causes for this?

No clue in your description.

updated, lmk if it helps

Where did you got the yolov3 models? Are you running the case as described in /opt/nvidia/deepstream/deepstream-5.1/sources/objectDetector_Yolo/README?

Yes. Same. Just within deepstream-test5 and 5 rtsp sources.

Can you send us your deepstream-test5 config file and yolov3 nvinfer config file?

config_infer_primary_yoloV3.txt (3.4 KB) test.txt (8.1 KB)
Here are the files. Here’s some more info: I tried clearing gstreamer cache. There are other processes running on the AGX too: 5 python processes, kafka broker and mongod. Mongo’s dbpath is set to an external SDcard but its mounted inside the deepstream-test5 folder.

Still having the same issue:

 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***

Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
[NvDCF] Initialized
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Deserialize yoloLayerV3 plugin: yolo_83
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
0:00:04.761050590 21867   0x5565f2c440 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test5/configs/model_b8_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 4
0   INPUT  kFLOAT data            3x608x608       
1   OUTPUT kFLOAT yolo_83         255x19x19       
2   OUTPUT kFLOAT yolo_95         255x38x38       
3   OUTPUT kFLOAT yolo_107        255x76x76       

0:00:04.761338188 21867   0x5565f2c440 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test5/configs/model_b8_gpu0_int8.engine
0:00:04.828050418 21867   0x5565f2c440 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test5/configs/config_infer_primary_yoloV3.txt sucessfully

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:181>: Pipeline ready

Fri Apr  9 06:40:03 2021
**PERF:  0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:167>: Pipeline running

H264: Profile = 66, Level = 0 
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.202/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.206/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.204/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.205/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.203/cam0_0] not conforming to timestamp format; check README; using system-time
Fri Apr  9 06:40:08 2021
**PERF:  8.10 (7.89)	7.25 (7.09)	8.10 (7.89)	9.39 (9.15)	8.56 (8.39)	
Fri Apr  9 06:40:13 2021
**PERF:  9.32 (8.62)	9.17 (8.19)	9.12 (8.51)	9.08 (8.81)	9.17 (8.73)	
Fri Apr  9 06:40:18 2021
**PERF:  9.14 (8.89)	9.14 (8.63)	9.14 (8.82)	9.04 (9.11)	8.95 (8.89)	
Fri Apr  9 06:40:23 2021
**PERF:  9.27 (8.92)	9.04 (8.73)	8.84 (8.82)	8.84 (9.03)	8.84 (8.87)	
Fri Apr  9 06:40:28 2021
**PERF:  8.93 (8.98)	8.55 (8.70)	8.75 (8.81)	8.95 (9.02)	8.95 (8.90)	
Fri Apr  9 06:40:33 2021
**PERF:  9.38 (9.02)	9.58 (8.82)	9.58 (8.91)	9.58 (9.09)	9.58 (8.98)	
Fri Apr  9 06:40:38 2021
**PERF:  8.89 (8.93)	9.40 (8.85)	9.52 (8.93)	9.40 (9.07)	9.40 (8.99)	
Fri Apr  9 06:40:43 2021
**PERF:  9.38 (9.06)	9.30 (8.97)	9.19 (9.04)	9.30 (9.17)	9.30 (9.09)	
Fri Apr  9 06:40:48 2021
**PERF:  9.01 (9.03)	9.01 (8.95)	9.01 (9.01)	8.68 (9.08)	9.01 (9.06)	
Fri Apr  9 06:40:53 2021
**PERF:  9.35 (9.07)	9.35 (9.00)	9.35 (9.05)	9.47 (9.13)	9.15 (9.07)	
Fri Apr  9 06:40:58 2021
**PERF:  9.19 (9.10)	9.19 (9.04)	9.19 (9.08)	9.19 (9.16)	8.99 (9.08)	
Fri Apr  9 06:41:03 2021
**PERF:  9.24 (9.11)	9.04 (9.03)	8.63 (9.04)	8.83 (9.13)	9.04 (9.08)	
Fri Apr  9 06:41:08 2021
**PERF:  8.61 (9.05)	8.61 (8.98)	9.02 (9.02)	9.02 (9.10)	9.02 (9.05)	
Fri Apr  9 06:41:13 2021
**PERF:  9.55 (9.09)	9.55 (9.03)	9.55 (9.06)	9.55 (9.14)	9.55 (9.09)	
Fri Apr  9 06:41:18 2021
**PERF:  9.18 (9.09)	9.15 (9.04)	9.35 (9.09)	8.95 (9.14)	9.15 (9.11)	
Fri Apr  9 06:41:23 2021
**PERF:  9.32 (9.12)	9.35 (9.06)	9.15 (9.09)	9.15 (9.13)	9.36 (9.12)	
Fri Apr  9 06:41:28 2021
**PERF:  9.04 (9.12)	9.04 (9.07)	9.04 (9.10)	9.24 (9.15)	8.65 (9.10)	
Fri Apr  9 06:41:33 2021
**PERF:  9.39 (9.12)	9.07 (9.06)	9.07 (9.08)	9.07 (9.13)	9.27 (9.09)	
Fri Apr  9 06:41:38 2021
**PERF:  9.34 (9.14)	9.25 (9.07)	9.06 (9.09)	9.25 (9.14)	9.25 (9.11)	
Fri Apr  9 06:41:43 2021
**PERF:  8.65 (9.11)	8.85 (9.05)	9.06 (9.07)	9.06 (9.13)	9.06 (9.10)	
Fri Apr  9 06:41:48 2021
**PERF:  8.98 (9.11)	9.17 (9.07)	9.54 (9.09)	9.17 (9.14)	9.17 (9.11)
.
.
.
Fri Apr  9 08:58:49 2021
**PERF:  8.80 (8.90)	8.80 (8.89)	9.00 (8.90)	8.39 (8.90)	8.59 (8.90)	
Fri Apr  9 08:58:54 2021
**PERF:  9.15 (8.90)	8.95 (8.89)	9.15 (8.90)	9.15 (8.90)	9.15 (8.90)	
Fri Apr  9 08:58:59 2021
**PERF:  9.11 (8.90)	8.83 (8.89)	8.83 (8.90)	9.04 (8.90)	9.04 (8.90)	
Fri Apr  9 08:59:04 2021
**PERF:  8.65 (8.90)	9.34 (8.90)	8.73 (8.90)	9.13 (8.90)	9.13 (8.90)	
Fri Apr  9 08:59:09 2021
**PERF:  9.06 (8.90)	8.86 (8.90)	9.06 (8.90)	9.06 (8.90)	8.86 (8.90)	
Segmentation fault

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

I have run the case for 2 hours. It works well. Can you provide a stable reproduce method?

We switched to a new AGX and the problem was solved. I’m guessing it was due to rtsp connection issue from the source with the previous device because recently I ran into a test video(mp4) that gave segmentation fault. So, it should be the consequence of bad data that was produced in earlier. Thank you for your support, you do great work.