Segmentation fault in running pipeline with yolov3 AGX

kd637 · April 8, 2021, 8:11am

•Hardware Platform : Jetson
•DeepStream Version 5.0
•JetPack Version 4.4 [L4T 32.4.3]
•TensorRT Version 7.1.3
•Issue Type(question/bugs)
Using deepstream-test5 with yolov3 that is bundled with DS over 5 RTSP cameras is throwing a segmentation fault after pipeline ran for some time. It goes something like:

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***

Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
[NvDCF] Initialized
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Deserialize yoloLayerV3 plugin: yolo_83
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
INFO: [Implicit Engine Info]: layers num: 4
0   INPUT  kFLOAT data            3x608x608       
1   OUTPUT kFLOAT yolo_83         255x19x19       
2   OUTPUT kFLOAT yolo_95         255x38x38       
3   OUTPUT kFLOAT yolo_107        255x76x76       


Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:181>: Pipeline ready
Thu Apr  8 03:40:21 2021
**PERF:  0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	

Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
** INFO: <bus_callback:167>: Pipeline running

WARNING; playback mode used with URI [rtsp://****/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://****/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://****/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://****/cam0_0] not conforming to timestamp format; check README; using system-time
Thu Apr  8 03:40:26 2021
**PERF:  9.56 (9.29)	10.38 (10.22)	10.07 (9.80)	10.06 (9.78)	9.26 (9.09)	
.
After few hours
.
Thu Apr  8 04:26:31 2021
**PERF:  10.11 (9.69)	10.21 (10.08)	10.28 (10.02)	10.11 (9.90)	10.37 (9.69)	
Quitting
Segmentation fault

What can be the possible causes for this?

Fiona.Chen · April 8, 2021, 7:30am

No clue in your description.

kd637 · April 8, 2021, 8:10am

updated, lmk if it helps

Fiona.Chen · April 8, 2021, 8:31am

Where did you got the yolov3 models? Are you running the case as described in /opt/nvidia/deepstream/deepstream-5.1/sources/objectDetector_Yolo/README?

kd637 · April 8, 2021, 8:38am

Yes. Same. Just within deepstream-test5 and 5 rtsp sources.

Fiona.Chen · April 9, 2021, 3:01am

Can you send us your deepstream-test5 config file and yolov3 nvinfer config file?

kd637 · April 9, 2021, 1:25pm

config_infer_primary_yoloV3.txt (3.4 KB) test.txt (8.1 KB)
Here are the files. Here’s some more info: I tried clearing gstreamer cache. There are other processes running on the AGX too: 5 python processes, kafka broker and mongod. Mongo’s dbpath is set to an external SDcard but its mounted inside the deepstream-test5 folder.

Still having the same issue:

 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***

Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
[NvDCF] Initialized
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Deserialize yoloLayerV3 plugin: yolo_83
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
0:00:04.761050590 21867   0x5565f2c440 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test5/configs/model_b8_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 4
0   INPUT  kFLOAT data            3x608x608       
1   OUTPUT kFLOAT yolo_83         255x19x19       
2   OUTPUT kFLOAT yolo_95         255x38x38       
3   OUTPUT kFLOAT yolo_107        255x76x76       

0:00:04.761338188 21867   0x5565f2c440 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test5/configs/model_b8_gpu0_int8.engine
0:00:04.828050418 21867   0x5565f2c440 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test5/configs/config_infer_primary_yoloV3.txt sucessfully

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:181>: Pipeline ready

Fri Apr  9 06:40:03 2021
**PERF:  0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:167>: Pipeline running

H264: Profile = 66, Level = 0 
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.202/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.206/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.204/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.205/cam0_0] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [rtsp://***:***@192.168.1.203/cam0_0] not conforming to timestamp format; check README; using system-time
Fri Apr  9 06:40:08 2021
**PERF:  8.10 (7.89)	7.25 (7.09)	8.10 (7.89)	9.39 (9.15)	8.56 (8.39)	
Fri Apr  9 06:40:13 2021
**PERF:  9.32 (8.62)	9.17 (8.19)	9.12 (8.51)	9.08 (8.81)	9.17 (8.73)	
Fri Apr  9 06:40:18 2021
**PERF:  9.14 (8.89)	9.14 (8.63)	9.14 (8.82)	9.04 (9.11)	8.95 (8.89)	
Fri Apr  9 06:40:23 2021
**PERF:  9.27 (8.92)	9.04 (8.73)	8.84 (8.82)	8.84 (9.03)	8.84 (8.87)	
Fri Apr  9 06:40:28 2021
**PERF:  8.93 (8.98)	8.55 (8.70)	8.75 (8.81)	8.95 (9.02)	8.95 (8.90)	
Fri Apr  9 06:40:33 2021
**PERF:  9.38 (9.02)	9.58 (8.82)	9.58 (8.91)	9.58 (9.09)	9.58 (8.98)	
Fri Apr  9 06:40:38 2021
**PERF:  8.89 (8.93)	9.40 (8.85)	9.52 (8.93)	9.40 (9.07)	9.40 (8.99)	
Fri Apr  9 06:40:43 2021
**PERF:  9.38 (9.06)	9.30 (8.97)	9.19 (9.04)	9.30 (9.17)	9.30 (9.09)	
Fri Apr  9 06:40:48 2021
**PERF:  9.01 (9.03)	9.01 (8.95)	9.01 (9.01)	8.68 (9.08)	9.01 (9.06)	
Fri Apr  9 06:40:53 2021
**PERF:  9.35 (9.07)	9.35 (9.00)	9.35 (9.05)	9.47 (9.13)	9.15 (9.07)	
Fri Apr  9 06:40:58 2021
**PERF:  9.19 (9.10)	9.19 (9.04)	9.19 (9.08)	9.19 (9.16)	8.99 (9.08)	
Fri Apr  9 06:41:03 2021
**PERF:  9.24 (9.11)	9.04 (9.03)	8.63 (9.04)	8.83 (9.13)	9.04 (9.08)	
Fri Apr  9 06:41:08 2021
**PERF:  8.61 (9.05)	8.61 (8.98)	9.02 (9.02)	9.02 (9.10)	9.02 (9.05)	
Fri Apr  9 06:41:13 2021
**PERF:  9.55 (9.09)	9.55 (9.03)	9.55 (9.06)	9.55 (9.14)	9.55 (9.09)	
Fri Apr  9 06:41:18 2021
**PERF:  9.18 (9.09)	9.15 (9.04)	9.35 (9.09)	8.95 (9.14)	9.15 (9.11)	
Fri Apr  9 06:41:23 2021
**PERF:  9.32 (9.12)	9.35 (9.06)	9.15 (9.09)	9.15 (9.13)	9.36 (9.12)	
Fri Apr  9 06:41:28 2021
**PERF:  9.04 (9.12)	9.04 (9.07)	9.04 (9.10)	9.24 (9.15)	8.65 (9.10)	
Fri Apr  9 06:41:33 2021
**PERF:  9.39 (9.12)	9.07 (9.06)	9.07 (9.08)	9.07 (9.13)	9.27 (9.09)	
Fri Apr  9 06:41:38 2021
**PERF:  9.34 (9.14)	9.25 (9.07)	9.06 (9.09)	9.25 (9.14)	9.25 (9.11)	
Fri Apr  9 06:41:43 2021
**PERF:  8.65 (9.11)	8.85 (9.05)	9.06 (9.07)	9.06 (9.13)	9.06 (9.10)	
Fri Apr  9 06:41:48 2021
**PERF:  8.98 (9.11)	9.17 (9.07)	9.54 (9.09)	9.17 (9.14)	9.17 (9.11)
.
.
.
Fri Apr  9 08:58:49 2021
**PERF:  8.80 (8.90)	8.80 (8.89)	9.00 (8.90)	8.39 (8.90)	8.59 (8.90)	
Fri Apr  9 08:58:54 2021
**PERF:  9.15 (8.90)	8.95 (8.89)	9.15 (8.90)	9.15 (8.90)	9.15 (8.90)	
Fri Apr  9 08:58:59 2021
**PERF:  9.11 (8.90)	8.83 (8.89)	8.83 (8.90)	9.04 (8.90)	9.04 (8.90)	
Fri Apr  9 08:59:04 2021
**PERF:  8.65 (8.90)	9.34 (8.90)	8.73 (8.90)	9.13 (8.90)	9.13 (8.90)	
Fri Apr  9 08:59:09 2021
**PERF:  9.06 (8.90)	8.86 (8.90)	9.06 (8.90)	9.06 (8.90)	8.86 (8.90)	
Segmentation fault

Fiona.Chen · April 27, 2021, 12:39am

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

I have run the case for 2 hours. It works well. Can you provide a stable reproduce method?

kd637 · May 18, 2021, 10:01am

We switched to a new AGX and the problem was solved. I’m guessing it was due to rtsp connection issue from the source with the previous device because recently I ran into a test video(mp4) that gave segmentation fault. So, it should be the consequence of bad data that was produced in earlier. Thank you for your support, you do great work.