Custom Yolov3-tiny ota model update, segmentation fault

mehmetduman · September 15, 2020, 2:26am

Hardware Platform (Jetson / GPU) :- Jetson Nano
• DeepStream Version :- 5.0
• JetPack Version (valid for Jetson only) - 4.4 [L4T 32.4.3]
• TensorRT Version :- 7.1.3.0
• CUDA : 10.2.89

I am working on the deepstream-test5 app to try out the OTA functionality on custom yolov3-tiny model. I tried the test file by just adding a white space at the end of the file and saved it, and it worked perfect (/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test5/configs/test5_ota_override_config.txt)

When I tried my code, the deepstream-test5-app detects a change in the ota override file, and update the model succesfully, but than i get ‘Segmentation fault (core dumped)’. I used the same custom yolov3-tiny model engine file i run on deepstream-test5-app and just add a white space on ota override config file.

Config File
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
custom-network-config=models/custom_yolov3_tiny.cfg
model-file=custom_yolov3_tiny.weights
model-engine-file=custom_yolov3_tiny_model_b1_gpu0_fp16.engine
labelfile-path=custom_yolov3_tiny.txt
batch-size=1
process-mode=1
#0=RGB, 1=BGR
model-color-format=0
##0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=2
interval=19
gie-unique-id=1
parse-bbox-func-name=NvDsInferParseCustomYoloV3Tiny
custom-lib-path=…/…/src/objectDetector_Yolo/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo_2class.so
engine-create-func-name=NvDsInferYoloCudaEngineGet

OTA config
model-engine-file=/home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny_model_b1_gpu0_fp16.engine
labelfile-path=/home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny.txt
config-file=/home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/config_infer_primary_custom_yolov3_tiny.txt

ERROR
nano@nano:~/vap/deepstream5$ deepstream-test5-app -c deepstream_app_config.txt -o test5_ota_override_config.txt
REAL PATH = /home/nano/vap/deepstream5/test5_ota_override_config.txt
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:06.716277142 21360 0x26a05290 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny_model_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x416x416
1 OUTPUT kFLOAT yolo_17 21x13x13
2 OUTPUT kFLOAT yolo_24 21x26x26

0:00:06.717009815 21360 0x26a05290 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny_model_b1_gpu0_fp16.engine
0:00:06.726807796 21360 0x26a05290 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/config_infer_primary_custom_yoloV3_tiny.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

    p: Pause
    r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg) FPS 4 (Avg) FPS 5 (Avg)
Mon Sep 14 21:40:57 2020
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
** INFO: <bus_callback:167>: Pipeline running

KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/xxx.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/yyy.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/zzz.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/ddd.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/vvvv.mp4] not conforming to timestamp format; check README; using system-time
Mon Sep 14 21:41:02 2020
**PERF: 33.37 (32.62) 33.37 (32.62) 33.37 (32.62) 33.37 (32.62) 33.37 (32.62) 33.37 (32.62)
File test5_ota_override_config.txt modified.

New Model Update Request primary_gie ----> /home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny_model_b1_gpu0_fp16.engine
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:14.109801595 21360 0x7ed801f200 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny_model_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x416x416
1 OUTPUT kFLOAT yolo_17 21x13x13
2 OUTPUT kFLOAT yolo_24 21x26x26

0:00:14.110406297 21360 0x7ed801f200 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny_model_b1_gpu0_fp16.engine
0:00:14.183509142 21360 0x264f0320 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny_model_b1_gpu0_fp16.engine sucessfully

Model Update Status: Updated model : /home/nano/vap/deepstream5/models/custom_yolov3_tiny_ggg/custom_yolov3_tiny_model_b1_gpu0_fp16.engine, OTATime = 2098.045000 ms, result: ok

Segmentation fault (core dumped)

AastaLLL · September 15, 2020, 4:13am

Hi,

We are trying to reproduce this issue.
Will update more information with you later.

AastaLLL · September 16, 2020, 9:30am

Hi,

We try to reproduce this issue but the OTA function works good in our environment.
Would you mind to try if the replaced model can work with deepstream correctly first?

Thanks.

mehmetduman · September 16, 2020, 9:31pm

Thank you for the quick response.

I tried the Primary Detector caffe model (/opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector) and ota worked perfectly fine when I add a whitespace on override file (2)

Then I downloaded yolov3-tiny.cfg and yolov3-tiny.weights files by following README (/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/README). Updated class number (static const int NUM_CLASSES_YOLO = 80;) and compiled it , and created .so shared file.

$ export CUDA_VER=10.2
$ make -C nvdsinfer_custom_impl_Yolo

I moved all files (.cfg, .weights, .label and .so file) into same folder and run deepstream-test5-app with -o option. When I changed the override file, deepstream catched the changes, but stopped by segmentation error (1)

Please let me know if you need more information and thanks for your help.

(1) Segmentation fault
nano@nano:~/vap/deepstream5$ deepstream-test5-app -c deepstream_app_config_6source_cartruck.txt -t -o test5_ota_override_config.txt

Gstreamer pipeline element nvinfer is yet to be created or invalid
Gstreamer pipeline element nvinfer is yet to be created or invalid
Gstreamer pipeline element nvinfer is yet to be created or invalid
Gstreamer pipeline element nvinfer is yet to be created or invalid
Gstreamer pipeline element nvinfer is yet to be created or invalid
Gstreamer pipeline element nvinfer is yet to be created or invalid
Gstreamer pipeline element nvinfer is yet to be created or invalid
Gstreamer pipeline element nvinfer is yet to be created or invalid
Gstreamer pipeline element nvinfer is yet to be created or invalid
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:05.996487265 15622 0x15650cc0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x416x416
1 OUTPUT kFLOAT yolo_17 255x13x13
2 OUTPUT kFLOAT yolo_24 255x26x26

0:00:05.996691695 15622 0x15650cc0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
0:00:06.030862380 15622 0x15650cc0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/yolov3_tiny/config_infer_primary_yolov3_tiny.txt sucessfully

Model Update Status: Updated model : /home/nano/vap/deepstream5/models/yolov3_tiny/config_infer_primary_yolov3_tiny.txt, OTATime = 1600290204403.004883 ms, result: ok

Runtime commands:
h: Print this help
q: Quit

    p: Pause
    r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg) FPS 4 (Avg) FPS 5 (Avg)
Wed Sep 16 17:03:24 2020
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
** INFO: <bus_callback:167>: Pipeline running

KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2771_2017-08-10_0235_N_FFT_Trim_1.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/hend_3.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/fake_2783_2017-08-09_2230_N_FFT_Trim_2_1min.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2754_2017-08-10_0027_N_FFT_Trim_1_5min_1.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2779_2017-08-10_0134_N_FFT_Trim_2.mp4] not conforming to timestamp format; check README; using system-time
Wed Sep 16 17:03:29 2020
**PERF: 27.44 (27.26) 27.44 (27.26) 27.44 (27.26) 27.44 (27.26) 27.44 (27.26) 27.44 (27.26)
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2783_2017-08-09_2230_N_FFT_Trim_1.mp4] not conforming to timestamp format; check README; using system-time
Wed Sep 16 17:03:34 2020
**PERF: 26.03 (26.59) 26.03 (26.59) 26.03 (26.59) 26.03 (26.59) 26.03 (26.59) 26.03 (26.59)
File test5_ota_override_config.txt modified.

New Model Update Request primary_gie ----> /home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:13.546191656 16192 0x7ecc1270d0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x416x416
1 OUTPUT kFLOAT yolo_17 255x13x13
2 OUTPUT kFLOAT yolo_24 255x26x26

0:00:13.546603330 16192 0x7ecc1270d0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
0:00:13.609942618 16192 0x6f6e190 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine sucessfully

Model Update Status: Updated model : /home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine, OTATime = 1109.016000 ms, result: ok

Segmentation fault (core dumped)
.

(2)

config_infer_primary_yolov3_tiny.txt

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-file=resnet10.caffemodel
proto-file=resnet10.prototxt
model-engine-file=resnet10.caffemodel_b1_gpu0_fp16.engine
labelfile-path=labels.txt
int8-calib-file=cal_trt.bin
batch-size=1
process-mode=1
#0=RGB, 1=BGR
model-color-format=0
##0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
num-detected-classes=4
interval=19
gie-unique-id=1
output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
force-implicit-batch-dim=1

Terminal - Model Updated

…
Wed Sep 16 16:55:00 2020
**PERF: 45.36 (45.34) 45.36 (45.34) 45.36 (45.34) 45.36 (45.34) 45.36 (45.34) 45.36 (45.34)
File test5_ota_override_config.txt modified.

New Model Update Request primary_gie ----> /home/nano/vap/deepstream5/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_fp16.engine
0:00:32.923954435 15390 0x7ec0710a10 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x368x640
1 OUTPUT kFLOAT conv2d_bbox 16x23x40
2 OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40

0:00:32.924253499 15390 0x7ec0710a10 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_fp16.engine
0:00:32.946766060 15390 0x557e990 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_fp16.engine sucessfully

Model Update Status: Updated model : /home/nano/vap/deepstream5/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_fp16.engine, OTATime = 412.229000 ms, result: ok

Wed Sep 16 16:55:05 2020
**PERF: 43.95 (45.12) 43.95 (45.12) 43.95 (45.12) 43.95 (45.12) 43.95 (45.12) 43.95 (45.12)
q
Quitting
App run successful

mehmetduman · September 16, 2020, 10:17pm

I also tried YoloV3 and got same Segmentation fault …

The caffe model works, but yolov3 and yolov3-tiny doesn’t !

nano@nano:~/vap/deepstream5$ deepstream-test5-app -c deepstream_app_config_6source_cartruck.txt -t -o test5_ota_override_config.txt
REAL PATH = /home/nano/vap/deepstream5/test5_ota_override_config.txt
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
Deserialize yoloLayerV3 plugin: yolo_83
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
0:00:16.875511124 3024 0x7f34002240 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/yolov3/model_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 4
0 INPUT kFLOAT data 3x608x608
1 OUTPUT kFLOAT yolo_83 255x19x19
2 OUTPUT kFLOAT yolo_95 255x38x38
3 OUTPUT kFLOAT yolo_107 255x76x76

0:00:16.875782848 3024 0x7f34002240 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/yolov3/model_b1_gpu0_fp16.engine
0:00:17.025639183 3024 0x7f34002240 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/yolov3/config_infer_primary_yolov3.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

    p: Pause
    r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg) FPS 4 (Avg) FPS 5 (Avg)
Wed Sep 16 18:11:05 2020
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
** INFO: <bus_callback:167>: Pipeline running

KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/hend_3.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2783_2017-08-09_2230_N_FFT_Trim_1.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2771_2017-08-10_0235_N_FFT_Trim_1.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2779_2017-08-10_0134_N_FFT_Trim_2.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2754_2017-08-10_0027_N_FFT_Trim_1_5min_1.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/fake_2783_2017-08-09_2230_N_FFT_Trim_2_1min.mp4] not conforming to timestamp format; check README; using system-time
Wed Sep 16 18:11:10 2020
**PERF: 53.84 (4.80) 53.84 (4.80) 53.84 (4.80) 53.84 (4.80) 53.84 (4.80) 53.84 (4.80)
Wed Sep 16 18:11:15 2020
**PERF: 3.73 (4.22) 3.73 (4.22) 3.73 (4.22) 3.73 (4.22) 3.73 (4.22) 3.73 (4.22)
Wed Sep 16 18:11:20 2020
**PERF: 3.72 (4.13) 3.72 (4.13) 3.72 (4.13) 3.72 (4.13) 3.72 (4.13) 3.72 (4.13)
Wed Sep 16 18:11:25 2020
**PERF: 3.72 (4.09) 3.72 (4.09) 3.72 (4.09) 3.72 (4.09) 3.72 (4.09) 3.72 (4.09)
Wed Sep 16 18:11:30 2020
**PERF: 3.71 (4.07) 3.71 (4.07) 3.71 (4.07) 3.71 (4.07) 3.71 (4.07) 3.71 (4.07)
File test5_ota_override_config.txt modified.

New Model Update Request primary_gie ----> /home/nano/vap/deepstream5/models/yolov3/model_b1_gpu0_fp16.engine
Wed Sep 16 18:11:35 2020
**PERF: 3.70 (3.68) 3.70 (3.68) 3.70 (3.68) 3.70 (3.68) 3.70 (3.68) 3.70 (3.68)
Wed Sep 16 18:11:40 2020
**PERF: 3.44 (3.73) 3.44 (3.73) 3.44 (3.73) 3.44 (3.73) 3.44 (3.73) 3.44 (3.73)
Deserialize yoloLayerV3 plugin: yolo_83
Wed Sep 16 18:11:45 2020
**PERF: 3.65 (3.77) 3.65 (3.77) 3.65 (3.77) 3.65 (3.77) 3.65 (3.77) 3.65 (3.77)
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
0:00:58.954565094 3024 0x536e9c60 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/yolov3/model_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 4
0 INPUT kFLOAT data 3x608x608
1 OUTPUT kFLOAT yolo_83 255x19x19
2 OUTPUT kFLOAT yolo_95 255x38x38
3 OUTPUT kFLOAT yolo_107 255x76x76

0:00:59.008610061 3024 0x536e9c60 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/yolov3/model_b1_gpu0_fp16.engine
Wed Sep 16 18:11:50 2020
**PERF: 3.46 (3.56) 3.46 (3.56) 3.46 (3.56) 3.46 (3.56) 3.46 (3.56) 3.46 (3.56)
0:01:03.493861629 3024 0x2df85050 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/yolov3/model_b1_gpu0_fp16.engine sucessfully

Model Update Status: Updated model : /home/nano/vap/deepstream5/models/yolov3/model_b1_gpu0_fp16.engine, OTATime = 17872.016000 ms, result: ok

Segmentation fault (core dumped)
nano@nano:~/vap/deepstream5$

Config File

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
custom-network-config=models/yolov3/yolov3.cfg
model-file=yolov3.weights
##*****_b_precision.enginef
model-engine-file=model_b1_gpu0_fp16.engine
labelfile-path=yolov3.txt
batch-size=1
process-mode=1
#0=RGB, 1=BGR
model-color-format=0
##0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=80
interval=9
gie-unique-id=1
##0=Group Rectangles, 1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering)
#cluster-mode=2
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseCustomYoloV3
custom-lib-path=libnvdsinfer_custom_impl_Yolo.so
engine-create-func-name=NvDsInferYoloCudaEngineGet

[class-attrs-all]
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

mehmetduman · September 17, 2020, 2:00pm

Additional tests with Docker (nvcr.io/nvidia/deepstream-l4t:5.0-20.07-iot) ;

I thought, it would make sense to do some test in container to eliminate environment issues.

I ran Yolov3 (80 class), Yolov3-Tiny (80 class) and caffe model (/opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector) on docker container and all three models ran perfect.

When I tried the OTA with those 3 models, all three started and did inferences until I update override file.The Yolov3 and Yolov3-Tiny models had same issue as above, after updated the model raised Segmentation fault. The Caffe model updated the OTA model and ran without any issue.

It looks like OTA has some issues with YoloV3 models.

AastaLLL · September 18, 2020, 4:20am

Hi,

Thanks for the update.

We try YOLOv3-tiny in a similar way as objectDetector_Yolo and it works.
Will check if anything different between our testing and give you an update.

Thanks.

AastaLLL · September 21, 2020, 9:06am

Hi,

We try to reproduce this issue with our deepstream container but the OTA process works good in our environment.

$ deepstream-test5-app -c deepstream_app_config_yoloV3_tiny.txt -o test5_ota_override_config.txt 
REAL PATH = /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/test5_ota_override_config.txt
Unknown or legacy key specified 'is-classifier' for group [property]
Warn: 'threshold' parameter has been deprecated. Use 'pre-cluster-threshold' instead.

Using winsys: x11 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:03.019099271  1315     0x15645c40 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/model_b1_gpu0_fp32.engine
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT data            3x416x416       
1   OUTPUT kFLOAT yolo_17         255x13x13       
2   OUTPUT kFLOAT yolo_24         255x26x26       

0:00:03.019328050  1315     0x15645c40 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/model_b1_gpu0_fp32.engine
0:00:03.023923374  1315     0x15645c40 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/config_infer_primary_yoloV3_tiny.txt sucessfully

Model Update Status: Updated model : /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/config_infer_primary_yoloV3_tiny.txt, OTATime = 1600678818764.145020 ms, result: ok 


Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:181>: Pipeline ready

Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:167>: Pipeline running

KLT Tracker Init
WARNING; playback mode used with URI [file:///opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/../../samples/streams/sample_1080p_h264.mp4] not conforming to timestamp format; check README; using system-time

**PERF:  FPS 0 (Avg)	
Mon Sep 21 09:00:20 2020
**PERF:  51.80 (51.75)	
...

May I know the class number of original YOLOv3_tiny and OTA YOLOv3_tiny first?
If the class number changes, the OTA will not work since the network architecture need to be updated.

We test this with default #class=8 (both model) and it can work correctly.

Thanks.

mehmetduman · September 22, 2020, 4:19am

Hi,

First, I used 2 custom YoloV3-Tiny custom models which both has same 2 classes and same architecture, and and got segmentation fault.

Then, I used official YoloV3-Tiny.weights model which has 80 classes and got segmentation fault

Downloaded by following README in (/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/README)

Then, I used official YoloV3.weights model which has 80 classes and got segmentation fault

Downloaded by following README in (/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/README)

Then, I said maybe my Jetson Nano environment has some issue and I tried the official YoloV3 and YoloV3-Tiny models on Docker container and got same segmentation fault.

How I created the YoloV3/YoloV3-Tiny .so shared library:

Followed the README in (in (/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/README)

static const int NUM_CLASSES_YOLO = 80
$ export CUDA_VER=10.2
$ make -C nvdsinfer_custom_impl_Yolo

Run Process

Run the deepstream-test5-app,
deepstream-test5-app -c deepstream_app_config_6source_cartruck.txt -t -o test5_ota_override_config.txt
The app applies the OTA update at starting just before the “Runtime commands” (Please see below output).
I add a white-space to the override file, use the same model that I run the deepstream-test5-app and save it
The deepstream app catch the changes on override file and start to model update (Please see below output)
The deepstream app complete the model update (Please see below output)
Raises Segmentation fault.

----------------

**Output **

$ deepstream-test5-app -c deepstream_app_config_6source_cartruck.txt -t -o test5_ota_override_config.txt
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:05.996487265 15622 0x15650cc0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x416x416
1 OUTPUT kFLOAT yolo_17 255x13x13
2 OUTPUT kFLOAT yolo_24 255x26x26
0:00:05.996691695 15622 0x15650cc0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
0:00:06.030862380 15622 0x15650cc0 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/yolov3_tiny/config_infer_primary_yolov3_tiny.txt sucessfully
Model Update Status: Updated model : /home/nano/vap/deepstream5/models/yolov3_tiny/config_infer_primary_yolov3_tiny.txt, OTATime = 1600290204403.004883 ms, result: ok

Runtime commands:
h: Print this help
q: Quit
p: Pause
r: Resume
NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.
**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg) FPS 4 (Avg) FPS 5 (Avg)
Wed Sep 16 17:03:24 2020
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
** INFO: <bus_callback:167>: Pipeline running
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2771_2017-08-10_0235_N_FFT_Trim_1.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/hend_3.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/fake_2783_2017-08-09_2230_N_FFT_Trim_2_1min.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2754_2017-08-10_0027_N_FFT_Trim_1_5min_1.mp4] not conforming to timestamp format; check README; using system-time
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2779_2017-08-10_0134_N_FFT_Trim_2.mp4] not conforming to timestamp format; check README; using system-time
Wed Sep 16 17:03:29 2020
**PERF: 27.44 (27.26) 27.44 (27.26) 27.44 (27.26) 27.44 (27.26) 27.44 (27.26) 27.44 (27.26)
WARNING; playback mode used with URI [file:///home/nano/vap/deepstream5/./videos/2783_2017-08-09_2230_N_FFT_Trim_1.mp4] not conforming to timestamp format; check README; using system-time
Wed Sep 16 17:03:34 2020
**PERF: 26.03 (26.59) 26.03 (26.59) 26.03 (26.59) 26.03 (26.59) 26.03 (26.59) 26.03 (26.59)
File test5_ota_override_config.txt modified.
New Model Update Request primary_gie ----> /home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
0:00:13.546191656 16192 0x7ecc1270d0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT data 3x416x416
1 OUTPUT kFLOAT yolo_17 255x13x13
2 OUTPUT kFLOAT yolo_24 255x26x26
0:00:13.546603330 16192 0x7ecc1270d0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine
0:00:13.609942618 16192 0x6f6e190 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine sucessfully
Model Update Status: Updated model : /home/nano/vap/deepstream5/models/yolov3_tiny/model_b1_gpu0_fp16.engine, OTATime = 1109.016000 ms, result: ok
Segmentation fault (core dumped)

AastaLLL · September 22, 2020, 5:54am

Hi,

Thanks for your detail update.
We will try this on another device to see if we can reproduce this issue or not.

Thanks.

AastaLLL · September 23, 2020, 8:50am

Hi,

We found this may be a memory issue.
Would you mind to check the memory status with tegrastats when running the OTA process?

$ sudo tegrastats

Thanks.

mehmetduman · September 24, 2020, 2:05pm

Hi,

Here are the $ sudo tegrastats result. I also added an Ubuntu internal error which I got first time. Hope these would help to resolve the problem

mehmetduman · September 24, 2020, 2:24pm

cuDNN version:

AastaLLL · September 25, 2020, 2:58am

Hi,

Based on your tegrastats log here, the failure may cause by out of memory.

Would you mind to add 4G swap space and try it again?

Thanks.

mehmetduman · September 25, 2020, 5:17am

Hi, I have added 8GB swap space, but still getting segmentation fault (core dumped)

AastaLLL · September 28, 2020, 7:03am

Hi,

Thanks for your testing.

We are discussing the possible cause internally.
Will share more information with you later.

AastaLLL · September 29, 2020, 8:55am

Hi,

Thanks for your patience.

It seems that some steps are incorrect in our previous testing.
Now, we can reproduce this error in our environment.

We are check this with our internal team.
Will update more information with you once we got a feedback.

Thanks.

AastaLLL · October 28, 2020, 2:52am

Hi,

Thanks for your patience.

The root cause of segmentation fault is from the custom parser lib.
Please noted that OTA is an alpha feature. We will provide a fix in a future release.

Thanks.