Segmentation Fault in Docker Deepstream Tracker

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
dGPU: NVIDIA RTX 3070

• DeepStream Version
nvcr.io/nvidia/deepstream:5.1-21.01-devel

• TensorRT Version
Not quite sure what TensorRT version has in the container…

• NVIDIA GPU Driver Version (valid for GPU only)
460.39

• Issue Type( questions, new requirements, bugs)
Bugs

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

Config

Deepstream

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=1

[tiled-display]
enable=1
rows=1
columns=1
width=1280 #640
height=960 #480
gpu-id=0
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
num-sources=1
uri=file://…/…/streams/face-detection-sample.mp4
gpu-id=0
cudadec-memtype=0

[streammux]
gpu-id=0
batch-size=1
batched-push-timeout=40000
##Set muxer output width and height
width=1920
height=1080

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
sync=0
#iframeinterval=10
bitrate=2000000
output-file=out.mp4
source-id=0

[sink1]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=dstest4_msgconv_config.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM - Custom schema payload
msg-conv-payload-type=0
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_kafka_proto.so
#Provide your msg-broker-conn-str here
msg-broker-conn-str=127.0.0.1;9092
topic=testTopic
#Optional:
#msg-broker-config=…/…/deepstream-test4/cfg_kafka.txt

[sink2]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=400000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=1
source-id=0
gpu-id=0

[osd]
enable=1
gpu-id=0
border-width=4
text-size=18
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial

[primary-gie]
enable=1
gpu-id=0
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=0;1;0;1
bbox-border-color1=1;0;0;1
#bbox-border-color2=0;0;1;1 # Blue
#bbox-border-color3=0;1;0;1
gie-unique-id=1
config-file=config_infer_primary_facenet.txt

[secondary-gie0]
enable=1
gpu-id=0
batch-size=4
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_secondary_facemask.txt

[tracker]
enable=1
#For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_iou.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_nvdcf.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
#ll-config-file=tracker_config.yml
ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process and enable-past-frame applicable to DCF only
enable-batch-process=0
enable-past-frame=0
display-tracking-id=1

[tests]
file-loop=0

Model Config

Facenet - Primary

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
labelfile-path=…/…/models/Primary_FaceDetection/facenet_labels.txt
tlt-encoded-model=…/…/models/Primary_FaceDetection/facenet.etlt
tlt-model-key=nvidia_tlt
infer-dims=3;416;736
uff-input-order=0
uff-input-blob-name=input_1
output-blob-names=output_bbox/BiasAdd;output_cov/Sigmoid
batch-size=1
##0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=1
interval=0
gie-unique-id=1
is-classifier=0
classifier-threshold=0.9

[class-attrs-all]
pre-cluster-threshold=0.3
group-threshold=1
eps=0.2
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

Facemask - Secondary

[property]
gpu-id=0
net-scale-factor=1
model-color-format=0
labelfile-path=…/…/models/Secondary_FaceMaskDetection/facemask_labels.txt
tlt-encoded-model=…/…/models/Secondary_FaceMaskDetection/facemask.etlt
tlt-model-key=[some key]
infer-dims=3;544;960
uff-input-order=0
uff-input-blob-name=input_1
output-blob-names=output_bbox/BiasAdd;output_cov/Sigmoid
batch-size=4
##0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=2
interval=0
gie-unique-id=1
is-classifier=1
classifier-threshold=0.1

Debug Result in Linux with GDB and GST Debug = 1

/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app# gdb --args deepstream-app -c deepstream_app_source1_facedetection.txt --gst-debug=1
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from deepstream-app…(no debugging symbols found)…done.
(gdb) r
Starting program: /usr/bin/deepstream-app -c deepstream_app_source1_facedetection.txt --gst-debug=1
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.
[New Thread 0x7f05ca5b4700 (LWP 1978)]
[New Thread 0x7f05c9db3700 (LWP 1979)]
[Thread 0x7f05c9db3700 (LWP 1979) exited]

(deepstream-app:1974): GLib-GObject-WARNING **: 04:08:54.949: value “TRUE” of type ‘gboolean’ is invalid or out of range for property ‘sync’ of type ‘gboolean’

(deepstream-app:1974): GLib-GObject-WARNING **: 04:08:54.949: value “TRUE” of type ‘gboolean’ is invalid or out of range for property ‘qos’ of type ‘gboolean’
[New Thread 0x7f05af18c700 (LWP 1980)]
[New Thread 0x7f05ae98b700 (LWP 1981)]
[New Thread 0x7f05ade7d700 (LWP 1982)]
[New Thread 0x7f05ad67c700 (LWP 1983)]
[New Thread 0x7f05ace7b700 (LWP 1984)]
[New Thread 0x7f058ffff700 (LWP 1985)]
[New Thread 0x7f058f7fe700 (LWP 1986)]
0:00:00.339374482 1974 0x5654ec0c9e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 4]: Trying to create engine from model files
[New Thread 0x7f058effd700 (LWP 1987)]
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors.
0:00:22.515209717 1974 0x5654ec0c9e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1749> [UID = 4]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.1/samples/models/Secondary_FaceMaskDetection/facemask.etlt_b4_gpu0_fp16.engine successfully
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x544x960
1 OUTPUT kFLOAT output_bbox/BiasAdd 8x34x60
2 OUTPUT kFLOAT output_cov/Sigmoid 2x34x60

[New Thread 0x7f058e7fc700 (LWP 1988)]
[New Thread 0x7f058dffb700 (LWP 1989)]
[New Thread 0x7f058d7fa700 (LWP 1990)]
0:00:22.521212760 1974 0x5654ec0c9e30 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<secondary_gie_0> [UID 4]: Load new model:/opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/config_infer_secondary_facemask.txt sucessfully
[New Thread 0x7f058cff9700 (LWP 1991)]
[New Thread 0x7f0585fff700 (LWP 1992)]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_iou.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
[New Thread 0x7f05857fe700 (LWP 1993)]
0:00:22.547412763 1974 0x5654ec0c9e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors.
0:00:40.029876276 1974 0x5654ec0c9e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1749> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.1/samples/models/Primary_FaceDetection/facenet.etlt_b1_gpu0_fp16.engine successfully
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x416x736
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x26x46
2 OUTPUT kFLOAT output_cov/Sigmoid 1x26x46

[New Thread 0x7f0584ffd700 (LWP 1994)]
[New Thread 0x7f055f872700 (LWP 1995)]
[New Thread 0x7f055f071700 (LWP 1996)]
0:00:40.036382218 1974 0x5654ec0c9e30 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/config_infer_primary_facenet.txt sucessfully
[New Thread 0x7f055e870700 (LWP 1997)]
[New Thread 0x7f0521fff700 (LWP 1998)]
[New Thread 0x7f05217fe700 (LWP 1999)]
0:00:40.037475356 1974 0x5654ec0c9e30 WARN basesrc gstbasesrc.c:3583:gst_base_src_start_complete: pad not activated yet
0:00:40.037686452 1974 0x5654ec0c9e30 WARN basesrc gstbasesrc.c:3583:gst_base_src_start_complete: pad not activated yet
[New Thread 0x7f0520ffd700 (LWP 2000)]

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg)
**PERF: 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

[New Thread 0x7f04c1f61700 (LWP 2001)]
0:00:40.050765616 1974 0x7f048c076680 WARN qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type SDLN
0:00:40.050791574 1974 0x7f048c076680 WARN qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type smrd
0:00:40.050800470 1974 0x7f048c076680 WARN qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type smta
0:00:40.050809136 1974 0x7f048c076680 WARN qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type keys
0:00:40.050818813 1974 0x7f048c076680 WARN qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type …
0:00:40.050828902 1974 0x7f048c076680 WARN qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type pasp
0:00:40.050867813 1974 0x7f048c076680 WARN qtdemux qtdemux.c:3031:qtdemux_parse_trex: failed to find fragment defaults for stream 1
0:00:40.050946928 1974 0x7f048c076680 WARN qtdemux qtdemux.c:3031:qtdemux_parse_trex: failed to find fragment defaults for stream 2
[New Thread 0x7f04c1760700 (LWP 2002)]
[New Thread 0x7f04c0f5f700 (LWP 2003)]
0:00:40.270244121 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270260230 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2924:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MJPG
0:00:40.270266953 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270272773 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2930:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MJPG
0:00:40.270293301 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270303069 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2924:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG4
0:00:40.270310883 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270317876 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2930:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG4
0:00:40.270329697 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270336009 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2924:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG2
0:00:40.270342982 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270349363 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2930:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG2
0:00:40.270364511 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270372445 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2924:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H265
0:00:40.270379598 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270387252 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2930:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H265
0:00:40.270400587 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270407489 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2924:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP90
0:00:40.270414673 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270424851 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2930:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP90
0:00:40.270442363 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270449526 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2924:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP80
0:00:40.270456529 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270464073 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2930:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP80
0:00:40.270479601 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270485752 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2924:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H264
0:00:40.270491232 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:40.270496793 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2930:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H264
0:00:40.270962253 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:40.270971941 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2924:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat NM12
0:00:40.270979124 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:3038:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:40.270986598 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2930:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat NM12
0:00:40.270996907 1974 0x7f048000f720 WARN v4l2 gstv4l2object.c:2375:gst_v4l2_object_add_interlace_mode:0x7f048401e020 Failed to determine interlace mode
[New Thread 0x7f0468c20700 (LWP 2004)]
0:00:40.376468955 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376488821 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376522292 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376530387 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376564349 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376576602 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376609422 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376618769 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376651809 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376662238 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376696240 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376706589 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376734470 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376742505 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376775986 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.376787898 1974 0x7f048000f720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

[New Thread 0x7f0463fff700 (LWP 2005)]
** INFO: <bus_callback:167>: Pipeline running

[New Thread 0x7f04637fe700 (LWP 2006)]
0:00:40.379772672 1974 0x7f048000f720 WARN v4l2videodec gstv4l2videodec.c:1673:gst_v4l2_video_dec_decide_allocation: Duration invalid, not setting latency
0:00:40.379846387 1974 0x7f048000f720 WARN v4l2bufferpool gstv4l2bufferpool.c:1066:gst_v4l2_buffer_pool_start:nvv4l2decoder0:pool:src Uncertain or not enough buffers, enabling copy threshold
[New Thread 0x7f0462ffd700 (LWP 2007)]
0:00:40.380603390 1974 0x7f048431dde0 WARN v4l2bufferpool gstv4l2bufferpool.c:1513:gst_v4l2_buffer_pool_dqbuf:nvv4l2decoder0:pool:src Driver should never set v4l2_buffer.field to ANY
[New Thread 0x7f04627fc700 (LWP 2008)]
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/iou_config.txt
IOU Tracker Init with threshold 0.600000

Thread 12 “deepstream-app” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f058e7fc700 (LWP 1988)]
0x00007f0631dc1426 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)
[6]+ Stopped gdb --args deepstream-app -c deepstream_app_source1_facedetection.txt --gst-debug=2
root@hanifs3-desktop:/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app# gdb --args deepstream-app -c deepstream_app_source1_facedetection.txt --gst-debug=1
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from deepstream-app…(no debugging symbols found)…done.
(gdb) r
Starting program: /usr/bin/deepstream-app -c deepstream_app_source1_facedetection.txt --gst-debug=1
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.
[New Thread 0x7f48ad828700 (LWP 2015)]
[New Thread 0x7f48ad027700 (LWP 2016)]
[Thread 0x7f48ad027700 (LWP 2016) exited]

(deepstream-app:2011): GLib-GObject-WARNING **: 04:15:52.037: value “TRUE” of type ‘gboolean’ is invalid or out of range for property ‘sync’ of type ‘gboolean’

(deepstream-app:2011): GLib-GObject-WARNING **: 04:15:52.037: value “TRUE” of type ‘gboolean’ is invalid or out of range for property ‘qos’ of type ‘gboolean’
[New Thread 0x7f488e1cc700 (LWP 2017)]
[New Thread 0x7f488d9cb700 (LWP 2018)]
[New Thread 0x7f488d0bd700 (LWP 2019)]
[New Thread 0x7f488c8bc700 (LWP 2020)]
[New Thread 0x7f487bfff700 (LWP 2021)]
[New Thread 0x7f487b7fe700 (LWP 2022)]
[New Thread 0x7f487affd700 (LWP 2023)]
0:00:00.339132308 2011 0x562337377e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 4]: Trying to create engine from model files
[New Thread 0x7f487a7fc700 (LWP 2024)]
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors.
0:00:22.678108032 2011 0x562337377e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1749> [UID = 4]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.1/samples/models/Secondary_FaceMaskDetection/facemask.etlt_b4_gpu0_fp16.engine successfully
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x544x960
1 OUTPUT kFLOAT output_bbox/BiasAdd 8x34x60
2 OUTPUT kFLOAT output_cov/Sigmoid 2x34x60

[New Thread 0x7f4879ffb700 (LWP 2025)]
[New Thread 0x7f48797fa700 (LWP 2026)]
[New Thread 0x7f4878ff9700 (LWP 2027)]
0:00:22.687921228 2011 0x562337377e30 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<secondary_gie_0> [UID 4]: Load new model:/opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/config_infer_secondary_facemask.txt sucessfully
[New Thread 0x7f4843872700 (LWP 2028)]
[New Thread 0x7f4843071700 (LWP 2029)]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_iou.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
[New Thread 0x7f4842870700 (LWP 2030)]
0:00:22.719413381 2011 0x562337377e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 2 output network tensors.
0:00:39.883786872 2011 0x562337377e30 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1749> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.1/samples/models/Primary_FaceDetection/facenet.etlt_b1_gpu0_fp16.engine successfully
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x416x736
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x26x46
2 OUTPUT kFLOAT output_cov/Sigmoid 1x26x46

[New Thread 0x7f4805fff700 (LWP 2031)]
[New Thread 0x7f48057fe700 (LWP 2032)]
[New Thread 0x7f4804ffd700 (LWP 2033)]
0:00:39.890136721 2011 0x562337377e30 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/config_infer_primary_facenet.txt sucessfully
[New Thread 0x7f47a5f61700 (LWP 2034)]
[New Thread 0x7f47a5760700 (LWP 2035)]
[New Thread 0x7f47a4f5f700 (LWP 2036)]
[New Thread 0x7f4799fff700 (LWP 2037)]

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg)
**PERF: 0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

[New Thread 0x7f47997fe700 (LWP 2038)]
[New Thread 0x7f4798ffd700 (LWP 2039)]
[New Thread 0x7f4763fff700 (LWP 2040)]
[New Thread 0x7f474bcf1700 (LWP 2041)]
0:00:40.285576795 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285603735 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285672472 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285680777 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285713458 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285721743 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285752901 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285762047 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285794538 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285801891 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285834702 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285843969 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285873774 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285881639 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285912225 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

0:00:40.285922895 2011 0x7f476400d720 ERROR v4l2 gstv4l2object.c:2077:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from v4l-utils.git - media (V4L2, DVB and IR) applications and libraries

[New Thread 0x7f474b4f0700 (LWP 2042)]
** INFO: <bus_callback:167>: Pipeline running

[New Thread 0x7f474acef700 (LWP 2043)]
[New Thread 0x7f474a4ee700 (LWP 2044)]
[New Thread 0x7f4749ced700 (LWP 2045)]
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/iou_config.txt
IOU Tracker Init with threshold 0.600000

Thread 12 “deepstream-app” received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f4879ffb700 (LWP 2025)]
0x00007f4915035426 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007f4915035426 in () at /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4915021aee in strdup () at /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f48a50f7d29 in merge_classification_output(_GstNvInferObjectHistory&, GstNvInferObjectInfo&) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
#3 0x00007f48a50dfa4c in gst_nvinfer_output_loop(void*) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
#4 0x00007f491563d175 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5 0x00007f4914b686db in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f49150a571f in clone () at /lib/x86_64-linux-gnu/libc.so.6

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

#0 0x00007f4915035426 in () at /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4915021aee in strdup () at /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f48a50f7d29 in merge_classification_output(_GstNvInferObjectHistory&, GstNvInferObjectInfo&) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
#3 0x00007f48a50dfa4c in gst_nvinfer_output_loop(void*) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so
#4 0x00007f491563d175 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5 0x00007f4914b686db in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f49150a571f in clone () at /lib/x86_64-linux-gnu/libc.so.6

Note

If i dont enable tracker in deepstream config, deepstream-app will works properly

@AastaLLL @dusty_nv
could you guys please help? i’ve been battling this bug for a week with no result…

You can check this link for more information. I’ve met the same problem when using python. Hope this helps.

Can the sample of deepstream-test2 run correctly in your platform?

Your log shows the failure is in merge_classification_output() function in /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvinfer/gstnvinefr.cpp. You can debug with the source code to find the reason.

@ductm thanks for your help, but currently im not using python app to running my deepstream code

@Fiona.Chen i just found the answer!
if i target the secondary-gie into tracker unique id instead of primary-gie unique id, deepstream-app is running properly…

enable=1
gpu-id=0
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=0;1;0;1
bbox-border-color1=1;0;0;1
#bbox-border-color2=0;0;1;1 # Blue
#bbox-border-color3=0;1;0;1
gie-unique-id=1
config-file=config_infer_primary_facenet.txt

[secondary-gie0]
enable=1
gpu-id=0
batch-size=4
gie-unique-id=3
operate-on-gie-id=2
operate-on-class-ids=0;
config-file=config_infer_secondary_facemask.txt

[tracker]
enable=1
#For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_iou.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_nvdcf.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.1/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process and enable-past-frame applicable to DCF only
enable-batch-process=1
enable-past-frame=1
display-tracking-id=1
gie-unique-id=2
operate-on-gie-id=1

i found the idea from here

just for my knowledge, is there any reason why we should put 2nd nvinfer after nvtracker?

Thanks

It is just a sample, in our sample, the SGIEs are all classifiers, tracker is more useful for detectors.

yes, i got SGIES as classifier too…
just for clarity, could we say if SGIE is classifier, it should target to tracker gie-unique-id instead of PGIE gie-unique-id?

Every detector can have a tracker since tracker needs the bbox and object type for tracking algorithm. SGIE(secondary GPU Inference Engine) is a concept which should be together with PGIE(Primary GPU Inference Engine), so SGIE targets to PGIE.

I’ve run into the same problem and the solution was the labels file. If you check the example label files in the ngc containers you’ll find that detection gies have newline-separated labels but classification gies have colon-separated labels. Switching to a single line, colon-separated label file fixed it for me.

This is one of the many issues arising from the severe lack of documentation and consistency across the several otherwise excellent tools that NVidia provides. Whenever I run into silly bugs that torment me for hours/days I call them “DeepStreams”.

Btw, the targeting to the tracker is a no-op. Without the tracker you should observe that it does not crash but it does not do what was intended either (at least my case was as such). The tracker actually exacerbates the issue.

2 Likes