DeepStream SDK 4.0.1 with Nvidia Tesla K40 - deepstream-app throwing Bus error

amertahir · October 7, 2019, 10:18pm

I have deepstream sdk 4.0.1 installed on ubuntu 18.04 and trying to run deepstream-app. My system has a Tesla K40 GPU and I don’t know if it’s supported by deepstream sdk.

Deepstream requirements are met, deepstream-app --version-all returns:

deepstream-app version 4.0.1
DeepStreamSDK 4.0.1
CUDA Driver Version: 10.1
CUDA Runtime Version: 10.1
TensorRT Version: 5.1
cuDNN Version: 7.6
libNVWarp360 Version: 2.0.0d5

nvidia-smi returns:

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

Before running deepstream-app, I modified config_infer_primary.txt config_infer_secondary_carcolor.txt config_infer_secondary_carmake.txt config_infer_secondary_vehicletypes.txt files and set network-mode=0 (i.e. FP32).

I ran deepstream-app with /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt

I changed source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt and enabled sink1 instead of sink0. The following is the output:

amer@amer-server1:~$ deepstream-app -c /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt

(deepstream-app:23824): GLib-GObject-WARNING **: 16:16:54.220: g_object_set_is_valid_property: object class ‘nvv4l2h264enc’ has no property named ‘bufapi-version’
Creating LL OSD context new
0:00:02.193202854 23824 0x562581464530 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:useEngineFile(): Failed to read from model engine file
0:00:02.193243618 23824 0x562581464530 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:initialize(): Trying to create engine from model files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:06.359376549 23824 0x562581464530 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarMake/resnet18.caffemodel_b16_fp32.engine
0:00:06.387154261 23824 0x562581464530 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:useEngineFile(): Failed to read from model engine file
0:00:06.387179747 23824 0x562581464530 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:initialize(): Trying to create engine from model files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:09.250068137 23824 0x562581464530 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_fp32.engine
0:00:09.268556302 23824 0x562581464530 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:useEngineFile(): Failed to read from model engine file
0:00:09.268580766 23824 0x562581464530 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:initialize(): Trying to create engine from model files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:11.922076729 23824 0x562581464530 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_VehicleTypes/resnet18.caffemodel_b16_fp32.engine
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
0:00:11.937927974 23824 0x562581464530 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:useEngineFile(): Failed to read from model engine file
0:00:11.937947124 23824 0x562581464530 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:14.021751631 23824 0x562581464530 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b4_fp32.engine

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg)
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:189>: Pipeline ready

Creating LL OSD context new
** INFO: <bus_callback:175>: Pipeline running

IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/iou_config.txt
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/iou_config.txt
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/iou_config.txt
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/iou_config.txt
IOU Tracker Init with threshold 0.600000
IOU Tracker Init with threshold 0.600000
IOU Tracker Init with threshold 0.600000
IOU Tracker Init with threshold 0.600000
Bus error (core dumped)

I tried running it again with gdb, here’s the output:

amer@amer-server1:~$ gdb -args deepstream-app -c /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
GNU gdb (Ubuntu 8.1-0ubuntu3.1) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from deepstream-app…(no debugging symbols found)…done.
(gdb) run
Starting program: /usr/bin/deepstream-app -c /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.
[New Thread 0x7fffbf14b700 (LWP 23909)]

(deepstream-app:23902): GLib-GObject-WARNING **: 16:17:28.664: g_object_set_is_valid_property: object class ‘nvv4l2h264enc’ has no property named ‘bufapi-version’
[New Thread 0x7fffb2bed700 (LWP 23910)]
[New Thread 0x7fffb1626700 (LWP 23911)]
[New Thread 0x7fffb0e25700 (LWP 23912)]
Creating LL OSD context new
[New Thread 0x7fff97fff700 (LWP 23913)]
[New Thread 0x7fff977fe700 (LWP 23914)]
[New Thread 0x7fff96ffd700 (LWP 23915)]
0:00:02.699772899 23902 0x55555c3c1b30 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:useEngineFile(): Failed to read from model engine file
0:00:02.699843948 23902 0x55555c3c1b30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:initialize(): Trying to create engine from model files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
[New Thread 0x7fff967fc700 (LWP 23916)]
0:00:06.771726248 23902 0x55555c3c1b30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarMake/resnet18.caffemodel_b16_fp32.engine
[New Thread 0x7fff956b6700 (LWP 23917)]
[New Thread 0x7fff94eb5700 (LWP 23918)]
0:00:06.799298800 23902 0x55555c3c1b30 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:useEngineFile(): Failed to read from model engine file
0:00:06.799319370 23902 0x55555c3c1b30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:initialize(): Trying to create engine from model files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:09.847326266 23902 0x55555c3c1b30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_fp32.engine
[New Thread 0x7fff7bfff700 (LWP 23919)]
[New Thread 0x7fff7b7fe700 (LWP 23920)]
0:00:09.885833106 23902 0x55555c3c1b30 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:useEngineFile(): Failed to read from model engine file
0:00:09.885876924 23902 0x55555c3c1b30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:initialize(): Trying to create engine from model files
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:12.574904717 23902 0x55555c3c1b30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_VehicleTypes/resnet18.caffemodel_b16_fp32.engine
[New Thread 0x7fff7affd700 (LWP 23921)]
[New Thread 0x7fff7a7fc700 (LWP 23922)]
[New Thread 0x7fff79ffb700 (LWP 23923)]
[New Thread 0x7fff797fa700 (LWP 23924)]
[New Thread 0x7fff78ff9700 (LWP 23925)]
[New Thread 0x7fff6bfff700 (LWP 23926)]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
[New Thread 0x7fff6b7fe700 (LWP 23927)]
0:00:12.659347855 23902 0x55555c3c1b30 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:useEngineFile(): Failed to read from model engine file
0:00:12.659368856 23902 0x55555c3c1b30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:14.743008689 23902 0x55555c3c1b30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b4_fp32.engine
[New Thread 0x7fff6affd700 (LWP 23928)]
[New Thread 0x7fff6a7fc700 (LWP 23929)]
[New Thread 0x7fff69ffb700 (LWP 23930)]
[New Thread 0x7fff697fa700 (LWP 23931)]
[New Thread 0x7fff68ff9700 (LWP 23932)]
[New Thread 0x7fff4bfff700 (LWP 23933)]
[New Thread 0x7fff4b7fe700 (LWP 23934)]
[New Thread 0x7fff4affd700 (LWP 23935)]
[New Thread 0x7fff4a7fc700 (LWP 23936)]
[New Thread 0x7fff49ffb700 (LWP 23937)]
[New Thread 0x7fff497fa700 (LWP 23938)]
[New Thread 0x7fff48ff9700 (LWP 23939)]
[New Thread 0x7fff1bfff700 (LWP 23940)]
[New Thread 0x7fff1b7fe700 (LWP 23941)]
[New Thread 0x7fff1affd700 (LWP 23942)]

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg)
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:189>: Pipeline ready

[New Thread 0x7fff1a7fc700 (LWP 23943)]
[New Thread 0x7fff19ffb700 (LWP 23944)]
[New Thread 0x7fff197fa700 (LWP 23945)]
[New Thread 0x7fff18ff9700 (LWP 23946)]
[New Thread 0x7ffef7fff700 (LWP 23947)]
[New Thread 0x7ffeef7fe700 (LWP 23948)]
[New Thread 0x7ffef77fe700 (LWP 23949)]
[New Thread 0x7ffef6ffd700 (LWP 23950)]
[New Thread 0x7ffef67fc700 (LWP 23951)]
[New Thread 0x7ffef5ffb700 (LWP 23952)]
[New Thread 0x7ffef57fa700 (LWP 23953)]
[New Thread 0x7ffef4ff9700 (LWP 23954)]
[New Thread 0x7ffec33cf700 (LWP 23955)]
[New Thread 0x7ffec2bce700 (LWP 23956)]
[New Thread 0x7ffec23cd700 (LWP 23957)]
[New Thread 0x7ffec1bcc700 (LWP 23958)]
[New Thread 0x7ffeb3fff700 (LWP 23959)]
[New Thread 0x7ffeb37fe700 (LWP 23960)]
[New Thread 0x7ffeb2ffd700 (LWP 23961)]
[New Thread 0x7ffeb27fc700 (LWP 23962)]
[New Thread 0x7ffeb1ffb700 (LWP 23963)]
Creating LL OSD context new
[New Thread 0x7ffeb17fa700 (LWP 23964)]
** INFO: <bus_callback:175>: Pipeline running

[New Thread 0x7ffeb0ff9700 (LWP 23965)]
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/iou_config.txt
IOU Tracker Init with threshold 0.600000
[New Thread 0x7ffea7fff700 (LWP 23966)]
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/iou_config.txt
IOU Tracker Init with threshold 0.600000
[New Thread 0x7ffea77fe700 (LWP 23967)]
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/iou_config.txt
IOU Tracker Init with threshold 0.600000
[New Thread 0x7ffea6ffd700 (LWP 23968)]
IOU Tracker: loading config file: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/iou_config.txt
IOU Tracker Init with threshold 0.600000
[New Thread 0x7ffea67fc700 (LWP 23969)]
[New Thread 0x7ffea5ffb700 (LWP 23970)]

Thread 62 “sink_sub_bin_en” received signal SIGBUS, Bus error.
[Switching to Thread 0x7ffea67fc700 (LWP 23969)]
__memmove_sse2_unaligned_erms () at …/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:423
423 …/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.

Not sure what’s going on, any ideas anyone?

ChrisDing · October 9, 2019, 9:40am

Can you “export DISPLAY=:0” and try again ?

amertahir · October 9, 2019, 2:33pm

echo $DISPLAY already gives :0

I tried “export DISPLAY=:0” anyway and ran the app, same bus error.

ChrisDing · October 10, 2019, 5:53am

Can you try the default config? What’s result?

amertahir · October 18, 2019, 10:35pm

I tried with default config, got bus error. I tried debugging with gdb, here’s the output and gdb backtrace:

amer@amer-server1:/opt/nvidia/deepstream/deepstream-4.0/samples/configs$ gdb --args deepstream-app -c ./deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
GNU gdb (Ubuntu 8.1-0ubuntu3.1) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from deepstream-app…(no debugging symbols found)…done.
(gdb) run
Starting program: /usr/bin/deepstream-app -c ./deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.
[New Thread 0x7fffbf14b700 (LWP 4611)]
libEGL warning: DRI2: failed to authenticate
[New Thread 0x7fff9c188700 (LWP 4612)]
[New Thread 0x7fff9b987700 (LWP 4613)]
[New Thread 0x7fff9b186700 (LWP 4614)]
[New Thread 0x7fff9a985700 (LWP 4615)]
[New Thread 0x7fff9a184700 (LWP 4616)]
[New Thread 0x7fff99983700 (LWP 4617)]
[New Thread 0x7fff99182700 (LWP 4618)]
[New Thread 0x7fff98981700 (LWP 4619)]
[New Thread 0x7fff7bfff700 (LWP 4620)]
[New Thread 0x7fff7b7fe700 (LWP 4621)]
[New Thread 0x7fff7affd700 (LWP 4622)]
[New Thread 0x7fff7a7fc700 (LWP 4623)]
[New Thread 0x7fff79ffb700 (LWP 4624)]
[New Thread 0x7fff797fa700 (LWP 4625)]
[New Thread 0x7fff78ff9700 (LWP 4626)]
[New Thread 0x7fff5bfff700 (LWP 4627)]
[New Thread 0x7fff5b7fe700 (LWP 4628)]
[New Thread 0x7fff5a64d700 (LWP 4629)]
[New Thread 0x7fff59e4c700 (LWP 4630)]
[New Thread 0x7fff5964b700 (LWP 4631)]
Creating LL OSD context new
[New Thread 0x7fff58e4a700 (LWP 4632)]
[New Thread 0x7fff43fff700 (LWP 4633)]
[New Thread 0x7fff437fe700 (LWP 4634)]
0:00:03.119772750 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:useEngineFile(): Failed to read from model engine file
0:00:03.119810924 4604 0x55555c3f2030 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:initialize(): Trying to create engine from model files
0:00:03.120400149 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:generateTRTModel(): INT8 not supported by platform. Trying FP16 mode.
0:00:03.120415356 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:generateTRTModel(): FP16 not supported by platform. Using FP32 mode.
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
[New Thread 0x7fff42ffd700 (LWP 4635)]
0:00:07.292287608 4604 0x55555c3f2030 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_2> NvDsInferContext[UID 6]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarMake/resnet18.caffemodel_b16_fp32.engine
[New Thread 0x7fff41eb7700 (LWP 4636)]
[New Thread 0x7fff416b6700 (LWP 4637)]
0:00:07.318316512 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:useEngineFile(): Failed to read from model engine file
0:00:07.318358224 4604 0x55555c3f2030 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:initialize(): Trying to create engine from model files
0:00:07.318829592 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:generateTRTModel(): INT8 not supported by platform. Trying FP16 mode.
0:00:07.318842583 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:generateTRTModel(): FP16 not supported by platform. Using FP32 mode.
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:10.178645941 4604 0x55555c3f2030 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_1> NvDsInferContext[UID 5]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_fp32.engine
[New Thread 0x7fff40eb5700 (LWP 4638)]
[New Thread 0x7fff23fff700 (LWP 4639)]
0:00:10.197496835 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:useEngineFile(): Failed to read from model engine file
0:00:10.197524012 4604 0x55555c3f2030 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:initialize(): Trying to create engine from model files
0:00:10.198025930 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:generateTRTModel(): INT8 not supported by platform. Trying FP16 mode.
0:00:10.198038058 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:generateTRTModel(): FP16 not supported by platform. Using FP32 mode.
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.
0:00:12.982820943 4604 0x55555c3f2030 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 4]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Secondary_VehicleTypes/resnet18.caffemodel_b16_fp32.engine
[New Thread 0x7fff237fe700 (LWP 4640)]
[New Thread 0x7fff22ffd700 (LWP 4641)]
[New Thread 0x7fff227fc700 (LWP 4642)]
[New Thread 0x7fff21ffb700 (LWP 4643)]
[New Thread 0x7fff217fa700 (LWP 4644)]
[New Thread 0x7fff20ff9700 (LWP 4645)]
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
[New Thread 0x7fff03fff700 (LWP 4646)]
0:00:13.106388959 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:useEngineFile(): Failed to read from model engine file
0:00:13.106437723 4604 0x55555c3f2030 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:13.107400633 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): INT8 not supported by platform. Trying FP16 mode.
0:00:13.107464644 4604 0x55555c3f2030 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): FP16 not supported by platform. Using FP32 mode.
0:00:15.094569779 4604 0x55555c3f2030 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b4_fp32.engine
[New Thread 0x7fff037fe700 (LWP 4647)]
[New Thread 0x7fff02ffd700 (LWP 4648)]
[New Thread 0x7fff027fc700 (LWP 4649)]
[New Thread 0x7fff01ffb700 (LWP 4650)]
[New Thread 0x7fff017fa700 (LWP 4651)]
[New Thread 0x7fff00ff9700 (LWP 4652)]
[New Thread 0x7ffedffff700 (LWP 4653)]
[New Thread 0x7ffedf7fe700 (LWP 4654)]
[New Thread 0x7ffedeffd700 (LWP 4655)]
[New Thread 0x7ffede7fc700 (LWP 4656)]
[New Thread 0x7ffeddffb700 (LWP 4657)]
[New Thread 0x7ffedd7fa700 (LWP 4658)]
[New Thread 0x7ffedcff9700 (LWP 4659)]
[New Thread 0x7ffebffff700 (LWP 4660)]
[New Thread 0x7ffebf7fe700 (LWP 4661)]
[New Thread 0x7ffebeffd700 (LWP 4662)]

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg) FPS 1 (Avg) FPS 2 (Avg) FPS 3 (Avg)
**PERF: 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
** INFO: <bus_callback:189>: Pipeline ready

[New Thread 0x7ffebe7fc700 (LWP 4663)]
[New Thread 0x7ffebdffb700 (LWP 4664)]
[New Thread 0x7ffebd7fa700 (LWP 4665)]
[New Thread 0x7ffebcff9700 (LWP 4666)]
[New Thread 0x7ffea7fff700 (LWP 4667)]
[New Thread 0x7ffea77fe700 (LWP 4668)]
[New Thread 0x7ffea6ffd700 (LWP 4669)]
[New Thread 0x7ffea67fc700 (LWP 4670)]
[New Thread 0x7ffea5ffb700 (LWP 4671)]
[New Thread 0x7ffea57fa700 (LWP 4672)]
[New Thread 0x7ffea4ff9700 (LWP 4673)]
[New Thread 0x7ffe93fff700 (LWP 4674)]
[New Thread 0x7ffe69c6b700 (LWP 4675)]
[New Thread 0x7ffe6946a700 (LWP 4676)]
[New Thread 0x7ffe68c69700 (LWP 4677)]
[New Thread 0x7ffe5bfff700 (LWP 4678)]
[New Thread 0x7ffe5abfb700 (LWP 4679)]
[New Thread 0x7ffe5a3fa700 (LWP 4680)]
[New Thread 0x7ffe59bf9700 (LWP 4681)]
** INFO: <bus_callback:175>: Pipeline running

[New Thread 0x7ffe593f8700 (LWP 4682)]
[New Thread 0x7ffe58bf7700 (LWP 4683)]
Creating LL OSD context new
[New Thread 0x7ffe43fff700 (LWP 4684)]
[New Thread 0x7ffe437fe700 (LWP 4685)]
[New Thread 0x7ffe40c15700 (LWP 4686)]
KLT Tracker Init
[New Thread 0x7ffe2ffff700 (LWP 4687)]

Thread 77 “deepstream-app” received signal SIGBUS, Bus error.
[Switching to Thread 0x7ffe40c15700 (LWP 4686)]
__memmove_sse2_unaligned_erms () at …/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:423
423 …/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0 0x00007ffff5e046c4 in __memmove_sse2_unaligned_erms () at …/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:423
#1 0x00007fff400772be in KLTTrackerBlock::updateObjects(unsigned char*, unsigned int, unsigned int, unsigned int) ()
at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
#2 0x00007fff4007c709 in NvMOT_Process () at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
#3 0x00007fffb419e70d in NvTrackerProc::processSurfaceStream(NvTrackerProc::DispatchInfo*) ()
at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#4 0x00007fffb41a6ce1 in void std::__invoke_impl<void, void (NvTrackerProc::)(NvTrackerProc::DispatchInfo), NvTrackerProc*, NvTrackerProc::DispatchInfo*>(std::__invoke_memfun_deref, void (NvTrackerProc::&&)(NvTrackerProc::DispatchInfo), NvTrackerProc*&&, NvTrackerProc::DispatchInfo*&&) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#5 0x00007fffb41a272d in std::__invoke_result<void (NvTrackerProc::)(NvTrackerProc::DispatchInfo), NvTrackerProc*, NvTrackerProc::DispatchInfo*>::type std::__invoke<void (NvTrackerProc::)(NvTrackerProc::DispatchInfo), NvTrackerProc*, NvTrackerProc::DispatchInfo*>(void (NvTrackerProc::&&)(NvTrackerProc::DispatchInfo), NvTrackerProc*&&, NvTrackerProc::DispatchInfo*&&) ()
at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#6 0x00007fffb41b82d9 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<void (NvTrackerProc::)(NvTrackerProc::DispatchInfo), NvTrackerProc*, NvTrackerProc::DispatchInfo*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#7 0x00007fffb41b8216 in std::thread::_Invoker<std::tuple<void (NvTrackerProc::)(NvTrackerProc::DispatchInfo), NvTrackerProc*, NvTrackerProc::DispatchInfo*> >::operator()() () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#8 0x00007fffb41b81c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (NvTrackerProc::)(NvTrackerProc::DispatchInfo), NvTrackerProc*, NvTrackerProc::DispatchInfo*> > >::_M_run() () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#9 0x00007fffcd15066f in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fffeb79d6db in start_thread (arg=0x7ffe40c15700) at pthread_create.c:463
#11 0x00007ffff5e6a88f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

Amycao · October 24, 2019, 6:51am

If TRT sample like trtexec works on K40 then DS should work - we have not tested on Kepler series like K40. you should move to a Tesla based architecture onwards GPU - latest GPU like P4 or P40 or T4.