Kernel panic occurs when deepstream service is configured and executed in systemd

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only) 5.1
• TensorRT Version 8.5.2-1+cuda11.4
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

@prominence_ai
The app I use is a dsl-based Deepstream-app made by him
Let’s take an example situation

When running the app directly after booting up —> Normal operation (Good)
When I try to test the app by rebooting after registering in systemd, it turns off after 3-4 seconds after booting. —> Bad operation (like written /etc/systemd/system/myapp.service)

However, by adding the code below to the first part of my main app to cause delay in the pipeline, the phenomenon disappears.

std::this_thread::sleep_for(std::chrono::milliseconds(5000));

...
Pipeline play()

Also to note, if I intentionally disconnect the camera for example to prevent the inference from starting, booting is fine. However, if I set it to be inference from bootup, it will be turned off.

Maybe it’s because loading the library took a while…
Do you know anything about it??

The attached log below is the log dumped when the system shuts down after booting, but there seems to be no noteworthy log.

fcam_gst_debug.log (90.3 KB)

Thank you
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

3

kernel panic ？

Do you mean that when deepstream starts using systemd, the system crashes?

Usually the application does not cause the system to crash

It may be caused by some drivers.

I guess it may be related to the camera driver.

Can you share your systemd configure ?
Or
Try start deepstream after system module loaded.

[Unit]
Requires=systemd-modules-load.service
After=systemd-modules-load.service
4

The same error occurs when the camera is set or video is set as input as well.
It is a dependent part of my service, but there is no doubt about it??

neuboat.service
● ├─-.mount
● ├─system.slice
● ├─network-online.target
● │ └─NetworkManager-wait-online.service
● └─sysinit.target
●   ├─apparmor.service
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ├─keyboard-setup.service
●   ├─kmod-static-nodes.service
●   ├─nv-tee-supplicant.service
●   ├─nvfb-udev.service
●   ├─plymouth-read-write.service
●   ├─plymouth-start.service
●   ├─proc-sys-fs-binfmt_misc.automount
●   ├─resolvconf.service
●   ├─setvtrgb.service
●   ├─sys-fs-fuse-connections.mount
●   ├─sys-kernel-config.mount
●   ├─sys-kernel-debug.mount
●   ├─sys-kernel-tracing.mount
●   ├─systemd-ask-password-console.path
●   ├─systemd-binfmt.service
●   ├─systemd-boot-system-token.service
●   ├─systemd-hwdb-update.service
●   ├─systemd-journal-flush.service
●   ├─systemd-journald.service
●   ├─systemd-machine-id-commit.service
●   ├─systemd-modules-load.service
●   ├─systemd-pstore.service
●   ├─systemd-random-seed.service
●   ├─systemd-sysctl.service
●   ├─systemd-sysusers.service
●   ├─systemd-timesyncd.service
●   ├─systemd-tmpfiles-setup-dev.service
●   ├─systemd-tmpfiles-setup.service
●   ├─systemd-udev-trigger.service
●   ├─systemd-udevd.service
●   ├─systemd-update-utmp.service
●   ├─cryptsetup.target
●   ├─local-fs.target
●   │ ├─-.mount
●   │ ├─systemd-fsck-root.service
●   │ └─systemd-remount-fs.service
●   └─swap.target
5

try the below command to got failure log.

journalctl -u neuboat.service > log.txt

I need to get the failure logs to know what happened. From your description, I can only guess that some dependencies are not started before your service

6

The components are built fine, there are no obvious error codes, but the kernel crashes.

root@K236ORUA000005:~# journalctl -u neuboat.service -f
-- Logs begin at Thu 2022-09-08 09:58:15 UTC. --
Jul 25 08:32:05 K236ORUA000005 tcd[2500]:  tile = ../../data/tm/aiboat.db
Jul 25 08:32:05 K236ORUA000005 tcd[2500]: =========================
Jul 25 08:32:05 K236ORUA000005 tcd[2500]: open_tile_db(../../data/tm/aiboat.db) failed. ret=-2
Jul 25 08:32:05 K236ORUA000005 tcd[2562]: === CM-RCS Start... ===
Jul 25 08:32:05 K236ORUA000005 tcd[2562]: === Running Options ===
Jul 25 08:32:05 K236ORUA000005 tcd[2562]:  ini  = ../cfg/cm.ini
Jul 25 08:32:05 K236ORUA000005 tcd[2562]:  acnt = ../cfg/acnt.info
Jul 25 08:32:05 K236ORUA000005 tcd[2562]:  tile = ../data/tilemap.db
Jul 25 08:32:05 K236ORUA000005 tcd[2562]: =========================
Jul 25 08:32:05 K236ORUA000005 tcd[2562]: open_tile_db(../data/tilemap.db) failed. ret=-2
Jul 25 08:32:07 K236ORUA000005 tcd[2867]: modprobe: FATAL: Module nvidia not found in directory /lib/modules/5.10.104-tegra
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: === INFERENCE Start... ===
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: === Running Options ===
Jul 25 08:32:12 K236ORUA000005 tcd[2650]:  ini  = ../cfg/fcam_od.ini
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: =========================
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: [ami_init] g_ami_data.ipstr=192.168.0.200, pCFG->nbnet_ipstr=192.168.0.200
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: <REG-DONE> appid=00001021, pss_id=21
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: Success load INI file
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: rtsp-source component add
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: primary-gie component add
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: tracker component add
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: fake-sink component add
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: Opening in BLOCKING MODE
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: gstnvtracker: Batch processing is ON
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: gstnvtracker: Past frame output is OFF
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: INFO: [Implicit Engine Info]: layers num: 4
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: 0   INPUT  kFLOAT input           3x544x960
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: 1   OUTPUT kFLOAT boxes           32130x4
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: 2   OUTPUT kFLOAT scores          32130x1
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: 3   OUTPUT kFLOAT classes         32130x1
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: previous state = DSL_STATE_NULL, new state = DSL_STATE_READY
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: previous state = DSL_STATE_READY, new state = DSL_STATE_PAUSED
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: previous state = DSL_STATE_PAUSED, new state = DSL_STATE_PLAYING
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: NvMMLiteOpen : Block : BlockType = 277
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: NVMEDIA: Reading vendor.tegra.display-size : status: 6
Jul 25 08:32:12 K236ORUA000005 tcd[2650]: NvMMLiteBlockCreate : Block : BlockType = 277
7

There is a similar question here

It provides a workaround.

What’s your pipeline ?

Use gst-launch to replace deepstream-app to help find where the problem is.

8

After checking, in my case… I suspect it’s a hardware issue.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.