Sysrq occurs constantly

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi,

I’m using TX2NX with JP 4.6.2.
On some of my TX2NX (2 of 20), with the same firmware version and hardware, the sysrq message occurs many time (every second)

~# uptime
 19:17:07 up  3:17,  1 user,  load average: 0.06, 0.26, 0.20
~# dmesg | grep sysrq | wc
    773   16990  258705
~# dmesg | head -n 50
[10583.123859] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)
[10583.837185] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)
[10584.129571] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)
[10584.363556] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)
[10584.772807] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)
[10586.033915] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)
[10587.345164] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)

Firstly, I thought is is because of noise on debug serial. Then I disabled the debug serial by systemctl stop serial-getty@ttyS0.service, the debug serial doesn’t work anymore, but the sysrq message is still occurs with the same frequency.

I can disable the sysrq by set kernel.sysrq = 0 in /etc/sysctl.conf. After disable, the sysrq message stop occuring.

But I wan’t to know why this happens (because it is not on all of my Jetson TX2 NX, just two). Do you have any idea about the root cause of this?

Thank you and Happy New Year.
Linh

3

Hi,
Do you observe the issue on TX2 NX developer kit? Or TX2 NX module + your carrier board?

1 Like
4

Hi, it is on our carrier board + TX2 NX module.

5

Hi,
We don’t have further idea about this. Seems to be an issue in the carrier board if this issue is not seen on TX2 NX + Xavier NX carrier board. If it is fine to disable sysrq in config file, you may simply disable it.

6

I am curious if you are using any of the serial UARTs? I ask because if the UART is also a serial console (and this happens a lot), then data over the UART might be triggering the sysrq.

A temporary workaround, as mentioned by @DaneLLL, is to disable sysrq via:
sudo echo 0 > /proc/sys/kernel/sysrq
(if this is a collision of data and serial console though, the real solution is to disable serial console)

If that is acceptable, then testing can in turn be made permanent by adding this to “/etc/sysctl.conf” (I’d add this near the end):
kernel.sysrq = 0

In any case where reliability is needed or other serious use is required I’d want to find out why this happens. It isn’t because the message is a problem (other than perhaps filling up the file system with log messages), but instead wanting to know if there is spurious input to a terminal (which in turn could imply spurious terminal output somewhere you don’t expect).

7

Hi linuxdev,

My carrier board has 3 serial UART ports. Debug port is TTL, two others are RS232/RS445 with tranceiver IC. At the moment, one of RS232 port is used to communicate with other MCU board.

I found one of the reason now, it is electric noise in debug UART TTL port (my carrier board doens’t have pull-up register on RX pin). After updating pinmux and enable internal pull-up for both RX, TX of debug UART. It solved the problem for one of the two Jetson TX2 NX (only these two having this problem). I’m waiting for testing on another board.
But on the not testing board, I also see the worse performance issue. The same firmware, but on that TX2NX, I need to reduce to lower setting of yolo4 model to run, while other TX2NXs work fine with higher yolov4 setting. Need to verify when it is online.

8

Are those UART ports external/add-on? Not the integrated ones? If so, then they shouldn’t have to worry about clock precision at higher speeds, but as you have noted, noise becomes a serious issue. Use shielding around the cables if you can, and the shortest possible length. Actual DB-9 RS-232 uses a common ground and not differential signalling, and so noise immunity is limited, but you could run twisted pair with RX/ground, as well as with TX/ground if you have the ground wires to do so (you don’t want to twist TX and RX). You’ll probably find this an odd statement, but if you want the best immunity, then you might consider adapting a piece of CAT-7 ethernet cable (often used with 10 Gb/s ethernet, it has extra shielding).

Note: I am assuming RS-232 mode (RS-445 I think can run balanced pair, but I don’t think the Jetson would support that without effort).

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.