Do You See Truncated Logs after rsyslog Update?

I still do apt-get upgrades regularly on Jetson, I don’t worry about the libglx issue (I keep both this and the entire tar file of nVidia files ready), and not long ago (my upgrade was on Oct. 11, 2014) there was an upgrade to package “rsyslog:armhf” (see /var/log/apt/history.log).

The following log files no longer log:

-rw-r--r--  1 root              root         0 Feb  1  2000 aptitude
-rw-r--r--  1 root              root         0 May 13 01:27 bootstrap.log
-rw-rw----  1 root              utmp         0 Oct  1 16:52 btmp
-rw-r--r--  1 root              root         0 Oct  3 18:17 kern.log
-rw-r--r--  1 root              root         0 Oct  3 18:17 syslog

It seems like files btmp, kern.log, and syslog just now stopped logging; others are far enough in the past that they are probably unrelated. When looking back at the original configuration from sample rootfs, as well as a system.img from a while back, I see the rsyslog configuration files in /etc have changed very little (or not changed at all). This was added to rsyslog.conf, but changing or removing this has no effect:

# Enable non-kernel facility klog messages
$KLogPermitNonKernelFacility on

This is bad for kernel development, no more printk statements show up in logs. I can still use the “dmesg” command and see current printk’s, apparently because dmesg reads the kernel’s in-memory buffer…no such information makes its way to the dmesg log file (or any other file). It is no long possible to “tail -f” the kern.log file while developing, I have to manually dmesg every time now.

Is this just me, or has this happened to others with the rsyslog updated? You can check if rsyslog was updated if you get an answer from this:

sudo egrep rsyslog /var/log/apt/history.log

You can check for zero length log files via:

ls -l /var/log | egrep ' 0 '

I’m interested in knowing if this is a new rsyslog bug to turn in to the maintainers, or if this is only happening to me. How many people have a zero length “/var/log/kern.log”? If you have zero length or not zero length, have you upgraded rsyslog?

My logs are not zero sized but I haven’t made any upgrades. Doing it now and I’ll try to remember to update later.

After a couple of reboots I still see kernel messages in kern.log.

I am thinking that there would be more responses to this question if there were not a fear of apt-get upgrade causing a failure. So check the EDIT on remotely restoring libglx.so here:

https://devtalk.nvidia.com/default/topic/775070/embedded-systems/notice-on-apt-get-upgrade-libglx-so-corruption/post/4309834/#4309834

Having multiple checks on whether kern.log is truncated would aid in debugging and reporting for rsyslog.

Shameless bump…I don’t want to report a bug to rsyslog if this is only happening to me or if it is an Ubuntu packaging issue.

My rsyslog version is “7.4.4-1ubuntu2.3”, found via:

apt show rsyslog | egrep Version

Do you have this or a newer version? If so, is /var/log/kern.log size 0?

I can’t actually mark this “solved”, but for those interested and who have lost kernel logging, here is a workaround.

It seems that starting with Ubuntu 14.04 (which L4T is a near duplicate of) there was a switch from syslogd to rsyslogd. Some of the software still tries to log to syslogd, and so rsyslogd was given options to log these older syslogd messages. This file:

/etc/rsyslog.d/50-default.conf

…contains commented out lines meant to catch some of the older software messages still attempting to go to syslog. Uncomment these and get logging going for those doing kernel development:

*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages

I can’t verify this, but it seems likely now that the rsyslog update was not the cause of files not logging, and was only coincidental…I can’t find differences between most recent rsyslog config and config from original R19.3. It sure looks like some other software which had been logging to rsyslog switched back to syslog…so catching syslog messages does the job.

1 Like

Hello,

Thanks for sharing a good article.
The jetson series, especially the production module using emmc, has a small basic storage capacity so that even the size of the log needs to be managed in detail.
I know these log management issues are not Jetson’s issue. (although it is a problem related to linux os management to be exact)
We hope that various experiences and know-hows on log management will be shared. Maybe the Raspberry Pi, a platform similar to jetson, has been in the community for a long time, so maybe you can get some good ideas through this forum, right?

Thank you.