Serial communication issue - "Got overrun errors"

I have a custom carrier board for the TX1 SOM that is connected to two different serial devices. One runs at 3 MB/s, and the other runs at 57600 baud. I’m communicating over /dev/ttyTHS1 and /dev/ttyTHS2 using L4T 24.1 (kernel 3.10.96). Immediately I’m seeing some overrun errors like this:

[  154.451593] serial-tegra 70006040.serial: Got overrun errors
[  154.457323] serial-tegra 70006040.serial: Got overrun errors
[  156.260219] serial-tegra 70006040.serial: Got overrun errors
[  156.265947] serial-tegra 70006040.serial: Got overrun errors
[  156.362059] serial-tegra 70006040.serial: Got overrun errors
[  156.367783] serial-tegra 70006040.serial: Got overrun errors
[  156.662330] serial-tegra 70006040.serial: Got overrun errors
[  156.668032] serial-tegra 70006040.serial: Got overrun errors

One instance ended like this:

[  364.883516] serial-tegra 70006040.serial: Got overrun errors
[  365.129182] serial-tegra 70006040.serial: Got overrun errors
[  365.134877] serial-tegra 70006040.serial: Got overrun errors
[  365.180827] Unhandled fault: alignment fault (0x92000061) at 0x000000000001fc45

and the program crashed.

Another instance ended like this:

[  420.760647] tty_insert_flip_string_fixed_flag_lock: No space: Requested 1, memory used: 65536
[  420.769236] serial-tegra 70006040.serial: RxData PIO to tty layer failed
[  420.776003] tty_insert_flip_string_fixed_flag_lock: No space: Requested 1, memory used: 65536
[  420.784592] serial-tegra 70006040.serial: RxData PIO to tty layer failed
[  420.791357] tty_insert_flip_string_fixed_flag_lock: No space: Requested 1, memory used: 65536
[  420.799944] serial-tegra 70006040.serial: RxData PIO to tty layer failed
[  420.806716] serial-tegra 70006040.serial: Got overrun errors
[  420.812437] tty_insert_flip_string_fixed_flag_lock: No space: Requested 1, memory used: 65536
[  420.821024] serial-tegra 70006040.serial: RxData PIO to tty layer failed
[  420.827792] tty_insert_flip_string_fixed_flag_lock: No space: Requested 1, memory used: 65536
[  420.836383] serial-tegra 70006040.serial: RxData PIO to tty layer failed
[  420.843157] serial-tegra 70006040.serial: Got overrun errors

and the entire system was hung.

I also saw similar behavior while running the older 3.10.67 kernel.

I used a scope to check the signal integrity of the 3 MB/s signal, and it looks VERY clean.
The slower device at 57600 also causes the system to crash.

At this point, it really looks like a kernel driver bug, perhaps a memory alignment problem. Has anyone else used the UARTs on the TX1, and seen any problems like this?

I’m going to write a simple loopback program that will pass data from one UART to another, and see if the problem still exists. If this exhibits the same problem, I’ll share the code and hopefully nVidia can help solve the problem.

-Dennis

Are you using CTS/DTSRTS, or software flow control? I am suspicious of flow control being set differently between one end of the serial port and the other.

Hmmm. Our devices do not support flow control, so we’re not using it. Neither hardware or software flow control.

I’m not where I can check right now on how much buffer is available (I doubt it is much), but is there anything in your software to tell an application sending parallel data out for serialization when to stop because of a full buffer? And for software receiving serial data that the UART has converted to parallel, does this software always consume data as fast as it is received? Even if consumers and producers of serial data are set to run at the same speed, there would likely be moments of dropped or overrun data at times, e.g., if the serial UART hardware IRQ can’t run the driver until a short delay from some other driver freeing time, and then there is no guarantee which serial UART driver would be serviced first…the one required to consume or produce to avoid lost data, or the one which needs to wait for the other one to avoid lost data.

Hi Linuxdev,

My name is Mark, I am working with Dennis, (dmillard), on the issue we are encountering.

We are not employing a software method to identify a full buffer, although simply the initial connection handshake before data is being queried/broadcast triggers the overrun error. The software we are running is an unmodified open source package, https://bitbucket.org/librepilot/librepilot.

When we open a serial connection at 57600b/s to the LibrePilot device we observe that simply connecting can result in crashing the TX1 OS after a few minutes of remaining connected. It doesn’t quite seem to be behaving like a full buffer, but rather like a memory misalignment/management issue.

We don’t see this issue on x86 machines running the identical software (compiled natively for x86 arch), so although we can’t spot where this is occurring yet, we are looking for ways it might be a result of the architecture difference as a memory alignment/handling oversight in the driver. Do you have any previous experiences that might support that hypothesis?

(We are still working on finishing the loopback test, the team will get that posted here as soon as it is completed.)

Thank you for your help with this!

-Mark

If you have JTAG, then something like the Lauterbach debugger would be the most direct way to quickly find out what’s going on…you could get a stack frame. You could also compile the kernel for kgdboc, which is more complicated, but you could possibly get a stack from on that error without special hardware.

Since you are using ttyTHS1 and 2, is there any error generated simply by query of device status (requires adding package “setserial”)?

sudo setserial -a /dev/ttyTHS1
sudo setserial -a /dev/ttyTHS2

Also, what is the reported kernel command line from “/proc/cmdline”?

EDIT: Just thought of one additional tool. If not familiar with magic sysrq key sequences, see:
https://en.wikipedia.org/wiki/Magic_SysRq_key
…It is possible that upon error you might be able to get info from ctrl-alt-sysrq-c, followed by dmesg or simply watching serial console output.

Hi Linuxdev,

Mark and I ran tests on 64 bit and 32 bit L4T 24.1 installations from Jetpack 2.2 standard installations.

The results from setserial -a /dev/ttyTHS1 and 2 are as follows:

/dev/ttyTHS1, Line 1, UART: undefined, Port: 0x0000, IRQ: 69
              Baud_base: 0, close_delay: 50, divisor:0
              closing_wait: 3000
              Flags: spd_normal
/dev/ttyTHS2, Line 1, UART: undefined, Port: 0x0000, IRQ: 78
              Baud_base: 0, close_delay: 50, divisor:0
              closing_wait: 3000
              Flags: spd_normal

And the results from /proc/cmdline on the 32 bit TX1 are:

fbcon=map:0 console=tty0 console=ttyS0,115200n8 androidboot.modem=none androidboot.serialno=P2180A00P00940c003fd androidboot.security=non-secure tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff2bf000 nvdumper_reserved=0xff23f000 core_edp_mv=1125 core_edp_ma=4000 gpt android.kerneltype=normal androidboot.touch_vendor_id=0 androidboot.touch_panel_id=63 androidboot.touch_feature=0 androidboot.bootreason=pmc:software_reset,pmic:0x0 root=/dev/mmcblk0p1 rw rootwait

We have created a loopback test that communicates between ttyTHS1 and ttyTHS2 which fails even at slow speeds. The failure mode is a complete hang of the OS, followed by a spontaneous reboot.

When we use a USB to serial converter and use the loopback test to run ttyUSB0 to ttyUSB1 it doesn’t crash the OS but is guaranteed to see significant amounts of dropped data at any baud rate higher than 9600b/s.

The same test code runs flawlessly on an x86 machine without any failures even at commanded speeds up to 3Mb/s (although actual throughput is around 1.2Mb/s). We can share the loopback test source if you can provide instructions of where to upload it or who to send it to.

The magic sysreq ctrl-alt-sysrq-c and dmesg fails to report anything back to the console.

Let us know if you would like a copy of the loopback test. Thanks so much for all of your help!

-Dennis

I am not specifically using any ttyTHS* (makes testing an issue), but I see something which does not seem right. Unfortunately I do not know an exact meaning of the “line” in setserial output, but it seems this must be unique (possibly IRQ hardware line?), even if IRQ is different. Both say “Line 1”. Does your working desktop have unique “Line” configuration for each “ttyTHS*”?

Hi Linuxdev,

I apologize, I made a mistake dictating to Dennis as we typed out our last response, ttyTHS1 and ttyTHS2 show Line 1 and Line 2 respectively. After verifying this again, to properly reflect what should have been above:

/dev/ttyTHS1, Line 1, UART: undefined, Port: 0x0000, IRQ: 69
                  Baud_base: 0, close_delay: 50, divisor:0
                  closing_wait: 3000
                  Flags: spd_normal
/dev/ttyTHS2, Line 2, UART: undefined, Port: 0x0000, IRQ: 78
                  Baud_base: 0, close_delay: 50, divisor:0
                  closing_wait: 3000
                  Flags: spd_normal

We just completed a fresh installation of L4T 24.1 64-bit using JetPack 2.2, updated and upgraded all packages, and retested the loopback test between UART0/UART1 (/dev/ttyTHS1 and /dev/ttyTHS2) on the TX1 development board by connecting them with jumpers on the development board as follows:

Header J17: Pin 4 -> Header J21: Pin 8
Header J17: Pin 5 -> Header J21: Pin 10

The result of running the loopback test at varying speeds will be either a hang or a complete crash/reboot of the OS. At speeds lower than 460800b/s, the loopback test will often run successfully for a while, sometimes a long while, particularly at slow speeds. At 460800b/s, however, the loopback will immediately crash the OS and trip a reboot simply by starting the test. Occasionally starting the receive part of the test will crash the OS, but more often the transmit will trigger the crash.

The test seems to point toward a low level driver issue, because it’s not an issue of just corrupting data, the simple transfer of data crashes the operating system with complete repeatability.

We really need this basic UART functionality to work, can you help us dig into this further? What else do you need from us? We can provide you a copy of the loopback test if you provide instructions on how to do so.

Thanks for all of your help so far!

-Mark

J17 is tied to the camera module. I wonder if the camera unit is interfering? If you have the camera installed, try removing that. There may also be some sort of driver related to that camera in need of disable.

If that doesn’t help, go ahead and give a brief method of how to reproduce the effect (e.g., jumper two pins, setserial, and echo).

Hi Again,

We’ve also tested this using two FTDI USB to serial converters, by simply connecting RX->TX and TX->RX and running the loopback between ttyUSB0 and ttyUSB1, and we can reliably crash the OS, (a system hang followed by a reboot), using this setup as well.

If you need additional information from us please let us know. Thanks again for your help!

-Mark

(P.S. We’ve also crashed the OS by copy/paste of large files from USB drives to the eMMC and vice/versa.)

Here is the source code for the loopback test so that you can reproduce the issue, simply compile it with gcc. (We had to use ttyTHS2 and ttyS0 on the development board with headers J17 and J21, but with USB to serial converters you can simply use ttyUSB0 and ttyUSB1 or whatever your devices show up as.)

/*******************************************************/
/* Program to test serial port throughput              */
/*                                                     */
/*******************************************************/

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <inttypes.h>
#include <time.h>
#include <fcntl.h> // Required for open()
#include <unistd.h> // Required for read(), write(), close()
#include <termios.h> // Required for tcgetattr(), cfsetispeed(), cfsetospeed(), tcflush()
#include <errno.h> // Require for errno
#include <unistd.h> // Required for error codes, among other things
#include <poll.h> // Required for poll()
#include <sys/types.h>
#include <sys/stat.h>

#define VERSION_MAJOR 1
#define VERSION_MINOR 0

#define DEVLENGTH 128
#define BAUDLENGTH 32

#define RX 1
#define TX 2

static const char *optString = "vd:b:rt";

typedef struct {
	int verbose;			/* enable verbose output */
	char devpath[DEVLENGTH];	/* -d serial device */
	int baudrate;			/* baudrate to use */
	int sermode;			/* serial mode RX or TX */
	int fd;				/* file descriptor */
} Settings_t;

/*************************************************************/
/* parse_args() - Parse command line arguments and set       */
/* values as necessary.                                      */
/* parameters: argc = argument count, argv = argument values */
/*************************************************************/
void parse_args(int argc, char *argv[], Settings_t *settings)
{
	int opt = 0;

	/* set default device path */
	strncpy(settings->devpath, "/dev/ttyUSB0", DEVLENGTH);
	/* set default baud rate */
	settings->baudrate = 9600;

	opt = getopt( argc, argv, optString );
	while( opt != -1 ) {
		switch( opt ) {
			case 'v':
				// turn on verbose mode
				settings->verbose = 1;
				break;
			case 'd':
				// sets the device path
				strncpy(settings->devpath, optarg, DEVLENGTH);
				break;
			case 'b':
				// sets the baud rate
				settings->baudrate = atoi(optarg);
				break;
			case 'r':
				// sets to serial receive mode
				settings->sermode = RX;
				break;
			case 't':
				// sets to serial transmit mode
				settings->sermode = TX;
				break;
			case 'h':
			case '?':
				printf("sertest version %d.%d\n", VERSION_MAJOR, VERSION_MINOR);
				printf("usage: ./%s [-v] [-d device] [-b baud] -t|-r\n", argv[0]);
				printf("  -v enables verbose mode\n");
				printf("  -d <devicename> sets the serial device\n");
				printf("  -b <baud> sets the baud rate\n");
				printf("  -t sets mode to transmit\n");
				printf("  -r sets mode to receive\n");
				exit( EXIT_FAILURE );
				break;
			default:
				/* You won't actually get here. */
				break;
		}
		opt = getopt( argc, argv, optString );
	}

	if (settings->verbose) {
		printf("Arguments:\n");
		printf("  -v: %d\n", settings->verbose);
		printf("  -d: %s\n", settings->devpath);
		printf("  -b: %d\n", settings->baudrate);
		printf("  -r/t: %d\n", settings->sermode);
	}
}

/*************************************************************/
/* openport() - opens the selected serial port and gets a    */
/* file descriptor                                           */
/*************************************************************/
int openport(Settings_t *settings)
{
    settings->fd = open(settings->devpath, O_RDWR | O_NOCTTY | O_NDELAY);
    if (settings->fd <= 0) {
        char errorString[0xFF];
        strerror_r(errno, errorString, sizeof(errorString));
        fprintf(stderr, "Open device failed: Unable to open device file %s. Error: %s\n", settings->devpath, errorString);
        return -1;
    } else {
	if (settings->verbose) {
		fprintf(stderr, "Got file descriptor: %d\n", settings->fd);
	}
    }

    // Reset the serial device file descriptor for non-blocking read / write
    fcntl(settings->fd, F_SETFL, 0);

    return 0;
}

/*************************************************************/
/* configureport() - configures the selected serial port     */
/*************************************************************/
int configureport(Settings_t *settings)
{
    // Modify the settings on the serial device (baud rate, 8n1, receiver enabled, ignore modem status, no flow control) and apply them
    struct termios deviceOptions;
    memset (&deviceOptions, 0, sizeof deviceOptions);
    tcgetattr(settings->fd, &deviceOptions);

    /* Set the baud rates... */
    speed_t MyBaud;
    switch (settings->baudrate) {
	case 9600:
		MyBaud = B9600;
		break;
	case 57600:
		MyBaud = B57600;
		break;
	case 115200:
		MyBaud = B115200;
		break;
	case 230400:
		MyBaud = B230400;
		break;
	case 460800:
		MyBaud = B460800;
		break;
	case 921600:
		MyBaud = B921600;
		break;
	case 1000000:
		MyBaud = B1000000;
		break;
	case 2000000:
		MyBaud = B2000000;
		break;
	case 3000000:
		MyBaud = B3000000;
		break;
	default:
		MyBaud = B9600;
    }

    //cfsetispeed(&deviceOptions, settings->baudrate);
    //cfsetospeed(&deviceOptions, settings->baudrate);
    cfsetispeed(&deviceOptions, MyBaud);
    cfsetospeed(&deviceOptions, MyBaud);

    // Set character length
    deviceOptions.c_cflag &= ~CSIZE;       // Mask the character size bits
    deviceOptions.c_cflag |= CS8;

    // No parity
    deviceOptions.c_cflag &= ~PARENB;
    deviceOptions.c_cflag |= PARODD;

    // Stick parity flag
    deviceOptions.c_cflag |= CMSPAR;

    // ignores parity errors and passes bytes regardless
    deviceOptions.c_iflag |= (IGNPAR);

    //causes parity errors to be 'marked' in the input stream using special characters.
    //If IGNPAR is enabled, a NUL character (0x00) is sent to your program before every character with a parity error.
    //Otherwise, a DEL (0x7F) and NUL character is sent along with the bad character.
    deviceOptions.c_iflag |= (PARMRK);

    //enable checking and stripping of the parity bit
    //NOTE: PARMRK and INPCK are exclusive of each other
    //options.c_iflag |= (INPCK);
    //options.c_iflag |= (ISTRIP);

    // One stop bit
    deviceOptions.c_cflag &= ~CSTOPB;

    /* No hardware flow control */
    deviceOptions.c_cflag &= ~CRTSCTS;

    // Raw input, no echo
    deviceOptions.c_lflag &= ~(ECHO | ECHOE | ECHONL | ICANON | IEXTEN | ISIG);
    deviceOptions.c_iflag &=  ~(IGNBRK | BRKINT | PARMRK | ISTRIP
                | INLCR | IGNCR | ICRNL | IXON | IXOFF | IXANY | INPCK );

    // Raw output
    deviceOptions.c_oflag &= ~(OCRNL | ONLCR | ONLRET |
                ONOCR | OFILL | OLCUC | OPOST);

    // Wait timeout for each character
    deviceOptions.c_cc[VMIN] = 1;
    deviceOptions.c_cc[VTIME] = 0;

    /* Enable the receiver and set local mode... */
    deviceOptions.c_cflag |= (CLOCAL | CREAD);

    tcflush(settings->fd, TCIFLUSH);
    if (tcsetattr(settings->fd, TCSANOW, &deviceOptions) != 0) {
        close(settings->fd);
        char errorString[0xFF];
        strerror_r(errno, errorString, sizeof(errorString));
        fprintf(stderr, "Create failed: Unable to set options on device. Error: %s\n", errorString);
        return -1;
    }
    if (settings->verbose) {
    	printf("Configured serial device file descriptor for 8n1 and no flow control: %s\n", settings->devpath);
    }
    return 0;
}

/*************************************************************/
/* readBytes() - reads bytes from the selected serial port   */
/*************************************************************/
ssize_t readBytes(int deviceFileDescriptor, void* destination, ssize_t size)
{
    ssize_t bytesRead = 0;
    while (bytesRead < (ssize_t)size) {
        ssize_t result = read(deviceFileDescriptor, destination + bytesRead, size - bytesRead);
        if (result <= 0 && errno != EWOULDBLOCK) {
            char errorString[0xFF];
            strerror_r(errno, errorString, sizeof(errorString));
            fprintf(stderr, "Failed to read from device descriptor. Error: %s\n", errorString);
            return result;
        }
        if (result <= 0 && errno != EAGAIN) {
            char errorString[0xFF];
            strerror_r(errno, errorString, sizeof(errorString));
            fprintf(stderr, "Error: %s\n", errorString);
            return result;
        }
        bytesRead += result;
    }
    return bytesRead;
}

/*************************************************************/
/* writeBytes() - writes bytes to the selected serial port   */
/*************************************************************/
ssize_t writeBytes(int deviceFileDescriptor, const void* source, ssize_t size) {
    ssize_t bytesWritten = 0;
    while (bytesWritten < size) {
        ssize_t result = write(deviceFileDescriptor, source + bytesWritten, size - bytesWritten);
        if (result <= 0 && errno != EWOULDBLOCK) {
            char errorString[0xFF];
            strerror_r(errno, errorString, sizeof(errorString));
            fprintf(stderr, "Failed to write to serial device descriptor. Error: %s\n", errorString);
            return result;
        }
        bytesWritten += result;
    }

    return bytesWritten;
}

/*************************************************************/
/* Main function                                             */
/*************************************************************/
int main(int argc, char *argv[])
{
	Settings_t MySettings;
	long rxcount = 0;
	long txcount = 0;

	/* parse the command line arguments */
	parse_args(argc, argv, &MySettings);

	/* open the port */
	if (openport(&MySettings) != 0) {
		return -1;
	}

	/* ---- RX mode ---- */
	if (MySettings.sermode == RX) {

		// Reset the serial device file descriptor for blocking read / write
		fcntl(MySettings.fd, F_SETFL, 0);

		if (configureport(&MySettings) != 0) {
			return -1;
		}

		// Begin reading
		char testchar;
		char expected_char = 'A';

		fprintf(stderr, "Waiting for data...\n");

		while (MySettings.fd > 0) {
			// Wait until there is data to read or a timeout happens
			struct pollfd pollOptions;
			pollOptions.fd = MySettings.fd;
			pollOptions.events = POLLIN;
			pollOptions.revents = 0;
			poll(&pollOptions, 1, 200);
			ssize_t result = readBytes(MySettings.fd, &testchar, 1);
			if (result != 1) {
				if (errno != EWOULDBLOCK) {
					char errorString[0xFF];
					strerror_r(errno, errorString, sizeof(errorString));
					fprintf(stderr, "Error reading data (%s). Exiting...\n", errorString);
					break;
				}
			}
			rxcount++;
			if (testchar != expected_char) {
				fprintf(stderr, "Error - unexpected value: %c, should be: %c\n", testchar, expected_char);
			}
			if (++expected_char > 'Z') {
				// wrap around
				expected_char = 'A';
			}

			// print status
			if (rxcount % 1000 == 0) {
				fprintf(stderr, "rx: %ld\n", rxcount);
			}
		}
	}

	/* ---- TX mode ---- */
	if (MySettings.sermode == TX) {

		// Reset the serial device file descriptor for non-blocking read / write
		fcntl(MySettings.fd, F_SETFL, O_NONBLOCK);

		if (configureport(&MySettings) != 0) {
			return -1;
		}

		// Begin writing
		char testchar = 'A';
		while (MySettings.fd > 0) {
			// Wait until port is writeable
			struct pollfd pollOptions;
			pollOptions.fd = MySettings.fd;
			pollOptions.events = POLLOUT;
			pollOptions.revents = 0;
			poll(&pollOptions, 1, 200);

			// write the data
			ssize_t result = writeBytes(MySettings.fd, &testchar, 1);
			if (result != 1) {
				if (errno != EWOULDBLOCK) {
					char errorString[0xFF];
					strerror_r(errno, errorString, sizeof(errorString));
					fprintf(stderr, "Error writing data (%s). Exiting...\n", errorString);
					break;
				}
			}
			txcount++;
			if (++testchar > 'Z') {
				// wrap around
				testchar = 'A';
			}
			//usleep(50);

			// print status
			if (txcount % 1000 == 0) {
				fprintf(stderr, "tx: %ld\n", txcount);
			}
		}
	}

	fprintf(stderr, "ERROR- you must select a mode with -r or -t\n");
	return 0;
}

Thanks again for your help. Please let us know if you have any questions on how to run the loopback test.

-Mark

If you have a serial console attached at the time of crash, you should be able to see a crash dump (or with magic-sysrq). A crash dump/OOPS message would help.

If any of the crashes are related to J17, you’ll probably need to be sure the camera module is not installed, and software related to the camera’s J17 disabled. See the previous message on J17 and camera.

(P.S. Thanks Dennis for creating such a nice MCVE test!)

In order to run the loopback with ttyS0 available for the serial console, we have to run the test with ttyUSB0 and ttyUSB1, and it seems to be more difficult to crash the OS, but we observed a lot of error messages when the loopback test is started or stopped:

ubuntu@tegra-ubuntu:~$ Loading the saved-state of the serial devices...
Illegal UART type: undefined
/dev/ttyS0 at 0x0000 (irq = 68) is a undefined
 * Setting up X socket directories...                                    [ OK ]
 * speech-dispatcher disabled; edit /etc/default/speech-dispatcher
 * Starting NTP server ntpd                                              [ OK ]
saned disabled; edit /etc/default/saned

ubuntu@tegra-ubuntu:~$ [  314.327382] ftdi_sio ttyUSB1: failed to get modem status: -110
[  319.324438] ftdi_sio ttyUSB1: error from flowcontrol urb

ubuntu@tegra-ubuntu:~$ [  388.152683] usb 1-3.2: device not accepting address 8, error -71
[  390.791265] usb 1-3.2: device not accepting address 9, error -71
[  393.439626] usb 1-3.2: device not accepting address 10, error -71
[  394.349230] usb 1-3.2: device not accepting address 11, error -71
[  394.356762] hub 1-3:1.0: unable to enumerate USB device on port 2

[  584.872425] tegra-xhci tegra-xhci: ERROR: unexpected command completion code 0x0.
[  586.150432] usb 1-3.4: device not accepting address 22, error -71
[  587.101089] tegra-xhci tegra-xhci: ERROR: unexpected command completion code 0x0.
[  588.380745] usb 1-3.4: device not accepting address 23, error -71
[  588.387612] hub 1-3:1.0: unable to enumerate USB device on port 4
[  589.335925] tegra-xhci tegra-xhci: ERROR: unexpected command completion code 0x0.
[  590.620557] usb 1-3.3: device not accepting address 24, error -71
[  591.581367] tegra-xhci tegra-xhci: ERROR: unexpected command completion code 0x0.
[  592.860441] usb 1-3.3: device not accepting address 25, error -71
[  593.821912] tegra-xhci tegra-xhci: ERROR: unexpected command completion code 0x0.
[  595.100790] usb 1-3.3: device not accepting address 26, error -71
[  596.051025] tegra-xhci tegra-xhci: ERROR: unexpected command completion code 0x0.
[  597.330315] usb 1-3.3: device not accepting address 27, error -71
[  597.337749] hub 1-3:1.0: unable to enumerate USB device on port 3

Sometimes the USB driver fails completely with output like this:

ubuntu@tegra-ubuntu:~$ [  738.873091] usb 1-3: device not accepting address 45, error -22
[  738.879255] tegra-xhci tegra-xhci: Error while assigning device slot ID
[  738.887038] hub 1-0:1.0: couldn't allocate port 3 usb_device

which requires the SOM to be hard reset/rebooted to get USB mouse/keyboard back.

But… here is the good stuff…

ubuntu@tegra-ubuntu:~$
ubuntu@tegra-ubuntu:~$
ubuntu@tegra-ubuntu:~$ [  152.835687] usb 1-3.4: device not accepting address 6, error -71
[  155.475352] usb 1-3.4: device not accepting address 6, error -71
[  158.126061] usb 1-3.4: device not accepting address 7, error -71
[  160.765790] usb 1-3.4: device not accepting address 8, error -71
[  162.355784] usb 1-3.3: device not accepting address 5, error -71
[  163.223472] tegra-xhci tegra-xhci: ERROR: unexpected command completion code 0x0.
[  164.505679] usb 1-3.4: device not accepting address 9, error -71
[  165.377791] hub 1-3:1.0: couldn't allocate port 4 usb_device
[  165.796830] usb 1-3.3: device not accepting address 5, error -71
[  167.075958] usb 1-3.3: device not accepting address 5, error -71
[  167.740476] Unable to handle kernel paging request at virtual address 00001c60
[  167.747748] pgd = ffffffc0c0be5000
[  167.751181] [00001c60] *pgd=0000000140be2003, *pmd=0000000000000000
[  167.757546] Internal error: Oops: 96000046 [#1] PREEMPT SMP
[  167.763145] Enter nvdumper_crash_setup_regs
[  167.767495] nvdumper: all registers are saved.
[  167.767508] nvdumper: all registers are saved.
[  167.767519] nvdumper: all registers are saved.
[  167.780917] nvdumper: all registers are saved.
[  167.785405] Modules linked in: rfcomm bnep bcmdhd cfg80211 bluedroid_pm
[  167.792280] CPU: 0 PID: 2376 Comm: sertest Not tainted 3.10.96-tegra #1
[  167.798929] task: ffffffc0c1800ec0 ti: ffffffc0c0bf0000 task.ti: ffffffc0c0bf0000
[  167.806464] PC is at handle_cmd_completion+0x2a8/0x370
[  167.811640] LR is at handle_cmd_completion+0x7c/0x370
[  167.816723] pc : [<ffffffc0006534b0>] lr : [<ffffffc000653284>] pstate: 60000185
[  167.824136] sp : ffffffc0c0bf3250
[  167.827478] x29: ffffffc0c0bf3250 x28: ffffffc000f44c50
[  167.832876] x27: ffffffc0012c720f x26: ffffffc0ffc0aac0
[  167.838268] x25: 0000000000000000 x24: ffffffc0fe7c9000
[  167.843657] x23: 0000000080014700 x22: ffffff8000678700
[  167.849051] x21: ffffffc0fb09bbc0 x20: ffffff800067c820
[  167.854441] x19: ffffffc0fe7c9000 x18: ffffffc000b46ff0
[  167.859827] x17: 000000000000000e x16: 0000000000000007
[  167.865215] x15: 0000000000000001 x14: 0000000000000019
[  167.870606] x13: 0000000000000033 x12: 000000000000004c
[  167.875999] x11: 0000000000000067 x10: 0000000000000000
[  167.881389] x9 : 00000000d6d6a3e8 x8 : 0000000000000003
[  167.886778] x7 : 0000000000000020 x6 : ffffff80006f0c00
[  167.892168] x5 : ffffff8000678ff0 x4 : 4000000000000080
[  167.897556] x3 : 0000000000003800 x2 : 0000000004002c01
[  167.902941] x1 : 0000000000000018 x0 : 0000000000000000
[  167.908335]
[  167.908335] PC: 0xffffffc000653430:
[  167.913327] 3430  9b030481 121e7800 b9004020 aa1303e0 2a1803e1 97fff844 14000044 f94092a0
[  167.921796] 3450  39402e81 b91c6001 f94092a0 91400400 91310000 97ea2ae3 1400003c 8b38ce78
[  167.930257] 3470  aa1303e0 f9409301 aa1403e2 97fff500 350006c0 f9409300 39402e81 b91c6001
[  167.938716] 3490  f9409300 91400400 91310000 97ea2ad5 1400002e 8b38ce78 f9409300 39402e81
[  167.947178] 34b0  b91c6001 9103c260 97ea2ace 14000027 aa1303e0 aa1603e1 aa1403e2 97fffb55
[  167.955640] 34d0  14000022 aa1303e0 aa1403e1 aa1603e2 97fff856 1400001d aa1303e0 aa1403e1
[  167.964100] 34f0  aa1603e2 97ffff25 14000018 53187c42 8b22ce60 f9409001 b40000a1 aa1303e0
[  167.972557] 3510  aa1403e2 97fff4da 14000010 f9400260 f9400000 b0004c01 91196021 97fa1a12
[  167.981026]
[  167.981026] LR: 0xffffffc000653204:
[  167.986019] 3204  d65f03c0 a9bc7bfd 910003fd a90153f3 a9025bf5 a90363f7 aa0003f3 aa0103f4
[  167.994474] 3224  b9400c38 f9400037 f9404415 f94016b6 f9401aa0 aa1603e1 97fff752 b50000a0
[  168.003130] 3244  b9498260 321c0000 b9098260 140000c5 eb0002ff 540000a0 b9498260 321b0000
[  168.011844] 3264  b9098260 140000bf 39402e81 51006020 7100041f 540001c8 aa1303e0 97fff838
[  168.020577] 3284  340000a0 aa1303e0 f9404661 97fff20b 140000b4 f9404660 f9401416 f9400800
[  168.029170] 32a4  eb0002df 540000a1 140000ae f9400aa0 eb0002df 54001560 53187f18 b9400ec2
[  168.037434] 32c4  12161440 52870003 6b03001f 540010c0 6b03001f 54000268 52858001 6b01001f
[  168.045695] 32e4  54000e00 6b01001f 54000108 52848001 6b01001f 540003a0 52850001 6b01001f
[  168.053956]
[  168.053956] SP: 0xffffffc0c0bf31d0:
[  168.058911] 31d0  0067c820 ffffff80 fb09bbc0 ffffffc0 00678700 ffffff80 80014700 00000000
[  168.067174] 31f0  fe7c9000 ffffffc0 00000000 00000000 ffc0aac0 ffffffc0 012c720f ffffffc0
[  168.075432] 3210  00f44c50 ffffffc0 c0bf3250 ffffffc0 00653284 ffffffc0 c0bf3250 ffffffc0
[  168.083691] 3230  006534b0 ffffffc0 60000185 00000000 fb09bbc0 ffffffc0 006786f0 ffffff80
[  168.091953] 3250  c0bf3290 ffffffc0 006548f4 ffffffc0 fe7c9000 ffffffc0 00000003 00000000
[  168.100213] 3270  fe7c9048 ffffffc0 006f0820 ffffff80 0067c7f0 ffffff80 00000047 00000000
[  168.108473] 3290  c0bf32b0 ffffffc0 00654b40 ffffffc0 fe7c9000 ffffffc0 00000000 00000000
[  168.116736] 32b0  c0bf32f0 ffffffc0 0064582c ffffffc0 fe652818 ffffffc0 fe65cc00 ffffffc0
[  168.125002]
[  168.125002] X18: 0xffffffc000b46f70:
[  168.130045] 6f70  00459af4 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
[  168.138305] 6f90  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.146570] 6fb0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.154834] 6fd0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.163094] 6ff0  00000000 3b6e20c8 76dc4190 4db26158 edb88320 d6d6a3e8 9b64c2b0 a00ae278
[  168.171355] 7010  6f636361 00746e75 00000000 00000000 00195ca0 ffffffc0 0045c3dc ffffffc0
[  168.179615] 7030  0045c1d0 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
[  168.187888] 7050  00000000 00000000 00000000 00000000 0045bf68 ffffffc0 00000000 00000000
[  168.196184]
[  168.196184] X19: 0xffffffc0fe7c8f80:
[  168.201231] 8f80  00000000 00000000 81a40002 00003274 fb09aec0 ffffffc0 00000000 00000000
[  168.209526] 8fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.217824] 8fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.226120] 8fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.234415] 9000  fe65cc00 ffffffc0 fbb3a400 ffffffc0 006f0000 ffffff80 006f0020 ffffff80
[  168.242714] 9020  006f0800 ffffff80 006f0c00 ffffff80 006f0820 ffffff80 09000124 140000fd
[  168.251015] 9040  00400002 0184f525 1ff21ff1 00000000 00000000 00000000 01000000 00000000
[  168.259316] 9060  00000000 00000000 00001000 0000000c 00000000 00000000 00000000 00000000
[  168.267626]
[  168.267626] X21: 0xffffffc0fb09bb40:
[  168.272676] bb40  0615f000 ffffff80 06161000 ffffff80 00000004 00000000 fe00b259 ffffffc0
[  168.280985] bb60  fe65aed8 ffffffc0 fe2c8858 ffffffc0 fb09a9f0 ffffffc0 fe2bccf0 ffffffc0
[  168.289289] bb80  ffffffff ffffffff ffffffff ffffffff f8af3d80 ffffffc0 ffffffff ffffffff
[  168.297597] bba0  ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
[  168.305908] bbc0  fab66400 ffffffc0 fab66400 ffffffc0 00678730 ffffff80 fab66400 ffffffc0
[  168.314226] bbe0  00000073 00000000 00678700 ffffff80 fab66400 ffffffc0 00000070 00000000
[  168.322546] bc00  fb09bc00 ffffffc0 fb09bc00 ffffffc0 00000001 00000000 00000001 000000fb
[  168.330861] bc20  00000000 00000005 00000000 00000000 00000000 00000000 00000000 00000000
[  168.339184]
[  168.339184] X24: 0xffffffc0fe7c8f80:
[  168.344235] 8f80  00000000 00000000 81a40002 00003274 fb09aec0 ffffffc0 00000000 00000000
[  168.352552] 8fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.360869] 8fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.369187] 8fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.377504] 9000  fe65cc00 ffffffc0 fbb3a400 ffffffc0 006f0000 ffffff80 006f0020 ffffff80
[  168.385824] 9020  006f0800 ffffff80 006f0c00 ffffff80 006f0820 ffffff80 09000124 140000fd
[  168.394140] 9040  00400002 0184f525 1ff21ff1 00000000 00000000 00000000 01000000 00000000
[  168.402461] 9060  00000000 00000000 00001000 0000000c 00000000 00000000 00000000 00000000
[  168.410782]
[  168.410782] X26: 0xffffffc0ffc0aa40:
[  168.415834] aa40  00000000 00000000 00000000 007b007b 00000000 00000000 00000000 00000000
[  168.424153] aa60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.432470] aa80  ffc0aa80 ffffffc0 ffc0aa80 ffffffc0 ff0ee0c0 ffffffc0 00000000 00000000
[  168.440791] aaa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.449111] aac0  00000047 00000000 00000047 00000000 00000000 00042004 01173de0 ffffffc0
[  168.457431] aae0  ffc04440 ffffffc0 00000000 00000000 01378598 ffffffc0 00000000 00000000
[  168.465753] ab00  0000000f 00000000 0112e2e0 ffffffc0 001206e4 ffffffc0 fb09adc0 ffffffc0
[  168.474076] ab20  00000104 00000000 00000000 00000000 0001074d 00000000 00000000 00000000
[  168.482400]
[  168.482400] X27: 0xffffffc0012c718f:
[  168.487451] 718c  ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.495771] 71ac  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.504089] 71cc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.512406] 71ec  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.520725] 720c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.529045] 722c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.537364] 724c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.545683] 726c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.553999] 728c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  168.562319]
[  168.562319] X28: 0xffffffc000f44bd0:
[  168.567369] 4bd0  5f515249 52504f4e 0045424f 00000000 5f515249 45524f4e 53455551 00000054
[  168.575692] 4bf0  5f515249 48544f4e 44414552 00000000 5f515249 55414f4e 4e454f54 00000000
[  168.584013] 4c10  53515249 5455415f 5445444f 00544345 53515249 5045525f 0059414c 00000000
[  168.592336] 4c30  53515249 4941575f 474e4954 00000000 53515249 4e45505f 474e4944 00000000
[  168.600658] 4c50  7376642f 7469672f 7269642f 672f7974 6d2d7469 65747361 696c5f72 2f78756e
[  168.608977] 4c70  6e72656b 6b2f6c65 656e7265 72692f6c 61682f71 656c646e 0000632e 00000000
[  168.617297] 4c90  20717269 68207525 6c646e61 25207265 65204670 6c62616e 69206465 7265746e
[  168.625616] 4cb0  74707572 00000a73 7376642f 7469672f 7269642f 672f7974 6d2d7469 65747361
[  168.633939]
[  168.633939] X29: 0xffffffc0c0bf31d0:
[  168.638991] 31d0  0067c820 ffffff80 fb09bbc0 ffffffc0 00678700 ffffff80 80014700 00000000
[  168.647312] 31f0  fe7c9000 ffffffc0 00000000 00000000 ffc0aac0 ffffffc0 012c720f ffffffc0
[  168.655634] 3210  00f44c50 ffffffc0 c0bf3250 ffffffc0 00653284 ffffffc0 c0bf3250 ffffffc0
[  168.663956] 3230  006534b0 ffffffc0 60000185 00000000 fb09bbc0 ffffffc0 006786f0 ffffff80
[  168.672279] 3250  c0bf3290 ffffffc0 006548f4 ffffffc0 fe7c9000 ffffffc0 00000003 00000000
[  168.680601] 3270  fe7c9048 ffffffc0 006f0820 ffffff80 0067c7f0 ffffff80 00000047 00000000
[  168.688922] 3290  c0bf32b0 ffffffc0 00654b40 ffffffc0 fe7c9000 ffffffc0 00000000 00000000
[  168.697246] 32b0  c0bf32f0 ffffffc0 0064582c ffffffc0 fe652818 ffffffc0 fe65cc00 ffffffc0
[  168.705565]
[  168.707066] Process sertest (pid: 2376, stack limit = 0xffffffc0c0bf0058)
[  168.713853] Stack: (0xffffffc0c0bf3250 to 0xffffffc0c0bf4000)
[  168.719604] 3240:                                     c0bf3290 ffffffc0 006548f4 ffffffc0
[  168.727784] 3260: fe7c9000 ffffffc0 00000003 00000000 fe7c9048 ffffffc0 006f0820 ffffff80
[  168.735963] 3280: 0067c7f0 ffffff80 00000047 00000000 c0bf32b0 ffffffc0 00654b40 ffffffc0
[  168.744138] 32a0: fe7c9000 ffffffc0 00000000 00000000 c0bf32f0 ffffffc0 0064582c ffffffc0
[  168.752316] 32c0: fe652818 ffffffc0 fe65cc00 ffffffc0 fe65282c ffffffc0 fe7c9000 ffffffc0
[  168.760492] 32e0: 00000000 00000000 00000047 00000000 c0bf3330 ffffffc0 0061d078 ffffffc0
[  168.768668] 3300: fe65cc00 ffffffc0 00000180 00000000 00000047 00000000 c0bf34a0 ffffffc0
[  168.776845] 3320: 00000000 00000000 00000004 00000000 c0bf3350 ffffffc0 0011d438 ffffffc0
[  168.785022] 3340: ffc0aac0 ffffffc0 fb09adc0 ffffffc0 c0bf33c0 ffffffc0 0011d5d4 ffffffc0
[  168.793200] 3360: ffc0aac0 ffffffc0 ffc0ab4c ffffffc0 fb09adc0 ffffffc0 c0bf34a0 ffffffc0
[  168.801378] 3380: 01378578 ffffffc0 012c7000 ffffffc0 012c7000 ffffffc0 00002010 ffffff80
[  168.809554] 33a0: fe65cc00 ffffffc0 c0bf0000 ffffffc0 00f44c90 ffffffc0 0011d5c8 ffffffc0
[  168.817732] 33c0: c0bf33f0 ffffffc0 001207a8 ffffffc0 ffc0aac0 ffffffc0 ffc0ab4c ffffffc0
[  168.825908] 33e0: 0000200c ffffff80 000b0e70 ffffffc0 c0bf3410 ffffffc0 0011c8f0 ffffffc0
[  168.834083] 3400: 00000047 00000000 00000000 00000000 c0bf3430 ffffffc0 000859dc ffffffc0
[  168.842259] 3420: 00000047 00000000 000813cc ffffffc0 c0bf3450 ffffffc0 000813d4 ffffffc0
[  168.850436] 3440: 00000047 00000000 00000047 00000000 c0bf35c0 ffffffc0 00084e04 ffffffc0
[  168.858612] 3460: 00000340 00000000 00000000 00000000 c0bf35c0 ffffffc0 00ad1f74 ffffffc0
[  168.866787] 3480: 60000345 00000000 00000001 00000000 00000340 00000000 00678720 ffffff80
[  168.874962] 34a0: 00000000 00000000 00001fed 00000000 00000000 00000000 00000000 00000000
[  168.883139] 34c0: 004c4b40 00000000 00000000 00000000 002f6b68 ffffffc0 002f6b38 ffffffc0
[  168.891314] 34e0: 00000006 00000000 00000074 00000000 00000063 00000000 00000061 00000000
[  168.899491] 3500: 00000020 00000000 00000020 00000000 00000072 00000000 0000006e 00000000
[  168.907668] 3520: 00000064 00000000 804ac01a 9144d442 420d19d2 009e55c8 00000340 00000000
[  168.915845] 3540: 00000000 00000000 fe7c9048 ffffffc0 00000340 00000000 00000000 00000000
[  168.924022] 3560: 00000000 00000000 00000340 00000000 00678720 ffffff80 fe65cc00 ffffffc0
[  168.932196] 3580: 0138c6e8 ffffffc0 c0bf35c0 ffffffc0 00ad1f70 ffffffc0 c0bf35c0 ffffffc0
[  168.940373] 35a0: 00ad1f74 ffffffc0 60000345 00000000 00000000 00000000 00000000 00000000
[  168.948548] 35c0: c0bf35e0 ffffffc0 006514d4 ffffffc0 fe7c9000 ffffffc0 00000000 00000000
[  168.956726] 35e0: c0bf3620 ffffffc0 0063ee58 ffffffc0 fe7c9000 ffffffc0 fc262000 ffffffc0
[  168.964903] 3600: f5a5e800 ffffffc0 050f9000 ffffff80 fe7c9048 ffffffc0 00678720 ffffff80
[  168.973081] 3620: c0bf3680 ffffffc0 00615684 ffffffc0 f5a5e800 ffffffc0 00000001 00000000
[  168.981257] 3640: 00000005 00000000 fa139400 ffffffc0 00000005 00000000 00000003 00000000
[  168.989433] 3660: 00000002 00000000 00000001 00000000 c0bf3680 ffffffc0 ffffffd0 00000000
[  168.997608] 3680: c0bf36c0 ffffffc0 006166b8 ffffffc0 f5a5e800 ffffffc0 00000001 00000000
[  169.005786] 36a0: f634a800 ffffffc0 ffffffff 00ffffff c0bf36f0 ffffffc0 0061e608 ffffffc0
[  169.013962] 36c0: c0bf3770 ffffffc0 00616c14 ffffffc0 f5a5e800 ffffffc0 ffffffb9 00000000
[  169.022139] 36e0: 00000003 00000000 fa139400 ffffffc0 00000003 00000000 00000000 00000000
[  169.030319] 3700: 00000003 00000000 fe65cc00 ffffffc0 011aa3c8 ffffffc0 fa139480 ffffffc0
[  169.038497] 3720: 00000002 0000000a f5a5e888 ffffffc0 00fcc220 ffffffc0 00fcc1f0 ffffffc0
[  169.046674] 3740: c0bf3770 ffffffc0 00616c00 ffffffc0 f5a5e800 ffffffc0 00616bb8 ffffffc0
[  169.054851] 3760: 00000000 00000000 e823c3c0 ffffffc0 c0bf3800 ffffffc0 00616efc ffffffc0
[  169.063028] 3780: f5a5e800 ffffffc0 ffffff92 00000000 00000000 00000000 fa119800 ffffffc0
[  169.071206] 37a0: 00000410 00000000 00000000 00000000 000d26b4 ffffffc0 00000000 00000000
[  169.079382] 37c0: fa36f570 ffffffc0 ff802ec0 ffffffc0 02000112 08000000 60010403 02010600
[  169.087558] 37e0: 00000103 00000000 fa119800 ffffffc0 c0bf3830 ffffffc0 006190c0 ffffffc0
[  169.095734] 3800: c0bf3830 ffffffc0 006190d0 ffffffc0 fa139400 ffffffc0 f5a5e800 ffffffc0
[  169.103910] 3820: 00000123 00000000 00000000 00000000 c0bf3880 ffffffc0 0062c82c ffffffc0
[  169.112086] 3840: f5a5e800 ffffffc0 00000410 00000000 00000410 00000000 f634a930 ffffffc0
[  169.120263] 3860: f5a5e9f0 ffffffc0 00ad1fc0 ffffffc0 00000003 01030004 0008cd68 ffffffc0
[  169.128439] 3880: c0bf3890 ffffffc0 00622ec8 ffffffc0 c0bf38b0 ffffffc0 00622f24 ffffffc0
[  169.136616] 38a0: f5a5e888 ffffffc0 f5a5e800 ffffffc0 c0bf38e0 ffffffc0 00623ed4 ffffffc0
[  169.144793] 38c0: f5a5e888 ffffffc0 00623ec0 ffffffc0 f634a888 ffffffc0 f634a930 ffffffc0
[  169.152970] 38e0: c0bf38f0 ffffffc0 004e9bc8 ffffffc0 c0bf3910 ffffffc0 004e9c5c ffffffc0
[  169.161147] 3900: f5a5e888 ffffffc0 f5a5e930 ffffffc0 c0bf3940 ffffffc0 004eac88 ffffffc0
[  169.169324] 3920: f5a5e888 ffffffc0 f5a5e930 ffffffc0 f634a888 ffffffc0 f5a5e930 ffffffc0
[  169.177502] 3940: c0bf39d0 ffffffc0 004eabe0 ffffffc0 fa149830 ffffffc0 fa1498d8 ffffffc0
[  169.185679] 3960: f5a5e888 ffffffc0 f5a5e930 ffffffc0 fa149998 ffffffc0 00000004 00000000
[  169.193856] 3980: 000d26b4 ffffffc0 00000000 00000000 fa36f570 ffffffc0 00ad18e0 ffffffc0
[  169.202031] 39a0: f5a5e930 ffffffc0 fa1498d8 ffffffc0 c0bf39d0 ffffffc0 004eabc4 ffffffc0
[  169.210209] 39c0: fa149830 ffffffc0 004eaba4 ffffffc0 c0bf3a60 ffffffc0 004eae84 ffffffc0
[  169.218387] 39e0: fa149830 ffffffc0 00000004 00000000 fa1498d8 ffffffc0 00000340 00000000
[  169.226565] 3a00: fe657440 ffffffc0 0bc00000 00000000 00b44a38 ffffffc0 00020902 00000000
[  169.234741] 3a20: fa36f570 ffffffc0 00ad1d74 ffffffc0 fa1498d8 ffffffc0 00000340 00000000
[  169.242916] 3a40: c0bf3a60 ffffffc0 004eae74 ffffffc0 fa149830 ffffffc0 00000004 00000000
[  169.251093] 3a60: c0bf3a90 ffffffc0 00623094 ffffffc0 fa149800 ffffffc0 ffffffed 00000000
[  169.259270] 3a80: fa149830 ffffffc0 fa36e000 ffffffc0 c0bf3ac0 ffffffc0 00662f30 ffffffc0
[  169.267446] 3aa0: fc676180 ffffffc0 ffffffed 00000000 df6d1000 ffffffc0 00000000 00000000
[  169.275621] 3ac0: c0bf3b00 ffffffc0 004308b4 ffffffc0 df6d1000 ffffffc0 fe657440 ffffffc0
[  169.283798] 3ae0: 00000000 00000000 01190248 ffffffc0 00b44808 ffffffc0 01190248 ffffffc0
[  169.291975] 3b00: c0bf3b10 ffffffc0 00432cfc ffffffc0 c0bf3b40 ffffffc0 00433540 ffffffc0
[  169.300152] 3b20: 00000000 00000000 ec307d80 ffffffc0 fe657440 ffffffc0 01190248 ffffffc0
[  169.308326] 3b40: c0bf3bb0 ffffffc0 0019ba00 ffffffc0 fe688000 ffffffc0 fa36f570 ffffffc0
[  169.316503] 3b60: ec307d80 ffffffc0 ec307d90 ffffffc0 f636d5c0 ffffffc0 0019b904 ffffffc0
[  169.324681] 3b80: 00000026 00000000 ec307d80 ffffffc0 c0bf3d58 ffffffc0 ff802ec0 ffffffc0
[  169.332856] 3ba0: 00000100 00000000 0019ba5c ffffffc0 c0bf3bf0 ffffffc0 0019332c ffffffc0
[  169.341032] 3bc0: ec307d80 ffffffc0 fa36f570 ffffffc0 00000000 00000000 ec307d90 ffffffc0
[  169.349208] 3be0: ec307d80 ffffffc0 fa36f570 ffffffc0 c0bf3c30 ffffffc0 0019345c ffffffc0
[  169.357384] 3c00: c0bf3d58 ffffffc0 c0bf3e80 ffffffc0 00020902 00000000 00000000 00000000
[  169.365558] 3c20: c0bf3d48 ffffffc0 00000000 00000000 c0bf3c50 ffffffc0 001a6814 ffffffc0
[  169.373733] 3c40: c0bf3db0 ffffffc0 c0bf3e80 ffffffc0 c0bf3ce0 ffffffc0 001a6a34 ffffffc0
[  169.381911] 3c60: ec307d80 ffffffc0 c0bf3db0 ffffffc0 c1f8a000 ffffffc0 c0bf3e80 ffffffc0
[  169.390088] 3c80: 00000041 00000000 ffffff9c 00000000 00000000 00000000 00000038 00000000
[  169.398264] 3ca0: 01150000 ffffffc0 c0bf0000 ffffffc0 fa36f570 ffffffc0 00000000 00000000
[  169.406440] 3cc0: 00000000 00000000 001a69c0 ffffffc0 00000000 00000000 00000000 00000000
[  169.414617] 3ce0: c0bf3d80 ffffffc0 001a7b60 ffffffc0 00000001 00000000 c0bf3e80 ffffffc0
[  169.422795] 3d00: c1f8a000 ffffffc0 ffffff9c 00000000 da231e7c 0000007f ffffff9c ffffffff
[  169.430972] 3d20: 00000116 00000000 00000038 00000000 01150000 ffffffc0 c0bf0000 ffffffc0
[  169.439148] 3d40: 00000000 00000000 fd64cea0 ffffffc0 fa5b35c0 ffffffc0 00000000 ffffffc0
[  169.447323] 3d60: c0bf3da0 ffffffc0 000e0fc0 ffffffc0 00000001 00000000 00ad1a20 ffffffc0
[  169.455499] 3d80: c0bf3e40 ffffffc0 00195660 ffffffc0 00000003 00000000 00000001 00000000
[  169.463676] 3da0: c1f8a000 ffffffc0 ffffff9c 00000000 fd64cea0 ffffffc0 fa5b35c0 ffffffc0
[  169.471851] 3dc0: 0842ec6d 00000007 c1f8a025 ffffffc0 00000000 00000000 fd8b96c0 ffffffc0
[  169.480027] 3de0: fa36f570 ffffffc0 00000101 00000004 00000000 00000000 ffffff9c 00000000
[  169.488203] 3e00: da231e7c 0000007f ffffff9c ffffffff 00000116 00000000 00000038 00000000
[  169.496380] 3e20: 01150000 ffffffc0 c0bf0000 ffffffc0 c0bf3e40 ffffffc0 00195644 ffffffc0
[  169.504556] 3e40: c0bf3e90 ffffffc0 001958f8 ffffffc0 da231e78 0000007f 00412000 00000000
[  169.512733] 3e60: 00000000 00000000 00000902 00000000 da231e7c 0000007f ffffff9c ffffffff
[  169.520908] 3e80: 00020902 00000000 00000026 00000100 da231ba0 0000007f 000853c8 ffffffc0
[  169.529084] 3ea0: da231e78 0000007f 00412000 00000000 ffffffff ffffffff a9ff8f8c 0000007f
[  169.537261] 3ec0: 80000000 00000000 00000015 00000000 ffffff9c ffffffff da231e7c 0000007f
[  169.545436] 3ee0: 00000902 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  169.553610] 3f00: aa082009 0000007f da231cbf 0000007f 00000038 00000000 fefefeff fefefefe
[  169.561786] 3f20: ffffffff 00000000 0000000c 00000000 00000000 00000000 ffffffff ffffffff
[  169.569962] 3f40: 00000040 00000000 aa0be000 0000007f a9ff8f20 0000007f 00412028 00000000
[  169.578140] 3f60: da231a10 0000007f da231e78 0000007f 00412000 00000000 da231e7c 0000007f
[  169.586318] 3f80: 00000000 00000000 00000000 00000000 00000000 00000000 00412000 00000000
[  169.594494] 3fa0: 00000000 00000000 00000000 00000000 00000000 00000000 da231ba0 0000007f
[  169.602670] 3fc0: 004012c0 00000000 da231ba0 0000007f a9ff8f8c 0000007f 80000000 00000000
[  169.610849] 3fe0: ffffff9c ffffffff 00000038 00000000 ef300a24 db300220 ff300a24 ff300a24
[  169.619019] Call trace:
[  169.621482] [<ffffffc0006534b0>] handle_cmd_completion+0x2a8/0x370
[  169.627669] [<ffffffc0006548f4>] xhci_handle_event+0xac/0x148
[  169.633419] [<ffffffc000654b40>] xhci_irq+0x1b0/0x2a4
[  169.638475] [<ffffffc00064582c>] tegra_xhci_irq+0xc0/0x100
[  169.643965] [<ffffffc00061d078>] usb_hcd_irq+0x40/0x78
[  169.649108] [<ffffffc00011d438>] handle_irq_event_percpu+0xd8/0x22c
[  169.655376] [<ffffffc00011d5d4>] handle_irq_event+0x48/0x74
[  169.660953] [<ffffffc0001207a8>] handle_fasteoi_irq+0xc4/0x10c
[  169.666789] [<ffffffc00011c8f0>] generic_handle_irq+0x28/0x40
[  169.672541] [<ffffffc0000859dc>] handle_IRQ+0x94/0xc8
[  169.677597] [<ffffffc0000813d4>] gic_handle_irq+0x74/0x194
[  169.683083] Exception stack(0xffffffc0c0bf3460 to 0xffffffc0c0bf3580)
[  169.689527] 3460: 00000340 00000000 00000000 00000000 c0bf35c0 ffffffc0 00ad1f74 ffffffc0
[  169.697704] 3480: 60000345 00000000 00000001 00000000 00000340 00000000 00678720 ffffff80
[  169.705879] 34a0: 00000000 00000000 00001fed 00000000 00000000 00000000 00000000 00000000
[  169.714056] 34c0: 004c4b40 00000000 00000000 00000000 002f6b68 ffffffc0 002f6b38 ffffffc0
[  169.722231] 34e0: 00000006 00000000 00000074 00000000 00000063 00000000 00000061 00000000
[  169.730407] 3500: 00000020 00000000 00000020 00000000 00000072 00000000 0000006e 00000000
[  169.738583] 3520: 00000064 00000000 804ac01a 9144d442 420d19d2 009e55c8 00000340 00000000
[  169.746758] 3540: 00000000 00000000 fe7c9048 ffffffc0 00000340 00000000 00000000 00000000
[  169.754935] 3560: 00000000 00000000 00000340 00000000 00678720 ffffff80 fe65cc00 ffffffc0
[  169.763112] [<ffffffc000084e04>] el1_irq+0x84/0xf0
[  169.767911] [<ffffffc0006514d4>] xhci_cancel_cmd+0xf0/0x108
[  169.773489] [<ffffffc00063ee58>] xhci_address_device+0x160/0x2d0
[  169.779496] [<ffffffc000615684>] hub_set_address+0x54/0xe8
[  169.784983] [<ffffffc0006166b8>] hub_port_init+0x3bc/0x7ac
[  169.790470] [<ffffffc000616c14>] usb_reset_and_verify_device+0x16c/0x3c8
[  169.797171] [<ffffffc000616efc>] finish_port_resume+0x8c/0x254
[  169.803005] [<ffffffc0006190d0>] usb_port_resume+0x278/0x2fc
[  169.808672] [<ffffffc00062c82c>] generic_resume+0x1c/0x24
[  169.814076] [<ffffffc000622ec8>] usb_resume_device+0x78/0x8c
[  169.819740] [<ffffffc000622f24>] usb_resume_both+0x48/0xe4
[  169.825232] [<ffffffc000623ed4>] usb_runtime_resume+0x14/0x1c
[  169.830984] [<ffffffc0004e9bc8>] __rpm_callback+0x38/0x68
[  169.836387] [<ffffffc0004e9c5c>] rpm_callback+0x64/0x8c
[  169.841616] [<ffffffc0004eac88>] rpm_resume+0x390/0x520
[  169.846842] [<ffffffc0004eabe0>] rpm_resume+0x2e8/0x520
[  169.852070] [<ffffffc0004eae84>] __pm_runtime_resume+0x6c/0x90
[  169.857909] [<ffffffc000623094>] usb_autopm_get_interface+0x24/0x68
[  169.864182] [<ffffffc000662f30>] serial_install+0x64/0xf4
[  169.869589] [<ffffffc0004308b4>] tty_driver_install_tty+0x18/0x28
[  169.875685] [<ffffffc000432cfc>] tty_init_dev+0x58/0x170
[  169.880999] [<ffffffc000433540>] tty_open+0x108/0x340
[  169.886059] [<ffffffc00019ba00>] chrdev_open+0xfc/0x1bc
[  169.891288] [<ffffffc00019332c>] do_dentry_open+0x1b8/0x284
[  169.896861] [<ffffffc00019345c>] finish_open+0x64/0x80
[  169.902006] [<ffffffc0001a6814>] do_last+0x570/0x6e0
[  169.906977] [<ffffffc0001a6a34>] path_openat+0xb0/0x494
[  169.912207] [<ffffffc0001a7b60>] do_filp_open+0x30/0x80
[  169.917433] [<ffffffc000195660>] do_sys_open+0x13c/0x1d8
[  169.922747] [<ffffffc0001958f8>] SyS_openat+0xb4/0x14c
[  169.927891] Code: 1400002e 8b38ce78 f9409300 39402e81 (b91c6001)
[  169.933989] ---[ end trace 2199a9b6cb3658a0 ]---
[  169.945533] Kernel panic - not syncing: Fatal exception in interrupt
[  169.951902] CPU2: stopping
[  169.954636] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D      3.10.96-tegra #1
[  169.962119] Call trace:
[  169.964583] [<ffffffc000089d28>] dump_backtrace+0x0/0xf4
[  169.969898] [<ffffffc00008a028>] show_stack+0x14/0x1c
[  169.974958] [<ffffffc0002f76dc>] dump_stack+0x20/0x28
[  169.980015] [<ffffffc00009184c>] ipi_cpu_stop+0x48/0x6c
[  169.985244] [<ffffffc000091ce8>] handle_IPI+0xc4/0x11c
[  169.990385] [<ffffffc00008146c>] gic_handle_irq+0x10c/0x194
[  169.995958] Exception stack(0xffffffc0ff417d80 to 0xffffffc0ff417ea0)
[  170.002404] 7d80: 1fe4b2b0 ffffffc0 00000001 00000000 ff417ee0 ffffffc0 0075eb64 ffffffc0
[  170.010584] 7da0: 80000145 00000000 00000001 00000000 8007b000 00000000 8007d000 00000000
[  170.018762] 7dc0: 00000000 00000000 34155555 00000000 592f5a00 00000027 0a46cc35 00000000
[  170.026938] 7de0: 00000018 00000000 82711000 002d511e 34155555 00000000 ffffffff 00ffffff
[  170.035114] 7e00: cfaa0b08 00000000 00000000 00000000 1ed21000 00000000 00000004 00000000
[  170.043292] 7e20: 6fda31d0 0000007f 6fda31cc 0000007f 6fda31c8 0000007f f3e76c53 00000000
[  170.051470] 7e40: 000e31b4 ffffffc0 80d3c430 0000007f 00000000 00000000 1fe4b2b0 ffffffc0
[  170.059645] 7e60: 00000001 00000000 919d92c6 00000027 919d4472 00000027 1fe4b620 ffffffc0
[  170.067822] 7e80: 00000002 00000000 8007b000 00000000 8007d000 00000000 00080200 ffffffc0
[  170.075997] [<ffffffc000084e04>] el1_irq+0x84/0xf0
[  170.080798] [<ffffffc00075ed68>] cpuidle_idle_call+0x188/0x294
[  170.086634] [<ffffffc000086724>] arch_cpu_idle+0xc/0x24
[  170.091863] [<ffffffc0000f7c5c>] cpu_idle_loop+0x21c/0x284
[  170.097351] [<ffffffc0000f7ce4>] freezing_slow_path+0x0/0x84
[  170.103015] [<ffffffc000acc4a4>] secondary_start_kernel+0x148/0x154
[  170.109282] CPU1: stopping
[  170.112006] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D      3.10.96-tegra #1
[  170.119484] Call trace:
[  170.121941] [<ffffffc000089d28>] dump_backtrace+0x0/0xf4
[  170.127254] [<ffffffc00008a028>] show_stack+0x14/0x1c
[  170.132311] [<ffffffc0002f76dc>] dump_stack+0x20/0x28
[  170.137366] [<ffffffc00009184c>] ipi_cpu_stop+0x48/0x6c
[  170.142593] [<ffffffc000091ce8>] handle_IPI+0xc4/0x11c
[  170.147734] [<ffffffc00008146c>] gic_handle_irq+0x10c/0x194
[  170.153306] Exception stack(0xffffffc0ff40fd80 to 0xffffffc0ff40fea0)
[  170.159752] fd80: 1fe3e2b0 ffffffc0 00000003 00000000 ff40fee0 ffffffc0 0075eb64 ffffffc0
[  170.167929] fda0: 80000345 00000000 00000001 00000000 8007b000 00000000 8007d000 00000000
[  170.176104] fdc0: 00000000 00000000 34155555 00000000 592f5a00 00000027 0a46cc35 00000000
[  170.184279] fde0: 00000018 00000000 82711000 002d511e 34155555 00000000 ffffffff 00ffffff
[  170.192454] fe00: cfaa0b08 00000000 01000000 00000000 4d0408b1 58083d8f 0048282b 537c7123
[  170.200629] fe20: 00000001 00000000 12d44a04 48401817 0a304088 41a5006c 04cd1844 44133374
[  170.208803] fe40: 000a1060 62040387 708059b0 2c21310e 121c225a 84acd677 1fe3e2b0 ffffffc0
[  170.216980] fe60: 00000003 00000000 919e8e72 00000027 9174f49f 00000027 1fe3e780 ffffffc0
[  170.225156] fe80: 00000001 00000000 8007b000 00000000 8007d000 00000000 00080200 ffffffc0
[  170.233329] [<ffffffc000084e04>] el1_irq+0x84/0xf0
[  170.238123] [<ffffffc00075ed68>] cpuidle_idle_call+0x188/0x294
[  170.243959] [<ffffffc000086724>] arch_cpu_idle+0xc/0x24
[  170.249188] [<ffffffc0000f7c5c>] cpu_idle_loop+0x21c/0x284
[  170.254677] [<ffffffc0000f7ce4>] freezing_slow_path+0x0/0x84
[  170.260340] [<ffffffc000acc4a4>] secondary_start_kernel+0x148/0x154
[  170.266608] CPU3: stopping
[  170.269331] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D      3.10.96-tegra #1
[  170.276810] Call trace:
[  170.279268] [<ffffffc000089d28>] dump_backtrace+0x0/0xf4
[  170.284583] [<ffffffc00008a028>] show_stack+0x14/0x1c
[  170.289638] [<ffffffc0002f76dc>] dump_stack+0x20/0x28
[  170.294692] [<ffffffc00009184c>] ipi_cpu_stop+0x48/0x6c
[  170.299921] [<ffffffc000091ce8>] handle_IPI+0xc4/0x11c
[  170.305061] [<ffffffc00008146c>] gic_handle_irq+0x10c/0x194
[  170.310633] Exception stack(0xffffffc0ff41bd80 to 0xffffffc0ff41bea0)
[  170.317078] bd80: 1fe582b0 ffffffc0 00000003 00000000 ff41bee0 ffffffc0 0075eb64 ffffffc0
[  170.325256] bda0: 80000345 00000000 00000001 00000000 8007b000 00000000 8007d000 00000000
[  170.333432] bdc0: 00000000 00000000 34155555 00000000 592f5a00 00000027 0a46cc35 00000000
[  170.341609] bde0: 00000018 00000000 82711000 002d511e 34155555 00000000 ffffffff 00ffffff
[  170.349784] be00: cfaa0b08 00000000 01000000 00000000 c85064ed 16065f06 79116044 1325b01c
[  170.357960] be20: 00000003 00000000 fffffffe fbf7ffff 502c4baf c6306a13 87420a56 c2344140
[  170.366134] be40: 340d7128 9e6f67ef 4932a5b7 fff65bbf 68828436 0ce454cb 1fe582b0 ffffffc0
[  170.374310] be60: 00000003 00000000 919ea2ca 00000027 9008767c 00000027 1fe588e0 ffffffc0
[  170.382485] be80: 00000003 00000000 8007b000 00000000 8007d000 00000000 00080200 ffffffc0
[  170.390659] [<ffffffc000084e04>] el1_irq+0x84/0xf0
[  170.395453] [<ffffffc00075ed68>] cpuidle_idle_call+0x188/0x294
[  170.401288] [<ffffffc000086724>] arch_cpu_idle+0xc/0x24
[  170.406515] [<ffffffc0000f7c5c>] cpu_idle_loop+0x21c/0x284
[  170.412003] [<ffffffc0000f7ce4>] freezing_slow_path+0x0/0x84
[  170.417666] [<ffffffc000acc4a4>] secondary_start_kernel+0x148/0x154
[  170.431039] ------------------------------------------------
[  170.436698] CPU0 Status: online
[  170.439843] current task: ffffffc0c1800ec0
[  170.443941] ARM_x0    = 0x0000000000000000
[  170.448039] ARM_x1    = 0xffffffc05e400000
[  170.452135] ARM_x2    = 0xffffffc05e400000
[  170.456232] ARM_x3    = 0x0000000000000001
[  170.460329] ARM_x4    = 0x0000000000000000
[  170.464427] ARM_x5    = 0xffffffc01fe32f48
[  170.468523] ARM_x6    = 0xffffffc0002f6b40
[  170.472619] ARM_x7    = 0xffffffc0002f6b38
[  170.476717] ARM_x8    = 0x0000000000000001
[  170.480812] ARM_x9    = 0x0000000000000001
[  170.484912] ARM_x10   = 0xffffffc0002f6b38
[  170.489009] ARM_x11   = 0xffffffc0002f6b40
[  170.493107] ARM_x12   = 0xffffffc01fe32f48
[  170.497204] ARM_x13   = 0x0000000000000000
[  170.501301] ARM_x14   = 0x0000000000000001
[  170.505399] ARM_x15   = 0xffffffc05e400000
[  170.509496] ARM_x16   = 0xffffffc05e400000
[  170.513593] ARM_x17   = 0x0000000000000000
[  170.517691] ARM_x18   = 0xffffffc0c0bf2d20
[  170.521788] ARM_x19   = 0xffffffc000807b5c
[  170.525885] ARM_x20   = 0xffffffc000807aa4
[  170.529981] ARM_x21   = 0x0000000000000000
[  170.534077] ARM_x22   = 0x0000000000000000
[  170.538174] ARM_x23   = 0xffffffc0c0bf2e30
[  170.542272] ARM_x24   = 0x0000000000000025
[  170.546369] ARM_x25   = 0x0000000000000000
[  170.550466] ARM_x26   = 0xffffffc0ffc0aac0
[  170.554564] ARM_x27   = 0xffffffc0012c720f
[  170.558661] ARM_x28   = 0xffffffc000f44c50
[  170.562758] ARM_x29   = 0xffffffc0c0bf2d20
[  170.566856] ARM_x30   = 0xffffffc000807ab4
[  170.570952] ARM_sp    = 0xffffffc0c0bf2d20
[  170.575049] ARM_pc    = 0xffffffc000807b5c
[  170.579146] ACTLR_EL1      = 0x0000000000000000
[  170.583677] AFSR0_EL1      = 0x0000000000000000
[  170.588207] AFSR1_EL1      = 0x0000000000000000
[  170.592737] AMAIR_EL1      = 0x0000000000000000
[  170.597266] CONTEXTIDR_EL1 = 0x0000000000000000
[  170.601796] CPACR_EL1      = 0x0000000000300000
[  170.606325] CSSELR_EL1     = 0x0000000000000000
[  170.610854] FAR_EL1        = 0x0000000000001c60
[  170.615383] ESR_EL1        = 0x0000000096000046
[  170.619912] PAR_EL1        = 0x0000000000000000
[  170.624442] MAIR_EL1       = 0x0000000000000000
[  170.628972] RMR_EL1        = 0x0000000000000000
[  170.633502] SCTLR_EL1      = 0x0000000000000000
[  170.638032] TEECR32_EL1    = 0x0000000000000000
[  170.642560] TEEHBR32_EL1   = 0x0000000000000000
[  170.647089] TPIDR_EL1      = 0x0000000000000000
[  170.651618] TCR_EL1        = 0x0000000000000000
[  170.656148] TTBR0_EL1      = 0x0000000000000000
[  170.660677] TTBR1_EL1      = 0x0000000000000000
[  170.665207] VBAR_EL1       = 0x0000000000000000
[  170.669736] TPIDRRO_EL0    = 0x0000000000000000
[  170.674267] TPIDR_EL0      = 0x0000000000000000
[  170.678796] ------------------------------------------------
[  170.684452] CPU1 Status: online
[  170.687596] current task: ffffffc0ff404180
[  170.691693] ARM_x0    = 0x00000000000002b8
[  170.695791] ARM_x1    = 0xffffffc05e4002b8
[  170.699887] ARM_x2    = 0xffffffc05e400000
[  170.703984] ARM_x3    = 0x0000000000000001
[  170.708081] ARM_x4    = 0x0000000000000001
[  170.712179] ARM_x5    = 0x002296898345f400
[  170.716276] ARM_x6    = 0x0000000034155555
[  170.720372] ARM_x7    = 0x00ffffffffffffff
[  170.724470] ARM_x8    = 0x00000000cd2b5f08
[  170.728566] ARM_x9    = 0x00000000cd2b5f08
[  170.732662] ARM_x10   = 0x00ffffffffffffff
[  170.736760] ARM_x11   = 0x0000000034155555
[  170.740856] ARM_x12   = 0x002296898345f400
[  170.744952] ARM_x13   = 0x0000000000000001
[  170.749050] ARM_x14   = 0x0000000000000001
[  170.753148] ARM_x15   = 0xffffffc05e400000
[  170.757244] ARM_x16   = 0xffffffc05e4002b8
[  170.761341] ARM_x17   = 0x00000000000002b8
[  170.765438] ARM_x18   = 0xffffffc0ff40fce0
[  170.769534] ARM_x19   = 0xffffffc000807b5c
[  170.773631] ARM_x20   = 0x0000000000000001
[  170.777727] ARM_x21   = 0x0000000000000200
[  170.781823] ARM_x22   = 0x0000000000000100
[  170.785919] ARM_x23   = 0xffffffc001378578
[  170.790017] ARM_x24   = 0xffffffc0012c7000
[  170.794114] ARM_x25   = 0xffffffc0012c7000
[  170.798211] ARM_x26   = 0xffffff8000002010
[  170.802308] ARM_x27   = 0xffffffc000080200
[  170.806405] ARM_x28   = 0xffffffc0ff40c000
[  170.810502] ARM_x29   = 0xffffffc0ff40fce0
[  170.814599] ARM_x30   = 0xffffffc000807ab4
[  170.818696] ARM_sp    = 0xffffffc0ff40fce0
[  170.822791] ARM_pc    = 0xffffffc000807b5c
[  170.826888] ACTLR_EL1      = 0x0000000000000000
[  170.831419] AFSR0_EL1      = 0x0000000000000000
[  170.835948] AFSR1_EL1      = 0x0000000000000000
[  170.840478] AMAIR_EL1      = 0x0000000000000000
[  170.845009] CONTEXTIDR_EL1 = 0x0000000000000000
[  170.849540] CPACR_EL1      = 0x0000000000300000
[  170.854069] CSSELR_EL1     = 0x0000000000000000
[  170.858600] FAR_EL1        = 0x0000000000000000
[  170.863129] ESR_EL1        = 0x0000000000000000
[  170.867658] PAR_EL1        = 0x0000000000000000
[  170.872189] MAIR_EL1       = 0x0000000000000000
[  170.876718] RMR_EL1        = 0x0000000000000000
[  170.881247] SCTLR_EL1      = 0x0000000000000000
[  170.885776] TEECR32_EL1    = 0x0000000000000000
[  170.890306] TEEHBR32_EL1   = 0x0000000000000000
[  170.894836] TPIDR_EL1      = 0x0000000000000000
[  170.899365] TCR_EL1        = 0x0000000000000000
[  170.903894] TTBR0_EL1      = 0x0000000000000000
[  170.908424] TTBR1_EL1      = 0x0000000000000000
[  170.912953] VBAR_EL1       = 0x0000000000000000
[  170.917483] TPIDRRO_EL0    = 0x0000000000000000
[  170.922012] TPIDR_EL0      = 0x0000000000000000
[  170.926542] ------------------------------------------------
[  170.932200] CPU2 Status: online
[  170.935342] current task: ffffffc0ff410c80
[  170.939439] ARM_x0    = 0x0000000000000570
[  170.943537] ARM_x1    = 0xffffffc05e400570
[  170.947633] ARM_x2    = 0xffffffc05e400000
[  170.951731] ARM_x3    = 0x0000000000000001
[  170.955828] ARM_x4    = 0x0000000000000002
[  170.959924] ARM_x5    = 0x002296898345f400
[  170.964020] ARM_x6    = 0x0000000034155555
[  170.968116] ARM_x7    = 0x00ffffffffffffff
[  170.972214] ARM_x8    = 0x00000000cd2b5f08
[  170.976309] ARM_x9    = 0x00000000cd2b5f08
[  170.980406] ARM_x10   = 0x00ffffffffffffff
[  170.984503] ARM_x11   = 0x0000000034155555
[  170.988599] ARM_x12   = 0x002296898345f400
[  170.992695] ARM_x13   = 0x0000000000000002
[  170.996793] ARM_x14   = 0x0000000000000001
[  171.000890] ARM_x15   = 0xffffffc05e400000
[  171.004987] ARM_x16   = 0xffffffc05e400570
[  171.009083] ARM_x17   = 0x0000000000000570
[  171.013180] ARM_x18   = 0xffffffc0ff417ce0
[  171.017278] ARM_x19   = 0xffffffc000807b5c
[  171.021376] ARM_x20   = 0x0000000000000001
[  171.025473] ARM_x21   = 0x0000000000000200
[  171.029571] ARM_x22   = 0x0000000000000100
[  171.033668] ARM_x23   = 0xffffffc001378578
[  171.037767] ARM_x24   = 0xffffffc0012c7000
[  171.041864] ARM_x25   = 0xffffffc0012c7000
[  171.045961] ARM_x26   = 0xffffff8000002010
[  171.050058] ARM_x27   = 0xffffffc000080200
[  171.054155] ARM_x28   = 0xffffffc0ff414000
[  171.058254] ARM_x29   = 0xffffffc0ff417ce0
[  171.062350] ARM_x30   = 0xffffffc000807ab4
[  171.066446] ARM_sp    = 0xffffffc0ff417ce0
[  171.070542] ARM_pc    = 0xffffffc000807b5c
[  171.074639] ACTLR_EL1      = 0x0000000000000000
[  171.079169] AFSR0_EL1      = 0x0000000000000000
[  171.083698] AFSR1_EL1      = 0x0000000000000000
[  171.088228] AMAIR_EL1      = 0x0000000000000000
[  171.092759] CONTEXTIDR_EL1 = 0x0000000000000000
[  171.097287] CPACR_EL1      = 0x0000000000300000
[  171.101816] CSSELR_EL1     = 0x0000000000000000
[  171.106345] FAR_EL1        = 0x0000000000000000
[  171.110874] ESR_EL1        = 0x0000000000000000
[  171.115404] PAR_EL1        = 0x0000000000000000
[  171.119934] MAIR_EL1       = 0x0000000000000000
[  171.124463] RMR_EL1        = 0x0000000000000000
[  171.128992] SCTLR_EL1      = 0x0000000000000000
[  171.133521] TEECR32_EL1    = 0x0000000000000000
[  171.138049] TEEHBR32_EL1   = 0x0000000000000000
[  171.142579] TPIDR_EL1      = 0x0000000000000000
[  171.147108] TCR_EL1        = 0x0000000000000000
[  171.151636] TTBR0_EL1      = 0x0000000000000000
[  171.156164] TTBR1_EL1      = 0x0000000000000000
[  171.160694] VBAR_EL1       = 0x0000000000000000
[  171.165222] TPIDRRO_EL0    = 0x0000000000000000
[  171.169750] TPIDR_EL0      = 0x0000000000000000
[  171.174279] ------------------------------------------------
[  171.179937] CPU3 Status: online
[  171.183079] current task: ffffffc0ff4101c0
[  171.187175] ARM_x0    = 0x0000000000000828
[  171.191273] ARM_x1    = 0xffffffc05e400828
[  171.195369] ARM_x2    = 0xffffffc05e400000
[  171.199466] ARM_x3    = 0x0000000000000001
[  171.203562] ARM_x4    = 0x0000000000000003
[  171.207659] ARM_x5    = 0x002296898345f400
[  171.211756] ARM_x6    = 0x0000000034155555
[  171.215854] ARM_x7    = 0x00ffffffffffffff
[  171.219951] ARM_x8    = 0x00000000cd2b5f08
[  171.224048] ARM_x9    = 0x00000000cd2b5f08
[  171.228144] ARM_x10   = 0x00ffffffffffffff
[  171.232240] ARM_x11   = 0x0000000034155555
[  171.236337] ARM_x12   = 0x002296898345f400
[  171.240435] ARM_x13   = 0x0000000000000003
[  171.244531] ARM_x14   = 0x0000000000000001
[  171.248629] ARM_x15   = 0xffffffc05e400000
[  171.252726] ARM_x16   = 0xffffffc05e400828
[  171.256823] ARM_x17   = 0x0000000000000828
[  171.260921] ARM_x18   = 0xffffffc0ff41bce0
[  171.265019] ARM_x19   = 0xffffffc000807b5c
[  171.269114] ARM_x20   = 0x0000000000000001
[  171.273211] ARM_x21   = 0x0000000000000200
[  171.277307] ARM_x22   = 0x0000000000000100
[  171.281404] ARM_x23   = 0xffffffc001378578
[  171.285502] ARM_x24   = 0xffffffc0012c7000
[  171.289599] ARM_x25   = 0xffffffc0012c7000
[  171.293696] ARM_x26   = 0xffffff8000002010
[  171.297792] ARM_x27   = 0xffffffc000080200
[  171.301888] ARM_x28   = 0xffffffc0ff418000
[  171.305985] ARM_x29   = 0xffffffc0ff41bce0
[  171.310081] ARM_x30   = 0xffffffc000807ab4
[  171.314178] ARM_sp    = 0xffffffc0ff41bce0
[  171.318274] ARM_pc    = 0xffffffc000807b5c
[  171.322371] ACTLR_EL1      = 0x0000000000000000
[  171.326900] AFSR0_EL1      = 0x0000000000000000
[  171.331430] AFSR1_EL1      = 0x0000000000000000
[  171.335958] AMAIR_EL1      = 0x0000000000000000
[  171.340489] CONTEXTIDR_EL1 = 0x0000000000000000
[  171.345018] CPACR_EL1      = 0x0000000000300000
[  171.349548] CSSELR_EL1     = 0x0000000000000000
[  171.354077] FAR_EL1        = 0x0000000000000000
[  171.358606] ESR_EL1        = 0x0000000000000000
[  171.363136] PAR_EL1        = 0x0000000000000000
[  171.367665] MAIR_EL1       = 0x0000000000000000
[  171.372194] RMR_EL1        = 0x0000000000000000
[  171.376722] SCTLR_EL1      = 0x0000000000000000
[  171.381252] TEECR32_EL1    = 0x0000000000000000
[  171.385781] TEEHBR32_EL1   = 0x0000000000000000
[  171.390310] TPIDR_EL1      = 0x0000000000000000
[  171.394841] TCR_EL1        = 0x0000000000000000
[  171.399371] TTBR0_EL1      = 0x0000000000000000
[  171.403900] TTBR1_EL1      = 0x0000000000000000
[  171.408430] VBAR_EL1       = 0x0000000000000000
[  171.412961] TPIDRRO_EL0    = 0x0000000000000000
[  171.417490] TPIDR_EL0      = 0x0000000000000000
[  171.422764] Rebooting in 30 seconds..[0000.214] [TegraBoot] (version 24.00.2015.42-mobile-f8f04b74)
[0000.219] Processing in cold boot mode Bootloader 2
[0000.224] A02 Bootrom Patch rev = 63
[0000.227] Power-up reason: software reset
[0000.231] No Battery Present
[0000.234] Platform has Ddr4 type ram
[0000.237] max77620 disabling SD1 Remote Sense
[0000.242] Setting Ddr voltage to 1125mv
[0000.246] Serial Number of Pmic Max77663: 0xf08a1
[0000.253] Entering ramdump check
[0000.256] Get RamDumpCarveOut = 0x0
[0000.260] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.265] Last reboot was clean, booting normally!
[0000.269] Sdram initialization is successful
[0000.274] SecureOs Carveout Base=0xff800000 Size=0x00800000
[0000.279] GSC1 Carveout Base=0xff700000 Size=0x00100000
[0000.284] GSC2 Carveout Base=0xff600000 Size=0x00100000
[0000.289] GSC3 Carveout Base=0xff500000 Size=0x00100000
[0000.294] GSC4 Carveout Base=0xff400000 Size=0x00100000
[0000.300] GSC5 Carveout Base=0xff300000 Size=0x00100000
[0000.305] BpmpFw Carveout Base=0xff2c0000 Size=0x00040000
[0000.310] Lp0 Carveout Base=0xff2bf000 Size=0x00001000
[0000.325] RamDump Carveout Base=0xff23f000 Size=0x00080000
[0000.331] Platform-DebugCarveout: 0
[0000.334] Nck Carveout Base=0xff03f000 Size=0x00200000
[0000.339] AOTAG Init Done
[0000.387] Using GPT Primary to query partitions
[0000.392] Loading Tboot-CPU binary
[0000.441] Verifying bootloader in OdmNonSecureSBK mode
[0000.451] Bootloader load address is 0xa0000000, entry address is 0xa0000258
[0000.458] Bootloader downloaded successfully.
[0000.462] Downloaded Tboot-CPU binary to 0xa0000258
[0000.467] MAX77620_GPIO1 Configured.
[0000.471] MAX77620_GPIO5 Configured.
[0000.475] CPU power rail is up
[0000.477] CPU clock enabled
[0000.481] Performing RAM repair
[0000.484] Updating A64 Warmreset Address to 0xa00002e9
[0000.501] Bootloader DTB Load Address: 0x83000000
[0000.518] Kernel DTB Load Address: 0x83080000
[0000.523] Loading cboot binary
[0000.617] Verifying bootloader in OdmNonSecureSBK mode
[0000.710] Bootloader load address is 0x8010fda8, entry address is 0x80110000
[0000.717] Bootloader downloaded successfully.
[0000.722] GPT: Partition NOT found !
[0000.725] Find Partition via GPT Failed
[0000.729] Find Partition via PT Failed
[0000.732] function NvTbootGetBinaryOffsets: 0x1 error
[0000.737] Error in NvTbootLoadBinary: 0x1 !
[0000.741] Next binary entry address: 0x80110000
[0000.745] BoardId: 2180
[0000.772] NvTbootI2cWrite(): error code 0x00045100 Error while starting write transaction
[0000.779] NvTbootI2cDeviceRead(): error code 0x00045001 Error while sending the offset to slave
[0000.788] NvTbootI2c: Read failed for slave 0xa2, offset 0x00 with error code 0x00045001
[0000.796] Display board id read failed
[0000.800] dram memory type is 3
[0000.803] WB0 init successful
[0000.830] Bpmp FW successfully loaded
[0000.833] Set NvDecSticky Bits
[0000.836] GSC1 address : ff700000
[0000.840] GSC2 address ff63fffc value c0edbbcc
[0000.845] GSC2 address : ff600000
[0000.848] GSC3 address : ff500000
[0000.852] GSC4 address : ff400000
[0000.856] GSC5 address : ff300000
[0000.859] GSC MC Settings done
[0000.863] TOS old plaintext Image length 65536
[0000.869] *** Secure OS image signature not verified ***
[0000.874] Loading and Validation of Secure OS Successful
[0000.879] NvTbootPackSdramParams: start.
[0000.884] NvTbootPackSdramParams: done.
[0000.888] Tegraboot started after 194218 us
[0000.892] Basic modules init took 307272 us
[0000.896] Sec Bootdevice Read Time = 194 ms, Read Size = 8464 KB
[0000.902] Sec Bootdevice Write Time = -1940251278 ms, Write Size = -738734375 KB
[0000.909] Next stage binary read took 12267 us
[0000.913] Carveout took 252377 us
[0000.916] CPU initialization took 143442 us
[0000.920] Total time taken by TegraBoot 715358 us

[0000.925] Starting CPU & Halting co-processor

64b[0001.063] LPDDR4 Training: Number of tables = 10
[0001.067] EMC Training (SRC-freq: 204000; DST-freq: 408000)
[0001.073] EMC Training Successful
[0001.076] EMC Training (SRC-freq: 204000; DST-freq: 665600)
[0001.082] EMC Training Successful
[0001.085] EMC Training (SRC-freq: 204000; DST-freq: 800000)
[0001.096] EMC Training Successful
[0001.099] EMC Training (SRC-freq: 204000; DST-freq: 1065600)
[0001.122] EMC Training Successful
[0001.125] EMC Training (SRC-freq: 204000; DST-freq: 1331200)
[0001.147] EMC Training Successful
[0001.150] EMC Training (SRC-freq: 204000; DST-freq: 1600000)
[0001.169] EMC Training Successful
[0001.173] Switching to 800000 KHz Success
[0001.212] LPDDR4 Training: Number of tables = 10

[code]

I perhaps should point out, though, that the above console output is with two USB to serial converters and could be a different manifestation than the issue we really hope to get addressed, (which is reliable UART to UART serial communication).

Unfortunately, it isn’t possible to get console output from a crash involving ttyTHS1 because it is also the console port.

We are working on reproducing the test with the camera module removed right now. We’ll post back shortly with results from that testing.

Thanks again for your help!

-Mark

Update: Removing the Camera module seems to make it more difficult to crash the OS, but it still crashes given enough time/data over the UARTs.

Is there ANYONE else out there who is actually using a /dev/ttyTHSX serial device to communicate with something? I’d be very suprised if that were the case.

Mark and I have tried many combinations of USB-serial converter and hardware UART and the resulting crash is VERY easy to duplicate. At this point I don’t think we should have to spend any more time trying things. We have provided a very specific recipe to recreate this crash, but it occurs much more easily than that. If any external device is connected to /dev/ttyTHS1 or THS2 or THS3, the act of simply passing data through that hardware UART can eventually cause the crash.

I think this shows that it is most certainly a kernel driver bug. At best it shows that the hardware UART driver has not been thoroughly tested. We cannot move forward on our project without reliable serial communications. Our only option now is to dig into the tegra-serial kernel driver and attempt to debug and fix it ourselves. I think it’s time for someone at nVidia to sit down and recreate it, and dedicate themselves to fixing it.

-Dennis

I’m going to somewhat randomly state a few things known about the problem (I don’t know how it all pieces together)…

In the OOPS, near the top of each call stack, I see “el1_irq”, which is the kernel mode IRQ. I can’t say exactly, but it seems the IRQ handler is crashing. Knowing your exact kernel config might help…does this differ from the default config, and are you using the 64-bit sample rootfs or the 32-bit sample rootfs? Regardless, there is either an issue with the IRQ handler or what the IRQ handler has available to determine which driver to load.

I’ve noticed in “setserial -a” outputs can have “UART: unknown”, but in “/sys/class/tty/ttySOMETHING_SPECIFIC/device/driver” that a device type is known. It seems device shows as “unknown” if /sys associates serial8250, but will show as 16550 if /sys uses 16550. The “unkown” may be related to some of the errors, and probably should show as 8250 in setserial if this is what /sys says. Just conjecture, perhaps the missing UART type showing as “unkown” in setserial also has an effect in the IRQ handler not knowing which driver to run. It seems those listing as 16550 do not have the issue, but this too is just a guess.

If this is correct, I suspect that adding an external PCIe 16550 type serial port card would cause the problem to go away. If this is true, then changing setup of any serial UART used to indicate 16550 instead of unknown would probably also eliminate the issue. If at all possible, do you have available a 16550 series serial port PCIe card you can try?

Hi Linuxdev,

To answer your 32/64 bit question, we have attempted standard installs using JetPack of both 32 and 64 bit installations and see the issue with both. (I believe the kernel space is still 64 bit in both cases, right?) Our exact kernel config is precisely what is in the JetPack installation… everything is NVIDIA stock for everything we are talking about in this post, nothing is home brewed.

At your suggestion we purchased a PCIe 16550 serial port card and installed it, but we have been unsuccessful in getting the device to be recognized. However, following a few LINUX setserial HOWTO’s and this posting: https://devtalk.nvidia.com/default/topic/936285/jetson-tx1/tx1-serial-port-configuration/ we have manually edited the /var/lib/setserial/autoserial.conf to include the following lines:

/dev/ttyTHS1 uart 16550A port 0x0000 irq 69 baud_base 115200 spd_normal
/dev/ttyTHS2 uart 16550A port 0x0000 irq 78 baud_base 115200 spd_normal
/dev/ttyTHS3 uart 16550A port 0x0000 irq 122 baud_base 115200 spd_normal

followed by a

sudo dpkg-reconfigure setserial

autosaving once and rebooting the computer.

After reboot, running “setserial -a /dev/ttyTHS1” for ttyTHS1, ttyTHS2, and ttyTHS3, all reflect the saved settings changes correctly as in the following:

/dev/ttyTHS1, Line 1, UART: 16550A, Port: 0x0000, IRQ: 69
              Baud_base: 115200, close_delay: 50, divisor:0
              closing_wait: 3000
              Flags: spd_normal

And even with this… we can crash the operating system. :/

What to investigate next?

-Mark

P.S. I have also attempted only setting THS2 and THS3, leaving the ttyS0 settings as original and setting nothing for THS1 to avoid a potential conflict with ttyS0, but the results are the same.