Hi,
We tested it with the source shared on May 26 with docker compose up -d.
Since the plan files are pre-compiled in the source, the test should be exactly the same as you provided recently.
Is there any difference between the setup?
We test it with Orin NX 16GB devkit with JetPack 6.1.
Is there setting have been applied between flashing and running the test?
Thanks.
The sources I shared yesterday are the scripts that generate the .plan files that I shared on May 26, I thought that might help. We have reproduced the problem both using the .plan files shared on the 26th and also regenerating everything on each device using the scripts from yesterday.
This is what we installed on each device after flashing:
apt-get)apt-get)apt-get)pip install)As I mentioned, we used 4 devices. Three of those devices are using JetPack 6.1, and the last one is using JetPack 6.2.
Hi,
Do you use apt-get to install docker?
If so, the latest version (default) should be v28.x instead of 26.1.3.
Do we miss anything in the setup?
Here are our detailed steps:
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
cd stress_test
docker compose up -d
Thanks.
We install Docker using the official Ubuntu version:
sudo apt-get install docker docker-compose-v2
Do you think this could be the issue?
While waiting to see if you can reproduce, we are going to execute the test on the 4 NX using the official Docker version.
Thanks.
Hi,
The docker version should not be the root cause.
But we get v28 when installing it with apt so want to double-confirm why the version is different.
In your environment, do you set up the BSP with standard JetPack 6.1/6.2?
Any customization in the kernel? Especially, do you use RT kernel?
Thanks.
On the devkit device, we use the standard BSP, yes.
On the Connecttech and Forecr devices, the installation was made by the providers.
We do not add any kernel customisation nor RT kernel.
Hi,
We would like to collect more info about this issue:
Thanks.
Hi,
Here are some updates for you.
The issue is reproduced in our environment with a DP display connected.
As we failed to reproduce the error without display, this issue should be related to HDA, just as your comment before.
Now our internal team is working on the issue. Will provide you with more info later.
Thanks.
Hi!
Thanks for the feedback! This is good news; hopefully, you will be able to see what’s going on.
Here are the answers to your previous questions:
As far as we know, there is no internal memory on the Jetson Orin NX, so the only ways to flash the system are either on an SD card or an NVME disk. On the four devices we have, only the DevKit has an SD card slot, so we consciously decided to use NVMe in all of the devices, including the DevKit.
Here are some details about the NMVE disks we use:
Two of the devices (DevKit and ForeCR) had displays connected to them, but the two ConnectTech devices do not and still fail.
Yes, we can. We will run the test on the DevKit with NVME.
Thanks for your help!
Alex
Hi,
We are now trying to reproduce this issue with a display connected but HDA is disabled.
Do you have some results about the below testing?
(we will try disabling the HDA hardware on the DTB too).
Thanks.
Hi,
Please also try to run the test without HDA.
To disable it, please change Orin NX platform DT file “hardware/nvidia/t23x/nv-public/tegra234-p3767.dtsi” in the codebase as mentioned below:
hda@3510000 {
- status = "okay";
+ status = "disabled";
};
Thanks.
Hello @AastaLLL,
I am working on this issue as well. I’m unable to find any reference for “hda” nor registry “3510000” even by reviewing all dtsi files from the L4T on JetPack 6.1.
Is there any way to disable it after flash? I guess the registry is also specific to the devkit and should be different with a custom carrier?
Thanks.
We urgently need a resolution on this issue affecting all NX devices. The devices crash so frequently that they are effectively non-functional. This is not just a development issue. We are now blocked from delivering systems to clients.
We’ve been raising these issues since March, and after four months with no meaningful progress, this is now a serious threat to our business.
Please escalate if necessary to provide us with concrete guidance or fixes.
Regards,
Alex
Hi!
We managed to find the file you asked to modify inside kernel_oot_modules_src.tbz2 inside Driver Package (BSP) Sources.
However, we still do not know where to go from here.
Are there instructions about how to deal with the files in that package?
Thanks!
Alex
Hi,
Sorry for keeping you waiting.
We did some testing and “might” found the cause of this issue.
(We are not 100% sure as it takes time to confirm if the WAR really works or not.)
nvme.use_threaded_interrupts=1)The NVMe WAR can be done by editing extlinux.conf or reflash with the corresponding command.
Please find the detailed information in the below document link:
VST crashes after a while when recording multiple streams on NVMe
Option 1:
…
Option 2:
Thanks.
Hi @AastaLLL ,
Thanks for the update and for investigating this further.
We will test the NVMe workaround (nvme.use_threaded_interrupts=1) immediately and report back as soon as we have results.
Please keep us informed of any progress on your end. If there’s an official bug ID or escalation path we can follow, we’d appreciate that as well.
Thanks again,
Alex
Hi,
Does the NVME WAR fix the issue on your side?
Our test can run for more than 2 days without hitting the error.
The fix (nvme.use_threaded_interrupts=1) is merged into our internal branch.
So you don’t need to manually apply it in the future release.
Thanks.
Hello,
We tried to implement the fix but I’m not sure it is actually taken in account from the kernel. We had a crash this morning on one of the unit.
Please note that I tried both methods, adding nvme.use_threaded_interrupts=1 to the APPEND section of extlinux.conf and reflashing the unit by adding -p “-C nvme.use_threaded_interrupts=1” to the initrd flash parameters.
With both methods I noticed that the nvme modules parameters does not contain use_threaded_interrupts
ls -l /sys/module/nvme/parameters/
total 0
-rw-r--r-- 1 root root 4096 Jun 27 13:44 io_queue_depth
-r--r--r-- 1 root root 4096 Jun 27 13:44 max_host_mem_size_mb
-r--r--r-- 1 root root 4096 Jun 27 13:44 noacpi
-rw-r--r-- 1 root root 4096 Jun 27 13:44 poll_queues
-rw-r--r-- 1 root root 4096 Jun 27 13:44 sgl_threshold
-r--r--r-- 1 root root 4096 Jun 27 13:44 use_cmb_sqes
-rw-r--r-- 1 root root 4096 Jun 27 13:44 write_queues
The only reference is found by querying the module info
modinfo -p nvme
use_threaded_interrupts: (int)
use_cmb_sqes:use controller's memory buffer for I/O SQes (bool)
max_host_mem_size_mb:Maximum Host Memory Buffer (HMB) size per controller (in MiB) (uint)
sgl_threshold:Use SGLs when average request segment size is larger or equal to this size. Use 0 to disable SGLs. (uint)
io_queue_depth:set io queue depth, should >= 2 and < 4096
write_queues:Number of queues to use for writes. If not set, reads and writes will share a queue set.
poll_queues:Number of queues to use for polled IO.
noacpi:disable acpi bios quirks (bool)
Can you please help confirm or not this parameter is actually used by the system?
Thanks.
Hi,
You can verify the setting by checking the /etc/modprobe.d/nvme.conf.
Below is our output for your reference:
$ cat /etc/modprobe.d/nvme.conf
...
options nvme use_threaded_interrupts=1
If the mode is already enabled in the testing, could you share more details about the experiments with us?
How many devices do you use and how long it takes to hit the error?
We didn’t reproduce the error with threaded IRQ mode for more than 45 hours.
Thanks.
Hi,
Just an update from our side for your reference.
Our device also met a system crash when running the app over the weekend without DISPLAY connected.
So this issue can still happen with the NVME threaded IRQ mode enabled.
To find the root cause, our internal team is dumping the kernel stack to check it further.
Will keep you updated once we get further info about the issue.
Thanks.