Orin Devkit uses jetpack 7.2 version. When using an nvme hard drive, there is a probability that the kernel will prompt nvme for prolonged continuous read and write operations
nvme0: I/O tag xxx (xxxx) QID x timeout, completion polled
I tested that both WD-SN7100 and Kingston KC3000 have this issue
Hi,
How do you flash the device?
Thanks
I am using Linux_for_Tegra
sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1
The nvme hard drive is only used as a data disk, and this issue can occur during continuous reading of the hard drive
Hi,
Could you please provide the flashing log and the serial log for us to review?
Thanks
uart.txt (99.9 KB)
flash.txt (112.3 KB)
I don’t think this has anything to do with the flashing method. Using a PCIe 4.0 NVMe hard drive to continuously read hundreds of GB of data makes it easy to reproduce this problem
I also tried to limit the PCIe bus to PCIe 3.0, but it didn’t work. This timeout will greatly reduce the read efficiency of the nvme hard drive
Hi,
Please refer to the similar topics R36.3 Patch to re-enable GICv2m for PCIe MSI interrupts and restore I/O performance and apply the patch to cross check the issue.
Thanks
dmesg.log (102.8 KB)
After modifying the patch to jp7.2, there were some errors that appeared to prevent GIC v2m from starting. At the same time, the performance of hard disk reading has decreased to 700mb/s. Through lspci vvv, it can be seen that MSI and MSI-X are not enabled
I encountered the same nvme ... timeout, completion polled problem and have a working port of the community GICv2m patch for JetPack 7.2.1 / L4T R39.2.1. Sharing the source changes and test results in case they help others, and for NVIDIA/community review.
Tested system
- Jetson AGX Orin developer kit, 32 GB
- L4T R39.2.1, Ubuntu 24.04
- Kernel
6.8.12-1021-tegra - WD_BLACK SN7100 1TB, firmware
7615M0WD - Onboard M.2 slot / PCIe C4 (
14160000.pcie), PCIe Gen4 x4
Before the patch
The drive sometimes reached several GB/s but also suffered repeated approximately 30-second stalls with timeout, completion polled. Even a read-only directory scan could trigger them while the OS was booted from eMMC. NVMe queue interrupts were all handled on CPU0.
Disabling APST, changing ASPM to performance, disabling HMB, disabling NVMe threaded interrupts, and disabling deep CPU idle did not resolve the problem in our tests. Temporarily bypassing MSI/MSI-X avoided the timeout messages in the short tests but reduced sequential throughput substantially, so we did not keep that workaround.
What changed
We ported the patches from Delwyn’s original R36.3 thread and Jonathan Schnitzler’s Linux 6.6 port. Credit goes to those authors for the underlying approach.
The attachment contains the port against NVIDIA’s R39.2.1 kernel sources and a device-tree conversion tool for the stock AGX Orin devkit DTB. It routes only C4 through GICv2m.
One important difference in this release: the stock GIC node has #address-cells = <0>. Adding the GICv2m child requires valid address/size cells and ranges. Our conversion uses two address cells and two size cells, and updates the existing PCI INTx maps to include the two parent address cells. This may help explain the Failed to allocate v2m resource / Failed to allocate gic_base resource errors reported above; it is not enough to paste the old child node into the new tree.
The kernel port also updates the DesignWare callbacks to .init / .msi_init, propagates IOMMU mapping failures, adjusts IRQ teardown ordering, and avoids disabling the unused DesignWare MSI IRQ during shutdown when GICv2m is in use.
The patched PCIe module must also be included in the initrd. Replacing only the root-filesystem copy does not update the driver loaded during early boot.
Results
GICv2m now initializes successfully, MSI-X remains enabled, and the NVMe queues deliver interrupts across all 12 CPU cores. PCIe remains at 16 GT/s x4, with normal power-management settings restored.
| Test | Result |
|---|---|
| Previously stalling directory scan, 154,640 files | 1.85 seconds |
| 120-second direct sequential read, eMMC-root boot | 482 GB read, 4.018 GB/s |
| 60-second direct sequential read, NVMe-root boot | 261 GB read, 4.357 GB/s |
| 60-second 4 KiB random read, queue depth 1 | About 29.3k IOPS |
| Four concurrent write jobs, 8 GiB total, CRC32C readback verification | Passed; 4.227 GB/s aggregate write throughput |
| Offline filesystem checks before/after the write test | Passed |
There were no NVMe timeout messages during those patched-kernel tests. SMART reported zero critical warnings, media errors and error-log entries. The system is now booting its root filesystem and EFI partition from NVMe; the original eMMC installation is retained for recovery.
Scope and precautions
This is a community workaround tested on one device, not an NVIDIA-approved fix or a long-term endurance result. It does not establish that CPU0 overload was the precise cause of every lost interrupt. The source attachment includes build notes, exact benchmark commands and selected logs, but no prebuilt boot images or automatic installer.
Keep a known-good kernel/initrd/DTB and a recoverable eMMC boot path. The device-tree tool deliberately checks the expected stock layout. Other boards/releases need separate review. Kernel upgrades require rebuilding and revalidating the changes. We checked module-version CRCs before reusing the installed NVIDIA modules: all 15,272 shared exported symbols matched in this build.
Could NVIDIA review the port, particularly the Tegra MSI/IOMMU mapping and the updated GIC device-tree layout, and advise whether a supported fix is planned?
Attachment: jetson-orin-r39.2.1-gicv2m-source-fix.zip
Hi,
Thanks for the patch. We will check it.
Starting from Jetpack 6, we use PCIe driver in upstream kernel so the function is maintained as offline patch. We have listed it in
Making sure you're not a bot!
Will check with our team and see if there is better approach for future releases.