Tx dma map failure on link flapping

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
, ,
1

We’re running an NVIDIA Jetson Orin with Jetpack 5.15. We are using a custom carrier board with a number of I226 NICs.

When sending UDP traffic over one of the interfaces, the remote side of that interface can bring down the link and cause a kernel softlock. The UDP sendto calls can take upwards of 10s in some instances, even though the socket is configured in a non-blocking manner.

Interestingly, enabling IOMMU passthrough in the device tree seems to remove the issue. i.e. the following diff was made to our device tree for the given interfaces

	pcie@141a0000 {
		compatible = "nvidia,tegra234-pcie\0snps,dw-pcie";
		power-domains = <0x02 0x05>;
		reg = <0x00 0x141a0000 0x00 0x20000 0x00 0x3a000000 0x00 0x40000 0x00 0x3a040000 0x00 0x40000 0x00 0x3a080000 0x00 0x40000 0x2b 0x30000000 0x00 0x10000000>;
		reg-names = "appl\0config\0atu_dma\0dbi\0ecam";
		status = "okay";
		#address-cells = <0x03>;
		#size-cells = <0x02>;
		device_type = "pci";
		num-lanes = <0x08>;
		num-viewport = <0x08>;
		linux,pci-domain = <0x05>;
		clocks = <0x02 0xe1 0x02 0xea>;
		clock-names = "core\0core_m";
		resets = <0x02 0x82 0x02 0x81>;
		reset-names = "apb\0core";
		interrupts = <0x00 0x35 0x04 0x00 0x36 0x04>;
		interrupt-names = "intr\0msi";
		interconnects = <0x44 0xe2 0x44 0xe3>;
		interconnect-names = "dma-mem\0dma-mem";
-		iommus = <0x03 0x14>;
-		iommu-map = <0x00 0x03 0x14 0x1000>;
		msi-parent = <0x35 0x14>;
		msi-map = <0x00 0x35 0x14 0x1000>;
		dma-coherent;
-		iommu-map-mask = <0x00>;
		#interrupt-cells = <0x01>;
		interrupt-map-mask = <0x00 0x00 0x00 0x00>;
		interrupt-map = <0x00 0x00 0x00 0x00 0x01 0x00 0x35 0x04>;
		nvidia,dvfs-tbl = <0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0x27ac4000 0xc28cb00 0xc28cb00 0x27ac4000 0x5f5e1000 0xc28cb00 0x27ac4000 0x5f5e1000 0x7f22ff40>;
		nvidia,max-speed = <0x04>;
		nvidia,disable-aspm-states = <0x0f>;
		nvidia,controller-id = <0x02 0x05>;
		nvidia,tsa-config = <0x200b004>;
		nvidia,disable-l1-cpm;
		nvidia,aux-clk-freq = <0x13>;
		nvidia,preset-init = <0x05>;
		nvidia,aspm-cmrt = <0x3c>;
		nvidia,aspm-pwr-on-t = <0x14>;
		nvidia,aspm-l0s-entrance-latency = <0x03>;
		nvidia,bpmp = <0x02 0x05>;
		nvidia,aspm-cmrt-us = <0x3c>;
		nvidia,aspm-pwr-on-t-us = <0x14>;
		nvidia,aspm-l0s-entrance-latency-us = <0x03>;
		bus-range = <0x00 0xff>;
		ranges = <0x81000000 0x00 0x3a100000 0x00 0x3a100000 0x00 0x100000 0x82000000 0x00 0x40000000 0x2b 0x28000000 0x00 0x8000000 0xc3000000 0x27 0x40000000 0x27 0x40000000 0x03 0xe8000000>;
		nvidia,cfg-link-cap-l1sub = <0x1c4>;
		nvidia,cap-pl16g-status = <0x174>;
		nvidia,cap-pl16g-cap-off = <0x188>;
		nvidia,event-cntr-ctrl = <0x1d8>;
		nvidia,event-cntr-data = <0x1dc>;
		nvidia,dl-feature-cap = <0x30c>;
		nvidia,ptm-cap-off = <0x318>;
		vddio-pex-ctl-supply = <0x36>;
		phys = <0x37 0x38 0x39 0x3a 0x3b 0x3c 0x3d 0x3e>;
		phy-names = "p2u-0\0p2u-1\0p2u-2\0p2u-3\0p2u-4\0p2u-5\0p2u-6\0p2u-7";
		vpcie3v3-supply = <0x3f>;
		vpcie12v-supply = <0x40>;
		phandle = <0x376>;
	};

attached is a kernel trace, configured with some kprobe symbols, that traces through one of the failures.

We’ve tried looking through newer versions of the kernel to look for fixes, but have not found anything definitive. Is this a known issue? If not, are there reasons to not use the iommu passthrough?

trace.txt (18.3 MB)

4

Sorry for the late response.
Is this still an issue to support? Any result can be shared?

5

Yes, we were able to root cause the issue. There is a bug in the 5.10 igc kernel driver in the teardown path.

In short, igc_main.c implements two functions to cleanup the ring. igc_clean_tx_irq when a packet is sent, and igc_clean_tx_ring when the link is detected down. igc_clean_tx_ring improperly unmaps entries by setting the next_to_watch to NULL on the last slot, whereas igc_clean_tx_irq sets it (properly) on the first slot.

The results in a double free and, if using iova, a double alloc that will fail with TX DMA map failed errors in dmesg (the storm of these errors can cause synchronous flushes to console depending on your system settings, hence causing the kernel softlocks).

The fix is quite simple, and is already upstreamed in the kernel. The diff shown here should be sufficient:

diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index 631ce79..84dc3f1 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -225,6 +225,22 @@ static void igc_clean_tx_ring(struct igc_ring *tx_ring)
 	/* reset BQL for queue */
 	netdev_tx_reset_queue(txring_txq(tx_ring));
 
+	/* Zero the per-buffer state and the descriptor ring before reuse.
+	 * The loop above unmaps each buffer but, unlike igc_clean_tx_irq(),
+	 * leaves dma_unmap_len()/next_to_watch/skb set and clears next_to_watch
+	 * on the EOP slot rather than the first-of-packet slot where
+	 * igc_tx_map() stored it; the reset path also reuses this descriptor
+	 * ring without zeroing it. Without this, the first igc_clean_tx_irq()
+	 * after igc_up() (next_to_clean == 0) re-validates the stale slots and
+	 * frees them a second time: a double DMA unmap, which behind an IOMMU
+	 * double-frees the IOVA and surfaces as "TX DMA map failed", and a
+	 * double dev_kfree_skb_any()/use-after-free of the skb. Matches
+	 * igb_clean_tx_ring().
+	 */
+	memset(tx_ring->tx_buffer_info, 0,
+	       sizeof(*tx_ring->tx_buffer_info) * tx_ring->count);
+	memset(tx_ring->desc, 0, tx_ring->size);
+
 	/* reset next_to_use and next_to_clean */
 	tx_ring->next_to_use = 0;
 	tx_ring->next_to_clean = 0;