Nvgpu PBDMA semaphore timeout on AGX Orin (JP6.2)

Dear NVIDIA team,
When I was testing iperf on the custom carrier board, the screen flickered and there were errors in the serial port log.

describe:
Module Orin 32GB JetPack6.2 L4T36.4.3
PCIE to 5-way gigabit network

Reproduction steps:
If Orin’s 5-road network is combined with X86 devices to form a local area network, and iperf is used simultaneously on the X86 end for 5-road testing, this phenomenon will occur:
The test instructions are as follows:

Orin :
iperf3 -s -p 5201
iperf3 -s -p 5202
iperf3 -s -p 5203
iperf3 -s -p 5204
iperf3 -s -p 5205

X86:
iperf3 -c 192.168.30.100 -b 1000M -t 10000 -p 5201
iperf3 -c 192.168.20.100 -b 1000M -t 10000 -p 5202
iperf3 -c 192.168.10.100 -b 1000M -t 10000 -p 5203
iperf3 -c 192.168.22.100 -b 1000M -t 10000 -p 5204
iperf3 -c 192.168.11.100 -b 1000M -t 10000 -p 5205

The problem phenomenon is as follows:

The DEBUG port log is as follows:

rk@Orin:~$ [  822.415948] nvgpu: 17000000.gpu ga10b_pbdma_handle_intr_0_acquire:646  [ERR]  semaphore acquire timeout!
[  822.436311] __ga10b__ Channel Status - chip ga10b
[  822.436316] __ga10b__ ---------------------------
[  822.436320] __ga10b__ 508-ga10b, TSG: 3, pid 2027, thread name gnome-shell, refs: 4, deterministic: no, domain name: (no domain)
[  822.436321] __ga10b__ channel status:  in use on_pbdma, on_eng, pbdma_busy busy
[  822.436324] __ga10b__ RAMFC: TOP: 8000001ff83c0960 PUT: 001ff83c0974 GET: 001ff83c0960 FETCH: 000000000000 HEADER: 2140006c COUNT: 11110000 SEMAPHORE: addr 002000130000 payload 0000000000000413 execute 00081003
[  822.436326] __ga10b__  
[  822.436328] __ga10b__ 509-ga10b, TSG: 2, pid 1174, thread name Xorg, refs: 2, deterministic: no, domain name: (no domain)
[  822.436329] __ga10b__ channel status:  in use idle not busy
[  822.436331] __ga10b__ RAMFC: TOP: 80000020040080c8 PUT: 0020040080c8 GET: 0020040080c8 FETCH: 000000000000 HEADER: 2140006c COUNT: 00000000 SEMAPHORE: addr 002004010000 payload 0000000000000000 execute 00100001
[  822.436332] __ga10b__  
[  822.436334] __ga10b__ 510-ga10b, TSG: 1, pid 1174, thread name Xorg, refs: 2, deterministic: no, domain name: (no domain)
[  822.436334] __ga10b__ channel status:  in use idle not busy
[  822.436336] __ga10b__ RAMFC: TOP: 8000002004033eb0 PUT: 002004033eb0 GET: 002004033eb0 FETCH: 000000000000 HEADER: 2140006c COUNT: 00000000 SEMAPHORE: addr 002004320000 paload rk@000Ori$ 0000000 execute 00000001
[  822.436337] __ga10b__  
[  822.436338] __ga10b__ 511-ga10b, TSG: 0, pid 1174, thread name Xorg, refs: 2, deterministic: no, domain name: (no domain)
[  822.436339] __ga10b__ channel status:  in use idle not busy
[  822.436340] __ga10b__ RAMFC: TOP: 80000020040480c0 PUT: 0020040480c0 GET: 0020040480c0 FETCH: 000000000000 HEADER: 2140006c COUNT: 00000000 SEMAPHORE: adayload 000Orin:~$ 0000000 execute 00100001
[  822.436341] __ga10b__  
[  822.436344] __ga10b__ PBDMA Status - chip ga10b
[  822.436345] __ga10b__ -------------------------
[  822.436347] __ga10b__ pbdma 0:
[  822.436351] __ga10b__   id: 3 - [tsg]     next_id: - -1 [channel] | status: valid
[  822.436356] __ga10b__   PBDMA_PUT 0000001ff83c0974 PBDMA_GET 0000001ff83c0960
[  
.436362] __ga10b__   4hPUTrk@   Orin:~000056  GP_GET  00000052  FETCH   00000052 HEADER 2140006c
[  822.436366] __ga10b__   HDR       2001001b  SHADOW0 04048098  SHADOW1 00002820
[  822.436368] __ga10b__ pbdma 1:
[  822.436369] __ga10b__   id: -1 - [channel] next_id: - -1 [channel] | status: invalid
[  822.436374] __ga10b__   PBDMA_PUT 00000018709db048 PBDMA_GET 00000018709db048
[  822.436380] __ga10b__   GP_PUT    00000000  GP_GET  590129af  FETCH   00000000 HEADER a097d380
638200__g4hb__rk@Ori       f2954bdb  SHADOW0 66d74091  SHADOW1 055b1a34
[  822.436385] __ga10b__ pbdma 2:
[  822.436386] __ga10b__   id: -1 - [channel] next_id: - -1 [channel] | status: invalid
[  822.436391] __ga10b__   PBDMA_PUT 000000272adb2760 PBDMA_GET 000000272adb2760
[  822.436397] __ga10b__   GP_PUT    00000000  GP_GET  4eadd294  FETCH   00000000 HEADER c1972304
OW1b2rk@Orin:~$ 822.436402] __ga10b__ pbdma 3: SHADOW0 600bb
[  822.436403] __ga10b__   id: -1 - [channel] next_id: - -1 [channel] | status: invalid
[  822.436408] __ga10b__   PBDMA_PUT 000000c8877258d4 PBDMA_GET 000000a013745d38
[  822.436414] __ga10b__   GP_PUT    00000000  GP_GET  1293bETCH   00Ori000000 HEADER e152ef60
[  822.436418] __ga10b__   HDR       e4d56e33  SHADOW0 099952d0  SHADOW1 295304a0
[  822.436419] __ga10b__ pbdma 4:
[  822.436420] __ga10b__   id: -1 - [channel] next_id: - -1 [channel] | status: invalid
[  822.436425] __ga10b__   PBDMA_PUT 000000af2ee96c54 PBDMA_GET 000000af2ee96c54
[  822.436430] __ga10b__   GP_PUT    00000000  GP_GET  12545100  FETCH   00000000 HEADER 0042211c
[  822.436434] __ga10b__   HDR       426804e3  SHADOW0 7c8c1382  SHADOW1 6229ee32

643__ga10b__ pbOrin:~$ [  822.436437] __ga10b__   id: -1 - [channel] next_id: - -1 [channel] | status: invalid
[  822.436442] __ga10b__   PBDMA_PUT 000000e2733791ac PBDMA_GET 000000e2733791ac
[  822.436448] __ga10b__   GP_PUT    00000000  GP_GET  2bbc00ec  FETCH   00000000 HEADER 00175894
[  822.436452] __ga10b__   HDR       3356347e  SHADOW0 2e10b2a8  SHADOW1 40a0a8a6
[  822.436453] __ga10b__  
[  822.436458] __ga10b__ ga10b eng 0: 
[  822.436459] __ga10b__ id: 3 (tsg), next_id: -1 (channel), ctx status: valid 
[  822.436460] __ga10b__  
[  822.436463] __ga10b__ ga10b eng 1: 
[  822.436464] __ga10b__ id: 3 (tsg), next_id: -1 (channel), ctx status: valid 
[  822.436465] __ga10b__  
[  822.436469] __ga10b__ ga10b eng 2: 
[  822.436470] __ga10b__ id: -1 (channel), next_id: -1 (channel), ctx status: invalid 
[  822.436471] __ga10b__  
[  822.436474] __ga10b__ ga10b eng 3: 
[  822.436474] __ga10b__ id: -1 (channel), next_id: -1 (channel), ctx status: invalid 
[  822.436475] __ga10b__  
[  822.436478] __ga10b__ ga10b eng 4: 
[  822.436479] __ga10b__ id: -1 (channel), next_id: -1 (channel), ctx status: invalid 
[  822.436479] __ga10b__  
[  822.436483] __ga10b__ ga10b eng 5: 
[  822.436483] __ga10b__ id: -1 (channel), next_id: -1 (channel), ctx status: invalid 
[  822.436484] __ga10b__  
[  822.436484] __ga10b__  
[  822.436487] nvgpu: 17000000.gpu          ga10b_pbdma_report_error:330  [ERR]  pbdma_intr_0(0)= 0x04000000 
[  822.436494] nvgpu: 17000000.gpu nvgpu_cic_mon_report_err_safety_services:97   [ERR]  Error reporting is not supported in this platform
[  822.436502] nvgpu: 17000000.gpu     nvgpu_set_err_notifier_locked:143  [ERR]  error notifier set to 24 for ch 508 owned by gnome-shell

I tried this method and replaced the nvgpu. Ko file, but the problem still exists

Hi,
Please also apply the patches listed in
Making sure you're not a bot!


Is it applying all patches?

Hi,
Please apply
"unable to allocate CUDA0 buffer" after Updating Ubuntu Packages - #168 by lavthyagarajan

This is critical PSIRT fix.