Jetson AGX Xavier GPU MMU fault

Robotics & Edge Computing Jetson Systems Jetson AGX Xavier
1

Hi,
We user Jetson AGX Xavier . The l4T version is R35.6.0 (JetPack 5. It includes Linux Kernel 5.10, an Ubuntu 20.04 based root file system.)

We killed the Gnome desktop environment and used /usr/bin/Xorg -ac -noreset -nolisten tcp & to start Xorg.
We wrote a Qt GUI program using Qt5OpenGL. We use xrandr to set the output of the HDMI before the program starts the Qt program.

Occasionally, the GPU reports the following error:

Jul 22 22:44:07 nvidia kernel: [  428.259814] nvgpu: 17000000.gv11b           nvgpu_report_err_to_sdl:66   [ERR]  Failed to report an error: hw_unit_id = 0x9, err_id=0x8, ss_err_id = 0x89
Jul 22 22:44:07 nvidia kernel: [  428.260166] nvgpu: 17000000.gv11b gv11b_mm_mmu_fault_handle_buf_valid_entry:525  [ERR]  page fault error: err_type = 0x8, fault_status = 0x200
Jul 22 22:44:07 nvidia kernel: [  428.260460] nvgpu: 17000000.gv11b      gv11b_fb_mmu_fault_info_dump:294  [ERR]  [MMU FAULT] mmu engine id:  64, ch id:  508, fault addr: 0x1fea0f0000, fault addr aperture: 0, fault type: invalid pde, access type: virt write, 
Jul 22 22:44:07 nvidia kernel: [  428.260957] nvgpu: 17000000.gv11b      gv11b_fb_mmu_fault_info_dump:307  [ERR]  [MMU FAULT] protected mode: 0, client type: gpc, client id:  prop 0, gpc id if client type is gpc: 0, 
Jul 22 22:44:07 nvidia kernel: [  428.261336] nvgpu: 17000000.gv11b                nvgpu_rc_mmu_fault:352  [ERR]  mmu fault id=3 id_type=1 act_eng_bitmask=00000001
Jul 22 22:44:07 nvidia kernel: [  428.261610] nvgpu: 17000000.gv11b       nvgpu_tsg_set_ctx_mmu_error:648  [ERR]  TSG 3 generated a mmu fault
Jul 22 22:44:07 nvidia kernel: [  428.261923] nvgpu: 17000000.gv11b     nvgpu_set_err_notifier_locked:149  [ERR]  error notifier set to 31 for ch 508
Jul 22 22:44:07 nvidia kernel: [  428.264151] __gv11b__ Channel Status - chip gv11b
Jul 22 22:44:07 nvidia kernel: [  428.264156] __gv11b__ ---------------------------
Jul 22 22:44:07 nvidia kernel: [  428.268453] __gv11b__ 507-gv11b, TSG: 4, pid 2572, refs: 2, deterministic: no, domain name: (default)
Jul 22 22:44:07 nvidia kernel: [  428.273204] __gv11b__ channel status:  in use idle not busy
Jul 22 22:44:07 nvidia kernel: [  428.282502] __gv11b__ RAMFC: TOP: 000000000000 PUT: 000000000000 GET: 000000000000 FETCH: 000000000000 HEADER: 20400000 COUNT: 00000000 SEMAPHORE: addr 000000000000 payload 0000000000000000 execute 00000000
Jul 22 22:44:07 nvidia kernel: [  428.288041] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.306528] __gv11b__ 508-gv11b, TSG: 3, pid 2801, refs: 4, deterministic: no, domain name: (default)
Jul 22 22:44:07 nvidia kernel: [  428.309393] __gv11b__ channel status:  in use on_eng_pending busy
Jul 22 22:44:07 nvidia kernel: [  428.319082] __gv11b__ RAMFC: TOP: 8000001ffa9ec2a8 PUT: 001ffa9ec408 GET: 001ffa9ec2a8 FETCH: 002000412008 HEADER: 20030230 COUNT: 01110004 SEMAPHORE: addr 001ffd17d960 payload 000000000000001f execute 02000001
Jul 22 22:44:07 nvidia kernel: [  428.324609] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.345272] __gv11b__ 509-gv11b, TSG: 2, pid 2572, refs: 3, deterministic: no, domain name: (default)
Jul 22 22:44:07 nvidia kernel: [  428.346300] __gv11b__ channel status:  in use on_pbdma_and_eng busy
Jul 22 22:44:07 nvidia kernel: [  428.355687] __gv11b__ RAMFC: TOP: 8000002000411888 PUT: 002000411888 GET: 002000411888 FETCH: 002000411888 HEADER: 60400000 COUNT: 80000000 SEMAPHORE: addr 002000408000 payload 0000000000000000 execute 00100001
Jul 22 22:44:07 nvidia kernel: [  428.361878] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.380527] __gv11b__ 510-gv11b, TSG: 1, pid 572, refs: 2, deterministic: no, domain name: (default)
Jul 22 22:44:07 nvidia kernel: [  428.382672] __gv11b__ channel status:  in use idle not busy
Jul 22 22:44:07 nvidia kernel: [  428.391867] __gv11b__ RAMFC: TOP: 8000002000429118 PUT: 002000429118 GET: 002000429118 FETCH: 002000429118 HEADER: 60400000 COUNT: 80000000 SEMAPHORE: addr 002000428000 payload 0000000000000000 execute 00000001
Jul 22 22:44:07 nvidia kernel: [  428.397601] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.416168] __gv11b__ 511-gv11b, TSG: 0, pid 572, refs: 2, deterministic: no, domain name: (default)
Jul 22 22:44:07 nvidia kernel: [  428.418830] __gv11b__ channel status:  in use idle not busy
Jul 22 22:44:07 nvidia kernel: [  428.427729] __gv11b__ RAMFC: TOP: 80000020004544b0 PUT: 0020004544b0 GET: 0020004544b0 FETCH: 0020004544b0 HEADER: 60400000 COUNT: 80000000 SEMAPHORE: addr 002000420000 payload 0000000000000000 execute 00100001
Jul 22 22:44:07 nvidia kernel: [  428.433669] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.452495] __gv11b__ PBDMA Status - chip gv11b
Jul 22 22:44:07 nvidia kernel: [  428.454613] __gv11b__ -------------------------
Jul 22 22:44:07 nvidia kernel: [  428.459353] __gv11b__ pbdma 0:
Jul 22 22:44:07 nvidia kernel: [  428.463898] __gv11b__   id: 2 - [tsg]     next_id: - -1 [channel] | status: valid
Jul 22 22:44:07 nvidia kernel: [  428.467015] __gv11b__   PBDMA_PUT 0000001ffcf88570 PBDMA_GET 0000001ffcf87fb0
Jul 22 22:44:07 nvidia kernel: [  428.474710] __gv11b__   GP_PUT    0000087a  GP_GET  00000879  FETCH   0000087a HEADER 20000df8
Jul 22 22:44:07 nvidia kernel: [  428.482104] __gv11b__   HDR       2002037e  SHADOW0 fcf87e74  SHADOW1 0006fc1f
Jul 22 22:44:07 nvidia kernel: [  428.490710] __gv11b__ pbdma 1:
Jul 22 22:44:07 nvidia kernel: [  428.498073] __gv11b__   id: -1 - [channel] next_id: - -1 [channel] | status: invalid
Jul 22 22:44:07 nvidia kernel: [  428.500997] __gv11b__   PBDMA_PUT 0000000008320400 PBDMA_GET 0000006812065838
Jul 22 22:44:07 nvidia kernel: [  428.508944] __gv11b__   GP_PUT    00000000  GP_GET  0a410416  FETCH   00000000 HEADER 01540400
Jul 22 22:44:07 nvidia kernel: [  428.516317] __gv11b__   HDR       18714882  SHADOW0 6b410010  SHADOW1 000a0000
Jul 22 22:44:07 nvidia kernel: [  428.524635] __gv11b__ pbdma 2:
Jul 22 22:44:07 nvidia kernel: [  428.531456] __gv11b__   id: 4 - [tsg]     next_id: - -1 [channel] | status: valid
Jul 22 22:44:07 nvidia kernel: [  428.534688] __gv11b__   PBDMA_PUT 00000020005a0168 PBDMA_GET 00000020005a0168
Jul 22 22:44:07 nvidia kernel: [  428.541978] __gv11b__   GP_PUT    00000022  GP_GET  00000022  FETCH   00000022 HEADER 60400000
Jul 22 22:44:07 nvidia kernel: [  428.549362] __gv11b__   HDR       00000000  SHADOW0 005a0140  SHADOW1 00002820
Jul 22 22:44:07 nvidia kernel: [  428.557779] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.565280] __gv11b__ gv11b eng 0: 
Jul 22 22:44:07 nvidia kernel: [  428.567768] __gv11b__ id: 3 (tsg), next_id: 2 (tsg), ctx status: switch 
Jul 22 22:44:07 nvidia kernel: [  428.571591] __gv11b__ busy 
Jul 22 22:44:07 nvidia kernel: [  428.578006] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.580850] __gv11b__ gv11b eng 1: 
Jul 22 22:44:07 nvidia kernel: [  428.583524] __gv11b__ id: 4 (tsg), next_id: -1 (channel), ctx status: valid 
Jul 22 22:44:07 nvidia kernel: [  428.587254] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.594168] __gv11b__ gv11b eng 2: 
Jul 22 22:44:07 nvidia kernel: [  428.596923] __gv11b__ id: -1 (channel), next_id: -1 (channel), ctx status: invalid 
Jul 22 22:44:07 nvidia kernel: [  428.600290] __gv11b__  
Jul 22 22:44:07 nvidia kernel: [  428.608321] __gv11b__ gv11b eng 3: 
Jul 22 22:44:07 nvidia kernel: [  428.610890] __gv11b__ id: -1 (channel), next_id: -1 (channel), ctx status: invalid 
Jul 22 22:44:07 nvidia kernel: [  428.614298] __gv11b__  
Jul 22 22:44:11 nvidia kernel: [  428.622243] __gv11b__

A detailed printout is attached below:
GPU error.txt (1.1 MB)

The Qt GUI seems to be rendering abnormally when the GPU reports an error. Dragging a Qt option box results in a long drag shadow that doesn’t go away.

image
image726×295 95.3 KB

The xrandr command to set the screen output is as follows:

  xrandr --output HDMI-1 --mode "1920x1200"
    xrandr --output HDMI-0 --mode "1920x1200"
    xrandr --output HDMI-2 --auto --primary --output HDMI-1 --auto --right-of HDMI-2

Our devkit is broken, so we can’t verify it with devkit .
Could you please tell me what might be the possible reasons for the following error? Without a development kit, how should we go about troubleshooting and locating the issue?

Jul 22 22:44:07 nvidia kernel: [  428.259814] nvgpu: 17000000.gv11b           nvgpu_report_err_to_sdl:66   [ERR]  Failed to report an error: hw_unit_id = 0x9, err_id=0x8, ss_err_id = 0x89
Jul 22 22:44:07 nvidia kernel: [  428.260166] nvgpu: 17000000.gv11b gv11b_mm_mmu_fault_handle_buf_valid_entry:525  [ERR]  page fault error: err_type = 0x8, fault_status = 0x200
Jul 22 22:44:07 nvidia kernel: [  428.260460] nvgpu: 17000000.gv11b      gv11b_fb_mmu_fault_info_dump:294  [ERR]  [MMU FAULT] mmu engine id:  64, ch id:  508, fault addr: 0x1fea0f0000, fault addr aperture: 0, fault type: invalid pde, access type: virt write, 
Jul 22 22:44:07 nvidia kernel: [  428.260957] nvgpu: 17000000.gv11b      gv11b_fb_mmu_fault_info_dump:307  [ERR]  [MMU FAULT] protected mode: 0, client type: gpc, client id:  prop 0, gpc id if client type is gpc: 0, 
Jul 22 22:44:07 nvidia kernel: [  428.261336] nvgpu: 17000000.gv11b                nvgpu_rc_mmu_fault:352  [ERR]  mmu fault id=3 id_type=1 act_eng_bitmask=00000001
Jul 22 22:44:07 nvidia kernel: [  428.261610] nvgpu: 17000000.gv11b       nvgpu_tsg_set_ctx_mmu_error:648  [ERR]  TSG 3 generated a mmu fault
Jul 22 22:44:07 nvidia kernel: [  428.261923] nvgpu: 17000000.gv11b     nvgpu_set_err_notifier_locked:149  [ERR]  error notifier set to 31 for ch 508
Jul 22 22:44:07 nvidia kernel: [  428.264151] __gv11b__ Channel Status - chip gv11b
Jul 22 22:44:07 nvidia kernel: [  428.264156] __gv11b__ ---------------------------
Jul 22 22:44:07 nvidia kernel: [  428.268453] __gv11b__ 507-gv11b, TSG: 4, pid 2572, refs: 2, deterministic: no, domain name: (default)
Jul 22 22:44:07 nvidia kernel: [  428.273204] __gv11b__ channel status:  in use idle not busy
Jul 22 22:44:07 nvidia kernel: [  428.282502] __gv11b__ RAMFC: TOP: 000000000000 PUT: 000000000000 GET: 000000000000 FETCH: 000000000000 HEADER: 20400000 COUNT: 00000000 SEMAPHORE: addr 000000000000 payload 0000000000000000 execute 00000000
Jul 22 22:44:07 nvidia kernel: [  428.288041] __gv11b__
2

Are you able to try with other Xavier module to see if this is HW problem?

5

yes, it seems devkit carrier board was broken.
So, we can’t verify it with devkit .

Could you please tell me what might be the possible reasons for the following error?

6

I mean other Xavier module, not devkit carier board.

7

Yes, we can reproduce it on other modules.
We have reproduced this issue on at least six other modules.

8

Hi,
Please try to reproduce it on AGX Xavier developer kit. If you can reproduce it on developer kit, please share us the steps. We will set up and check.

9

Hi,
Our devkit carrier board was broken.
So, we can’t verify it with devkit .

Could you please tell me what might be the possible reasons for the following error?

10

Hi,
From the log it seems to be memory corruption in application level. It may be specific to using Qt5OpenGL. For further analysis we would need to replicate the error and check.

11

Hello, have you been able to reproduce this issue?

If you are unable to reproduce it, we can purchase another Jetson AGX Xavier devkit and try to reproduce it on that devkit.

12

Hi,
We would like to eliminate QT frameworks. Please check if you can reproduce it by running either sample in:

/usr/src/nvidia/graphics_demos