GPU error when capturing and rendering images from multiple cameras

yangtzj · December 16, 2019, 2:17am

Hi,

We develop an application based on Tegra_multimedia_API to render multiple cameras videos simultaneously. The data flow path is as follows:

V4L2 CSI (DMABUF, UYVY) ----> NvBufferTransform (DMABUF, ARGB32, Blocklinear layout) ----> OSD (DMABUF, ARGB) --> EglRender

Image is captured via V4l2 API in DMA Buffer. And its format is UYVY.
To put OSD text on image, we use NvBufferTransform() API to convert UYVY to ARGB format.
Then nvosd_put_text() is called to add OSD on the image.
Finally, the OSD image is rendered by EglRender.

The application works well if only one process is running.

However, if multiple instances of the application are running simultaneously,
while each process captures and renders image from different, single camera,
e.g, process 1 captures and renders image from CAM1,
process 5 captures and renders image from CAM5,
we get the following dmesgs from time to time,
and the rendering FPS decreases while error msg appears.

22794.631451] ---- syncpts ----
[22794.631464] id 5 (disp_d) min 15886 max 15886 refs 1 (previous client : )
[22794.631469] id 6 (disp_e) min 975 max 975 refs 1 (previous client : )
[22794.631474] id 7 (disp_f) min 4 max 4 refs 1 (previous client : )
[22794.631479] id 8 (vblank1) min 1365780 max -6 refs 1 (previous client : )
[22794.631495] id 20 (gp10b_507) min 666354 max 666396 refs 1 (previous client : )
[22794.631499] id 21 (gp10b_506) min 64 max 64 refs 1 (previous client : )
[22794.631503] id 22 (gp10b_505) min 82202 max 82202 refs 1 (previous client : gp10b_505)
[22794.631508] id 23 (15340000.vic_robot_send_0) min 13302 max 13302 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631513] id 24 (15340000.vic_robot_send_0) min 13298 max 13298 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631517] id 25 (15340000.vic_robot_send_0) min 39119 max 39119 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631524] id 28 (15340000.vic_robot_send_0) min 54216 max 54216 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631529] id 29 (tegra-vi4) min 22427 max 22427 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631533] id 30 (tegra-vi4) min 1213 max 1214 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631538] id 31 (tegra-vi4) min 1213 max 1213 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631542] id 32 (tegra-vi4) min 1212 max 1213 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631547] id 33 (tegra-vi4) min 7677 max 7678 refs 1 (previous client : 15480000.nvdec_video_decode_0)
[22794.631551] id 34 (tegra-vi4) min 7670 max 7672 refs 1 (previous client : 15480000.nvdec_video_decode_0)
[22794.631555] id 35 (tegra-vi4) min 7670 max 7670 refs 1 (previous client : 15480000.nvdec_video_decode_0)
[22794.631560] id 36 (tegra-vi4) min 1211 max 1212 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631564] id 37 (gp10b_503) min 16146 max 16146 refs 1 (previous client : gp10b_503)
[22794.631568] id 38 (gp10b_502) min 9694 max 9698 refs 1 (previous client : 15340000.vic_video_decode_0)
[22794.631573] id 39 (gp10b_500) min 16142 max 16142 refs 1 (previous client : 15480000.nvdec_video_decode_0)
[22794.631577] id 40 (gp10b_501) min 35458 max 35458 refs 1 (previous client : gp10b_503)

[22794.632135] ---- channels ----
[22794.632148]
channel 2 - 15820000.se

[22794.632151] NvHost basic channel registers:
[22794.632155] CMDFIFO_STAT_0: 00002040
[22794.632158] CMDFIFO_RDATA_0: 840c1100
[22794.632162] CMDP_OFFSET_0: 00000000
[22794.632165] CMDP_CLASS_0: 00000000
[22794.632168] CHANNELSTAT_0: 00000000
[22794.632171] The CDMA sync queue is empty.

[22794.632176]
channel 3 - 15830000.se

[22794.632178] NvHost basic channel registers:
[22794.632180] CMDFIFO_STAT_0: 00002040
[22794.632183] CMDFIFO_RDATA_0: 9410d4d7
[22794.632187] CMDP_OFFSET_0: 00000000
[22794.632190] CMDP_CLASS_0: 00000000
[22794.632192] CHANNELSTAT_0: 00000000
[22794.632194] The CDMA sync queue is empty.

[22794.632199]
channel 4 - 15840000.se

[22794.632201] NvHost basic channel registers:
[22794.632204] CMDFIFO_STAT_0: 00002040
[22794.632206] CMDFIFO_RDATA_0: 614947c8
[22794.632210] CMDP_OFFSET_0: 00000000
[22794.632212] CMDP_CLASS_0: 00000000
[22794.632215] CHANNELSTAT_0: 00000000
[22794.632217] The CDMA sync queue is empty.

[22794.632224]
---- host general irq ----

[22794.632227] sync_intc0mask = 0x00000001
[22794.632230] sync_intmask = 0x50000003
[22794.632232]
---- host syncpt irq mask ----

[22794.632234]
---- host syncpt irq status ----

[22794.632237] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[22794.632240] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[22794.632243] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[22794.632246] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[22794.632249] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[22794.632252] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[22794.632255] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[22794.632258] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[22794.632260] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[22794.632263] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[22794.632266] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[22794.632269] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[22794.632272] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[22794.632275] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[22794.632278] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[22794.632281] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[22794.632284] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[22794.632287] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[22794.632292] gp10b pbdma 0:
[22794.632296] id: 8 (tsg), next_id: 8 (tsg) chan status: invalid
[22794.632314] PBDMA_PUT: 0000001e01521234 PBDMA_GET: 0000001e01521220 GP_PUT: 00000ebc GP_GET: 00000eb7 FETCH: 00000ebc HEADER: 20001b08
HDR: 200406c0 SHADOW0: 01521158 SHADOW1: 0000dc1e

[22794.632320] gp10b eng 0:
[22794.632323] id: 8 (tsg), next_id: 8 (tsg), ctx status: valid
[22794.632325] faulted
[22794.632327] busy

[22794.632331] gp10b eng 1:
[22794.632334] id: 4 (tsg), next_id: 4 (tsg), ctx status: valid

[22794.632508] 500-gp10b, pid 10207, refs 2:
[22794.632512] channel status: in use idle not busy
[22794.632519] RAMFC : TOP: 8000001f0000dda0 PUT: 0000001f0000dda0 GET: 0000001f0000dda0 FETCH: 0000001f0000dda0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002701 SEMAPHORE 0000001e 01460aa0 00000000 00000002

[22794.632525] 501-gp10b, pid 10209, refs 2:
[22794.632527] channel status: in use idle not busy
[22794.632533] RAMFC : TOP: 8000001f0000dd20 PUT: 0000001f0000dd20 GET: 0000001f0000dd20 FETCH: 0000001f0000dd20
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002801 SEMAPHORE 0000001e 01460aa0 00000000 00000002

[22794.632538] 502-gp10b, pid 10201, refs 6:
[22794.632541] channel status: in use on_eng_pending busy
[22794.632547] RAMFC : TOP: 8000001e01521220 PUT: 0000001e01521234 GET: 0000001e01521220 FETCH: 00000c1e01521244
HEADER: 20001b08 COUNT: 01110002
SYNCPOINT 00000000 00002601 SEMAPHORE 0000001e 01660aa0 00000000 00000002

[22794.632552] 503-gp10b, pid 10208, refs 2:
[22794.632555] channel status: in use idle not busy
[22794.632560] RAMFC : TOP: 8000001f00007f40 PUT: 0000001f00007f40 GET: 0000001f00007f40 FETCH: 0000001f00007f40
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002501 SEMAPHORE 0000001e 01420aa0 000018a8 00000002

[22794.632565] 505-gp10b, pid 7115, refs 2:
[22794.632568] channel status: in use idle not busy
[22794.632573] RAMFC : TOP: 8000001f0000f150 PUT: 0000001f0000f150 GET: 0000001f0000f150 FETCH: 0000001f0000f150
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001601 SEMAPHORE 0000001e 04370080 00000001 00001001

[22794.632578] 506-gp10b, pid 6588, refs 2:
[22794.632581] channel status: in use idle not busy
[22794.632586] RAMFC : TOP: 8000001f0000c300 PUT: 0000001f0000c300 GET: 0000001f0000c300 FETCH: 0000001f0000c300
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001501 SEMAPHORE 00000000 00000000 00000000 00000000

[22794.632591] 507-gp10b, pid 6588, refs 60:
[22794.632594] channel status: in use pending busy
[22794.632599] RAMFC : TOP: 8000001f00006f20 PUT: 0000001f00006f20 GET: 0000001f00006f20 FETCH: 0000001f00006f20
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001401 SEMAPHORE 0000000d fc001000 05e014e1 01100002

[22794.632604] 508-gp10b, pid 5412, refs 2:
[22794.632607] channel status: in use idle not busy
[22794.632612] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[22794.632617] 509-gp10b, pid 5412, refs 2:
[22794.632620] channel status: in use idle not busy
[22794.632625] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[22794.632630] 510-gp10b, pid 5412, refs 2:
[22794.632632] channel status: in use idle not busy
[22794.632637] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[22794.632642] 511-gp10b, pid 5412, refs 2:
[22794.632644] channel status: in use idle not busy
[22794.632649] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[22794.632666] nvgpu: 17000000.gp10b gk20a_fifo_handle_mmu_fault_locked:1722 [ERR] mmu fault on engine 0, engine subid 0 (gpc), client 4 (t1 1), addr 0x1e02050000, type 2 (pte), access_type 0x00000000,inst_ptr 0x1fffb3f000
[22794.658385] nvgpu: 17000000.gp10b gk20a_fifo_set_ctx_mmu_error_tsg:1543 [ERR] TSG 8 generated a mmu fault
[22794.668159] nvgpu: 17000000.gp10b gk20a_fifo_set_ctx_mmu_error_ch:1532 [ERR] channel 502 generated a mmu fault
[22794.678568] nvgpu: 17000000.gp10b nvgpu_set_error_notifier_locked:135 [ERR] error notifier set to 31 for ch 502
[22794.689283] nvgpu: 17000000.gp10b fifo_error_isr:2497 [ERR] channel reset initiated from fifo_error_isr; intr=0x10000000

[22799.117457] ---- mlocks ----

What does this error mean and how to fix it?

DaneLLL · December 16, 2019, 7:45am

Hi,
Please share your release version(r32.2.1 or r28.2.1). And we would need your help to share a patch that we can apply to 12_camera_v4l2_cuda sample to reproduce the issue.

yangtzj · December 18, 2019, 9:36am

Hi DaneLLL,

The release version is R32.2, and the test platform is TX2.

And to reproduce the problem, we develope a new test program,
which is fairly simple and basically works as follows:

Create a HW buffer (Layout_Pitch, UYVY) as data source;
Create a HW buffer (Layout_BlockLinear, NV12） as NvBufferTransform destionation;
Create an ELG render and set FPS to 60;
Loop infinitely:
4.1 Change source buffer data (color);
4.2 Convert source buffer to dst buffer by calling NvBufferTransform();
4.3 Render the converted buffer by EGL and print profiling data

So no V4L2 image capturing is involved.
Only buffer transforming and EGL rendering.

When running 9 instances of this application,
we get the same error dmesg occurs from time to time, plus some kernel warning.

To launch 9 instances of the test application, just type:
<application_name> 0
<application_name> 1
<application_name> 2
<application_name> 3
<application_name> 4
<application_name> 5
<application_name> 6
<application_name> 7
<application_name> 8

Attached please find the test program (main.cpp), Makefile(Makefile.txt), launch (shell) script (test.sh.txt) and error log (err.txt).

==================================
err.txt:

[ 529.839544] CPU: 1 PID: 5755 Comm: nvgpu_channel_p Not tainted 4.9.140-tegra #187
[ 529.839547] Hardware name: quill (DT)
[ 529.839571] task: ffffffc1ebfb5400 task.stack: ffffffc1d1998000
[ 529.839577] PC is at dma_buf_put+0x34/0x48
[ 529.840025] LR is at nvgpu_vm_unmap_system+0x40/0x50 [nvgpu]
[ 529.840029] pc : [] lr : [] pstate: 20400045
[ 529.840031] sp : ffffffc1d199bc10
[ 529.840034] x29: ffffffc1d199bc10 x28: ffffff8012875200
[ 529.840039] x27: ffffffc1cbd92000 x26: ffffff8012875720
[ 529.840044] x25: 0000000000000038 x24: ffffffc1d199bcce
[ 529.840048] x23: ffffffc1cbd92058 x22: ffffffc1d199bcce
[ 529.840052] x21: ffffffc1e21f0000 x20: ffffffc1cbd92000
[ 529.840056] x19: 0000000000000000 x18: 0000000000000001
[ 529.840060] x17: 0000000000000000 x16: 0000000000000005
[ 529.840064] x15: 0000000000000016 x14: 000000000000002d
[ 529.840068] x13: 000000000000002d x12: ffffff8001064bf0
[ 529.840072] x11: 000000000000000b x10: 0101010101010101
[ 529.840077] x9 : fffffffffffffffa x8 : 7f7f7f7f7f7f7f7f
[ 529.840081] x7 : fefefeff646c606d x6 : 00170401e9e1acf4
[ 529.840085] x5 : 742c616901041700 x4 : 0000000000000001
[ 529.840089] x3 : 0000000000000000 x2 : 0000000000000000
[ 529.840093] x1 : fffffffffffff000 x0 : ffffff8000f773a8

[ 529.840099] —[ end trace 792283d54ecb7390 ]—
[ 529.844712] Call trace:
[ 529.844736] [] dma_buf_put+0x34/0x48
[ 529.845180] [] nvgpu_vm_unmap_system+0x40/0x50 [nvgpu]
[ 529.845374] [] nvgpu_pd_free+0x2f8/0x318 [nvgpu]
[ 529.845568] [] nvgpu_vm_put_buffers+0xb0/0xf8 [nvgpu]
[ 529.845765] [] gk20a_channel_clean_up_jobs+0x2bc/0x7a0 [nvgpu]
[ 529.845998] [] gk20a_channel_clean_up_jobs+0x484/0x7a0 [nvgpu]
[ 529.846208] [] nvgpu_thread_proxy+0x20/0x30 [nvgpu]
[ 529.846217] [] kthread+0xec/0xf0
[ 529.846223] [] ret_from_fork+0x10/0x40
[ 1096.964865] nvgpu: 17000000.gp10b railgate_enable_store:343 [INFO] railgate is disabled.
[ 1847.240766] ---- mlocks ----

[ 1847.240797] ---- syncpts ----
[ 1847.240811] id 5 (disp_d) min 96049 max 96050 refs 1 (previous client : )
[ 1847.240815] id 6 (disp_e) min 2 max 2 refs 1 (previous client : )
[ 1847.240820] id 7 (disp_f) min 2 max 2 refs 1 (previous client : )
[ 1847.240825] id 8 (vblank1) min 109782 max -4 refs 1 (previous client : )
[ 1847.240841] id 20 (gp10b_507) min 392598 max 392598 refs 1 (previous client : )
[ 1847.240845] id 21 (gp10b_506) min 46 max 46 refs 1 (previous client : )
[ 1847.240848] id 22 (gp10b_505) min 493436 max 493438 refs 1 (previous client : gp10b_505)
[ 1847.240852] id 23 (15340000.vic_robot_send_0) min 147672 max 147672 refs 1 (previous client : gp10b_504)
[ 1847.240855] id 24 (15340000.vic_robot_send_0) min 147651 max 147651 refs 1 (previous client : )
[ 1847.240858] id 25 (15340000.vic_robot_send_0) min 147660 max 147660 refs 1 (previous client : )
[ 1847.240863] id 28 (15340000.vic_robot_send_0) min 147654 max 147654 refs 1 (previous client : )
[ 1847.240866] id 29 (15340000.vic_robot_send_0) min 147654 max 147654 refs 1 (previous client : )
[ 1847.240869] id 30 (15340000.vic_robot_send_0) min 147660 max 147660 refs 1 (previous client : )
[ 1847.240872] id 31 (15340000.vic_robot_send_0) min 147657 max 147657 refs 1 (previous client : )
[ 1847.240875] id 32 (15340000.vic_robot_send_0) min 147657 max 147657 refs 1 (previous client : )
[ 1847.240878] id 33 (15340000.vic_robot_send_0) min 147651 max 147651 refs 1 (previous client : )
[ 1847.240881] id 34 (gp10b_504) min 393766 max 393766 refs 1 (previous client : )
[ 1847.240884] id 35 (gp10b_503) min 393766 max 393766 refs 1 (previous client : )
[ 1847.240887] id 36 (gp10b_502) min 393766 max 393766 refs 1 (previous client : )
[ 1847.240890] id 37 (gp10b_501) min 393760 max 393760 refs 1 (previous client : )
[ 1847.240893] id 38 (gp10b_500) min 393754 max 393758 refs 1 (previous client : )
[ 1847.240896] id 39 (gp10b_499) min 393752 max 393752 refs 1 (previous client : )
[ 1847.240899] id 40 (gp10b_498) min 393750 max 393750 refs 1 (previous client : )
[ 1847.240902] id 41 (gp10b_497) min 393740 max 393742 refs 1 (previous client : )
[ 1847.240905] id 42 (gp10b_496) min 393742 max 393742 refs 1 (previous client : )

[ 1847.241429] ---- channels ----
[ 1847.241440]
channel 2 - 15820000.se

[ 1847.241441] NvHost basic channel registers:
[ 1847.241444] CMDFIFO_STAT_0: 00002040
[ 1847.241447] CMDFIFO_RDATA_0: 640c1f12
[ 1847.241450] CMDP_OFFSET_0: 00000000
[ 1847.241452] CMDP_CLASS_0: 00000000
[ 1847.241454] CHANNELSTAT_0: 00000000
[ 1847.241456] The CDMA sync queue is empty.

[ 1847.241459]
channel 3 - 15830000.se

[ 1847.241460] NvHost basic channel registers:
[ 1847.241462] CMDFIFO_STAT_0: 00002040
[ 1847.241464] CMDFIFO_RDATA_0: d710fc56
[ 1847.241467] CMDP_OFFSET_0: 00000000
[ 1847.241469] CMDP_CLASS_0: 00000000
[ 1847.241471] CHANNELSTAT_0: 00000000
[ 1847.241472] The CDMA sync queue is empty.

[ 1847.241476]
channel 4 - 15840000.se

[ 1847.241477] NvHost basic channel registers:
[ 1847.241479] CMDFIFO_STAT_0: 00002040
[ 1847.241481] CMDFIFO_RDATA_0: 6b410728
[ 1847.241484] CMDP_OFFSET_0: 00000000
[ 1847.241485] CMDP_CLASS_0: 00000000
[ 1847.241487] CHANNELSTAT_0: 00000000
[ 1847.241488] The CDMA sync queue is empty.

[ 1847.241494]
---- host general irq ----

[ 1847.241496] sync_intc0mask = 0x00000001
[ 1847.241498] sync_intmask = 0x50000003
[ 1847.241499]
---- host syncpt irq mask ----

[ 1847.241501]
---- host syncpt irq status ----

[ 1847.241503] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 1847.241505] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 1847.241508] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 1847.241510] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 1847.241512] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 1847.241514] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 1847.241516] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 1847.241518] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 1847.241521] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 1847.241523] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 1847.241525] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 1847.241527] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 1847.241529] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 1847.241531] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 1847.241534] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 1847.241536] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 1847.241538] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 1847.241540] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[ 1847.241545] gp10b pbdma 0:
[ 1847.241548] id: 14 (tsg), next_id: 14 (tsg) chan status: invalid
[ 1847.241563] PBDMA_PUT: 0000001e01684800 PBDMA_GET: 0000001e016847ec GP_PUT: 0000062e GP_GET: 0000062d FETCH: 0000062e HEADER: 20001b08
HDR: 200406c0 SHADOW0: 01684724 SHADOW1: 0000dc1e

[ 1847.241567] gp10b eng 0:
[ 1847.241569] id: 14 (tsg), next_id: 14 (tsg), ctx status: valid
[ 1847.241570] faulted
[ 1847.241572] busy

[ 1847.241575] gp10b eng 1:
[ 1847.241577] id: 15 (tsg), next_id: 15 (tsg), ctx status: invalid

[ 1847.241794] 496-gp10b, pid 7974, refs 2:
[ 1847.241797] channel status: in use idle not busy
[ 1847.241801] RAMFC : TOP: 8000001f0000a0e0 PUT: 0000001f0000a0e0 GET: 0000001f0000a0e0 FETCH: 0000001f0000a0e0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002a01 SEMAPHORE 0000001e 01160aa0 00000000 00000002

[ 1847.241806] 497-gp10b, pid 7973, refs 4:
[ 1847.241807] channel status: in use on_eng_pending busy
[ 1847.241811] RAMFC : TOP: 8000001e016847ec PUT: 0000001e01684800 GET: 0000001e016847ec FETCH: 0000001f0000a0e0
HEADER: 20001b08 COUNT: 01110002
SYNCPOINT 00000000 00002901 SEMAPHORE 0000001e 01160aa0 00000000 00000002

[ 1847.241815] 498-gp10b, pid 7972, refs 2:
[ 1847.241817] channel status: in use idle not busy
[ 1847.241820] RAMFC : TOP: 8000001f0000a160 PUT: 0000001f0000a160 GET: 0000001f0000a160 FETCH: 0000001f0000a160
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002801 SEMAPHORE 0000001e 01160aa0 00000000 00000002

[ 1847.241824] 499-gp10b, pid 7971, refs 2:
[ 1847.241825] channel status: in use idle not busy
[ 1847.241828] RAMFC : TOP: 8000001f0000a180 PUT: 0000001f0000a180 GET: 0000001f0000a180 FETCH: 0000001f0000a180
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002701 SEMAPHORE 0000001e 01160aa0 00000000 00000002

[ 1847.241832] 500-gp10b, pid 7970, refs 6:
[ 1847.241833] channel status: in use pending busy
[ 1847.241837] RAMFC : TOP: 8000001f0000a1a0 PUT: 0000001f0000a1a0 GET: 0000001f0000a1a0 FETCH: 0000001f0000a1a0
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002601 SEMAPHORE 0000001e 01160aa0 00000000 00000002

[ 1847.241840] 501-gp10b, pid 7969, refs 2:
[ 1847.241841] channel status: in use idle not busy
[ 1847.241845] RAMFC : TOP: 8000001f0000a200 PUT: 0000001f0000a200 GET: 0000001f0000a200 FETCH: 0000001f0000a200
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002501 SEMAPHORE 0000001e 01160aa0 00000000 00000002

[ 1847.241848] 502-gp10b, pid 7968, refs 2:
[ 1847.241849] channel status: in use idle not busy
[ 1847.241853] RAMFC : TOP: 8000001f0000a260 PUT: 0000001f0000a260 GET: 0000001f0000a260 FETCH: 0000001f0000a260
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002401 SEMAPHORE 0000001e 01160aa0 00000000 00000002

[ 1847.241856] 503-gp10b, pid 7966, refs 2:
[ 1847.241858] channel status: in use idle not busy
[ 1847.241861] RAMFC : TOP: 8000001f0000a260 PUT: 0000001f0000a260 GET: 0000001f0000a260 FETCH: 0000001f0000a260
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002301 SEMAPHORE 0000001e 01460aa0 00000000 00000002

[ 1847.241864] 504-gp10b, pid 7967, refs 2:
[ 1847.241866] channel status: in use idle not busy
[ 1847.241869] RAMFC : TOP: 8000001f0000a260 PUT: 0000001f0000a260 GET: 0000001f0000a260 FETCH: 0000001f0000a260
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00002201 SEMAPHORE 0000001e 01460aa0 00000000 00000002

[ 1847.241872] 505-gp10b, pid 7068, refs 4:
[ 1847.241874] channel status: in use pending_acquire busy
[ 1847.241877] RAMFC : TOP: 8000001e03560010 PUT: 0000001e03560014 GET: 0000001e03560010 FETCH: 00000c1e03560010
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00017732 00000510 SEMAPHORE 0000001e 043700d0 00000001 00081001
[ 1847.241880] Waiting on syncpt 5 (disp_d) val 96050

[ 1847.241883] 506-gp10b, pid 6260, refs 2:
[ 1847.241885] channel status: in use idle not busy
[ 1847.241888] RAMFC : TOP: 8000001f0000c228 PUT: 0000001f0000c228 GET: 0000001f0000c228 FETCH: 0000001f0000c228
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001501 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1847.241892] 507-gp10b, pid 6260, refs 2:
[ 1847.241894] channel status: in use idle not busy
[ 1847.241897] RAMFC : TOP: 8000001f00005960 PUT: 0000001f00005960 GET: 0000001f00005960 FETCH: 0000001f00005960
HEADER: 60400000 COUNT: 80000000
SYNCPOINT 00000000 00001401 SEMAPHORE 0000000d fc001000 03282438 01100002

[ 1847.241900] 508-gp10b, pid 5705, refs 2:
[ 1847.241902] channel status: in use idle not busy
[ 1847.241905] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1847.241909] 509-gp10b, pid 5705, refs 2:
[ 1847.241910] channel status: in use idle not busy
[ 1847.241913] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1847.241916] 510-gp10b, pid 5705, refs 2:
[ 1847.241918] channel status: in use idle not busy
[ 1847.241921] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1847.241924] 511-gp10b, pid 5705, refs 2:
[ 1847.241926] channel status: in use idle not busy
[ 1847.241928] RAMFC : TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
HEADER: 60400000 COUNT: 00000000
SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

[ 1847.241942] nvgpu: 17000000.gp10b gk20a_fifo_handle_mmu_fault_locked:1722 [ERR] mmu fault on engine 0, engine subid 0 (gpc), client 4 (t1 1), addr 0x1e02400000, type 2 (pte), access_type 0x00000000,inst_ptr 0x1ff6d3d000
[ 1847.266648] nvgpu: 17000000.gp10b gk20a_fifo_set_ctx_mmu_error_tsg:1543 [ERR] TSG 14 generated a mmu fault
[ 1847.276499] nvgpu: 17000000.gp10b gk20a_fifo_set_ctx_mmu_error_ch:1532 [ERR] channel 497 generated a mmu fault
[ 1847.286762] nvgpu: 17000000.gp10b nvgpu_set_error_notifier_locked:135 [ERR] error notifier set to 31 for ch 497
[ 1847.297220] nvgpu: 17000000.gp10b fifo_error_isr:2497 [ERR] channel reset initiated from fifo_error_isr; intr=0x10000000
[ 1847.406215] ---- mlocks ----
[ 1847.406237] 17: locked by channel 1
Makefile.txt (2.28 KB)
err.txt (13.5 KB)

main .cpp (2.38 KB)

test .sh.txt (144 Bytes)

carolyuu · December 20, 2019, 4:43am

Hi yangtzj,

We can’t reproduce your issue with JetPack-4.2.3 (r32.2.3) and JetPack-4.3 (r32.3.1) on TX2.
Please check what’s JetPack version are you testing.
$ head -1 /etc/nv_tegra_release.

DaneLLL · December 20, 2019, 9:32am

Hi,
We would also suggest you upgrade to r32.2.3.

nadeemm · October 18, 2021, 6:19pm