Gpu report: "nvgpu: 17000000.ga10b nvgpu_gr_intr_handle_sm_exception:390" then reboot

Hi
I use Orin AGX in my robot. It sometimes will report error in the title and reboot by itself. Detail log check below:


Dec  8 15:47:35 kernel: [ 2712.740516] nvgpu: 17000000.ga10b nvgpu_gr_intr_handle_sm_exception:390  [ERR]  could not pre-process sm error!
Dec  8 15:47:35 kernel: [ 2712.752355] nvgpu: 17000000.ga10b gr_intr_handle_exception_interrupts:759  [ERR]  set gr exception notifier
Dec  8 15:47:35 kernel: [ 2712.762720] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2712.770783] nvgpu: 17000000.ga10b     nvgpu_set_err_notifier_locked:149  [ERR]  error notifier set to 13 for ch 447
Dec  8 15:47:35 kernel: [ 2712.781790] __ga10b__ Channel Status - chip ga10b
Dec  8 15:47:35 kernel: [ 2712.781791] __ga10b__ ---------------------------
Dec  8 15:47:35 kernel: [ 2712.786664] __ga10b__ 432-ga10b, TSG: 2, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2712.791522] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2712.801198] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000201390020 GET: 000201390020 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000000000000 payload 0000000000000000 execute 00000000
Dec  8 15:47:35 kernel: [ 2712.806961] __ga10b__
Dec  8 15:47:35 kernel: [ 2712.827496] __ga10b__ 433-ga10b, TSG: 2, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2712.830115] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2712.830715] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2712.855515] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2712.855519] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000201290020 GET: 000201290020 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000000000000 payload 0000000000000000 execute 00000000
Dec  8 15:47:35 kernel: [ 2712.861269] __ga10b__
Dec  8 15:47:35 kernel: [ 2712.880196] __ga10b__ 434-ga10b, TSG: 2, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 phc2sys: [2712.896] eth0 sys offset        14 s2 freq  +32655 delay   4480
Dec  8 15:47:35 kernel: [ 2712.882738] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2712.892414] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000201190020 GET: 000201190020 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000000000000 payload 0000000000000000 execute 00000000
Dec  8 15:47:35 kernel: [ 2712.898182] __ga10b__
Dec  8 15:47:35 kernel: [ 2712.918472] __ga10b__ 435-ga10b, TSG: 2, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2712.921056] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2712.921627] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2712.947015] __ga10b__ channel status:  in use on_pbdma, pbdma_busy, acquire_fail busy
Dec  8 15:47:35 kernel: [ 2712.947020] __ga10b__ RAMFC: TOP: 000000000000 PUT: 0002010c38b8 GET: 0002010c387c FETCH: 000000000000 HEADER: 20140244 COUNT: 33330002 SEMAPHORE: addr 000201ba7df4 payload 0000000000304c4a execute 00081003
Dec  8 15:47:35 kernel: [ 2712.955083] __ga10b__
Dec  8 15:47:35 kernel: [ 2712.973960] __ga10b__ 436-ga10b, TSG: 1, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2712.976485] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2712.986146] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200f90020 GET: 000200f90020 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000000000000 payload 0000000000000000 execute 00000000
Dec  8 15:47:35 kernel: [ 2712.991880] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.012392] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.012405] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.030748] __ga10b__ 437-ga10b, TSG: 1, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.030751] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.040429] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200e90020 GET: 000200e90020 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000000000000 payload 0000000000000000 execute 00000000
Dec  8 15:47:35 kernel: [ 2713.046186] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.065057] __ga10b__ 438-ga10b, TSG: 1, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.067577] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.077236] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200d9c7b0 GET: 000200d9c7b0 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000201ba7dfc payload 0000000000304bae execute 00001003
Dec  8 15:47:35 kernel: [ 2713.084392] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.103511] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.103677] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.122101] __ga10b__ 439-ga10b, TSG: 1, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.122104] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.131783] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200c902cc GET: 000200c902cc FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 00020067fff0 payload 000000000000183e execute 00001003
Dec  8 15:47:35 kernel: [ 2713.137520] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.156393] __ga10b__ 440-ga10b, TSG: 0, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.158921] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.170411] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.170429] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 prediction: I1208 15:47:35.372900 37767 SysLog:0]  [AUTOOS]ClearProcess prediction
Dec  8 15:47:35 kernel: [ 2713.191624] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200b9cb68 GET: 000200b9cb68 FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004350000 payload 0000000000025630 execute 00000001
Dec  8 15:47:35 kernel: [ 2713.191627] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.210526] __ga10b__ 441-ga10b, TSG: 0, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.213059] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.222743] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200a9cb20 GET: 000200a9cb20 FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004340000 payload 0000000000025631 execute 00000001
Dec  8 15:47:35 kernel: [ 2713.228483] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.247361] __ga10b__ 442-ga10b, TSG: 0, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.249901] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.261267] __ga10b__ RAMFC: TOP: 000000000000 PUT: 00020099cb98 GET: 00020099cb98 FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004330000 payload 0000000000025631 execute 00000001
Dec  8 15:47:35 kernel: [ 2713.267218] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.267410] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.302220] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.302226] __ga10b__ 443-ga10b, TSG: 0, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.304753] __ga10b__ channel status:  in use pending, acquire_fail busy
Dec  8 15:47:35 kernel: [ 2713.314415] __ga10b__ RAMFC: TOP: 000000000000 PUT: 00020089cbdc GET: 00020089cbc0 FETCH: 000000000000 HEADER: 2010006c COUNT: 33330000 SEMAPHORE: addr 00020067ff30 payload 000000000003b57f execute 00081003
Dec  8 15:47:35 kernel: [ 2713.321334] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.341739] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.341791] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.360230] __ga10b__ 444-ga10b, TSG: 0, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.360232] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.369935] __ga10b__ RAMFC: TOP: 000000000000 PUT: 00020079cc3c GET: 00020079cc3c FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004300000 payload 0000000000025635 execute 00000001
Dec  8 15:47:35 kernel: [ 2713.375678] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.394570] __ga10b__ 445-ga10b, TSG: 0, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.397104] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.406770] __ga10b__ RAMFC: TOP: 000000000000 PUT: 00020069cb64 GET: 00020069cb64 FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004070000 payload 0000000000025635 execute 00000001
Dec  8 15:47:35 kernel: [ 2713.412516] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.433039] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.433336] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.451696] __ga10b__ 446-ga10b, TSG: 0, pid 36713, refs: 4, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.451698] __ga10b__ channel status:  in use on_eng, pending, eng_busy busy
Dec  8 15:47:35 kernel: [ 2713.461371] __ga10b__ RAMFC: TOP: 000000000000 PUT: 0002005aacb4 GET: 0002005aab64 FETCH: 000000000000 HEADER: 2010006c COUNT: 33330000 SEMAPHORE: addr 000201ba7ddc payload 0000000000304bfa execute 00080003
Dec  8 15:47:35 kernel: [ 2713.468638] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.487509] __ga10b__ 447-ga10b, TSG: 0, pid 36713, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.490037] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.499702] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200478ad0 GET: 000200478ad0 FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004010000 payload 00000000000255ac execute 00000001
Dec  8 15:47:35 kernel: [ 2713.505440] __ga10b__
Dec  8 15:47:35 kernel: [ 2713.526116] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.526677] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:35 kernel: [ 2713.544308] __ga10b__ 448-ga10b, TSG: 5, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:35 kernel: [ 2713.544311] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:35 kernel: [ 2713.553998] __ga10b__ RAMFC: TOP: 000000000000 PUT: 00020139ffcc GET: 00020139ffcc FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000201ba7144 payload 000000000000002e execute 00001003
Dec  8 15:47:36 kernel: [ 2713.559738] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.578629] __ga10b__ 449-ga10b, TSG: 5, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.581148] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.590808] __ga10b__ RAMFC: TOP: 000000000000 PUT: 00020129a134 GET: 00020129a134 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000201ba7084 payload 0000000000000036 execute 00001003
Dec  8 15:47:36 kernel: [ 2713.598246] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.617251] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.618179] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.636274] __ga10b__ 450-ga10b, TSG: 5, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.636276] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.645966] __ga10b__ RAMFC: TOP: 000000000000 PUT: 00020119fa98 GET: 00020119fa98 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000201ba70c4 payload 0000000000000036 execute 00001003
Dec  8 15:47:36 kernel: [ 2713.651720] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.670594] __ga10b__ 451-ga10b, TSG: 5, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.673126] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.684554] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.684918] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.706228] __ga10b__ RAMFC: TOP: 000000000000 PUT: 0002010912cc GET: 0002010912cc FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000201ba7104 payload 000000000000002e execute 00001003
Dec  8 15:47:36 kernel: [ 2713.706231] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.725115] __ga10b__ 452-ga10b, TSG: 4, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.727631] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.737299] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200f99a08 GET: 000200f99a08 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 00020067fff0 payload 00000000000d83ad execute 00001003
Dec  8 15:47:36 kernel: [ 2713.743034] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.761899] __ga10b__ 453-ga10b, TSG: 4, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.765799] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.775503] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.775718] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.797475] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200e97404 GET: 000200e97404 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 00020067fff0 payload 00000000000d7c1c execute 00001003
Dec  8 15:47:36 kernel: [ 2713.797477] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.816347] __ga10b__ 454-ga10b, TSG: 4, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.818873] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.828547] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200da2060 GET: 000200da2060 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 00020067ff80 payload 00000000000181f3 execute 00001003
Dec  8 15:47:36 kernel: [ 2713.834282] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.854516] __ga10b__ 455-ga10b, TSG: 4, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.857150] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.857284] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 phc2sys: [2713.896] eth0 sys offset        13 s2 freq  +32658 delay   4512
Dec  8 15:47:36 kernel: [ 2713.883074] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.883079] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200c98e80 GET: 000200c98e80 FETCH: 000000000000 HEADER: 21540300 COUNT: 00000000 SEMAPHORE: addr 000201ba7f9c payload 00000000000664be execute 00001003
Dec  8 15:47:36 kernel: [ 2713.888825] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.907737] __ga10b__ 456-ga10b, TSG: 3, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.910257] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.919924] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200bad91c GET: 000200bad91c FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004350000 payload 0000000000036b53 execute 00000001
Dec  8 15:47:36 kernel: [ 2713.925667] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.946428] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.946466] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2713.964991] __ga10b__ 457-ga10b, TSG: 3, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2713.964994] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2713.974663] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200b3eb8c GET: 000200b3eb8c FETCH: 000000000000 HEADER: 21511b0c COUNT: 00000000 SEMAPHORE: addr 002004340000 payload 0000000000036aa8 execute 00000001
Dec  8 15:47:36 kernel: [ 2713.980413] __ga10b__
Dec  8 15:47:36 kernel: [ 2713.999297] __ga10b__ 458-ga10b, TSG: 3, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2714.001828] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2714.011494] __ga10b__ RAMFC: TOP: 000000000000 PUT: 0002009bc450 GET: 0002009bc450 FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004330000 payload 000000000003477a execute 00000001
Dec  8 15:47:36 kernel: [ 2714.017228] __ga10b__
Dec  8 15:47:36 kernel: [ 2714.037800] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2714.037822] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2714.056076] __ga10b__ 459-ga10b, TSG: 3, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2714.056078] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2714.065756] __ga10b__ RAMFC: TOP: 000000000000 PUT: 00020094e55c GET: 00020094e55c FETCH: 000000000000 HEADER: 21511b0c COUNT: 00000000 SEMAPHORE: addr 002004320000 payload 0000000000036f2a execute 00000001
Dec  8 15:47:36 kernel: [ 2714.071493] __ga10b__
Dec  8 15:47:36 kernel: [ 2714.090363] __ga10b__ 460-ga10b, TSG: 3, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2714.092890] __ga10b__ channel status:  in use on_pbdma, on_eng, pbdma_busy, eng_busy busy
Dec  8 15:47:36 kernel: [ 2714.102560] __ga10b__ RAMFC: TOP: 000000000000 PUT: 0002007952b4 GET: 0002007952b4 FETCH: 000000000000 HEADER: 21511b0c COUNT: 00000000 SEMAPHORE: addr 002004300000 payload 0000000000037160 execute 00000001
Dec  8 15:47:36 kernel: [ 2714.112274] __ga10b__
Dec  8 15:47:36 kernel: [ 2714.131563] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2714.135088] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2714.143441] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Dec  8 15:47:36 kernel: [ 2714.158983] __ga10b__ 461-ga10b, TSG: 3, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2714.158986] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2714.168651] __ga10b__ RAMFC: TOP: 000000000000 PUT: 0002006903e8 GET: 0002006903e8 FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004070000 payload 0000000000036e00 execute 00000001
Dec  8 15:47:36 kernel: [ 2714.174394] __ga10b__
Dec  8 15:47:36 kernel: [ 2714.194592] __ga10b__ 462-ga10b, TSG: 3, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2714.197130] __ga10b__ channel status:  in use idle not busy
Dec  8 15:47:36 kernel: [ 2714.206824] __ga10b__ RAMFC: TOP: 000000000000 PUT: 000200570488 GET: 000200570488 FETCH: 000000000000 HEADER: 2150006c COUNT: 00000000 SEMAPHORE: addr 002004060000 payload 00000000000370bd execute 00000001
Dec  8 15:47:36 kernel: [ 2714.212571] __ga10b__
Dec  8 15:47:36 kernel: [ 2714.231460] __ga10b__ 463-ga10b, TSG: 3, pid 37194, refs: 2, deterministic: yes, domain name: (default)
Dec  8 15:47:36 kernel: [ 2714.233979] __ga10b_Mar 28 01:54:07 systemd-modules-load[381]: Inserted module 'nvmap'

My jetpack version: 5.1.1.
My question is what is this log stand for? Did GPU abornormal then make OS reboot?
Thanks for your help!
BR/Tim

Can anyone answer this question?
I also found tegra-capture-vi report: “uncorr_err: request timed out after 2500 ms”. And “tegra194-vi5 13e40000.host1x:vi1@14c00000: capture control message timed out”.
Does it means RTCPU also has some problem?
Thanks for your reply!

Hi,

The error is abnormal. Please flash your board to latest jetpack version and see if you still see such issue.

Hi Wayne
Thanks for your help. Could it be an hardware problem? Maybe power or EMC problem? Since this error only happened on one of my board.

Yes, I think it could be hardware related. But need to flash new BSP to confirm.

1 Like

Thanks!
Please let me confirm that, any bad cuda code can’t make this exception? Right?

If you have a certain method to trigger this error, try the same method on other module and see if they can reproduce or not. A software bug shall be stably reproduced.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.