Agx orin gpu无法工作

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
, ,
1

agx orin 32g
客户自定义板子
软件版本：jp5.1.2

设备运行过程中偶尔出现如下错误，遇到系统很卡，网络也不通，无法正常工作，请问一下，这是什么原因导致的

Jul  9 11:55:38 ubuntu kernel: [251005.240803] nvgpu: 17000000.ga10b            nvgpu_finalize_poweron:1010 [ERR]  Failed initialization for: g->ops.gr.gr_init_support
Jul  9 11:55:38 ubuntu kernel: [251005.275367] nvgpu: 17000000.ga10b nvgpu_gpu_set_deterministic_ch_railgate:1858 [WRN]  cannot busy to restore deterministic ch
Jul  9 11:55:38 ubuntu kernel: [251005.275751] CPU:0, Error: cbb-fabric@0x13a00000, irq=34
Jul  9 11:55:38 ubuntu kernel: [251005.275890] **************************************
Jul  9 11:55:38 ubuntu kernel: [251005.276023] CPU:0, Error:cbb-fabric, Errmon:2
Jul  9 11:55:38 ubuntu kernel: [251005.276136] 	  Error Code		: PWRDOWN_ERR
Jul  9 11:55:38 ubuntu kernel: [251005.278287] 
Jul  9 11:55:38 ubuntu kernel: [251005.279856] 	  Error Code		: PWRDOWN_ERR
Jul  9 11:55:38 ubuntu kernel: [251005.284056] 	  MASTER_ID		: CCPLEX
Jul  9 11:55:38 ubuntu kernel: [251005.287729] 	  Address		: 0x17bb0090
Jul  9 11:55:38 ubuntu kernel: [251005.291404] 	  Cache			: 0x0 -- Device Non-Bufferable
Jul  9 11:55:38 ubuntu kernel: [251005.296481] 	  Protection		: 0x2 -- Unprivileged, Non-Secure, Data Access
Jul  9 11:55:38 ubuntu kernel: [251005.303481] 	  Access_Type		: Write
Jul  9 11:55:38 ubuntu kernel: [251005.307155] 	  Access_ID		: 0x7
Jul  9 11:55:38 ubuntu kernel: [251005.307156] 	  Fabric		: cbb-fabric
Jul  9 11:55:38 ubuntu kernel: [251005.314241] 	  Slave_Id		: 0xc
Jul  9 11:55:38 ubuntu kernel: [251005.317392] 	  Burst_length		: 0x0
Jul  9 11:55:38 ubuntu kernel: [251005.321067] 	  Burst_type		: 0x1
Jul  9 11:55:38 ubuntu kernel: [251005.324480] 	  Beat_size		: 0x2
Jul  9 11:55:38 ubuntu kernel: [251005.327806] 	  VQC			: 0x0
Jul  9 11:55:38 ubuntu kernel: [251005.330693] 	  GRPSEC		: 0x7e
Jul  9 11:55:38 ubuntu kernel: [251005.333667] 	  FALCONSEC		: 0x0
Jul  9 11:55:38 ubuntu kernel: [251005.336992] 	**************************************
Jul  9 11:55:38 ubuntu kernel: [251005.341999] ------------[ cut here ]------------
Jul  9 11:55:38 ubuntu kernel: [251005.342010] WARNING: CPU: 0 PID: 224501 at drivers/soc/tegra/cbb/tegra234-cbb.c:577 tegra234_cbb_isr+0x130/0x170
Jul  9 11:55:38 ubuntu kernel: [251005.352130] Modules linked in: fuse(E) veth(E) nf_conntrack_netlink(E) xt_nat(E) iptable_raw(E) xt_conntrack(E) xt_MASQUERADE(E) nfnetlink(E) ip6table_nat(E) ip6table_filter(E) ip6_tables(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) xt_addrtype(E) nvidia_modeset(OE) mttcan(E) can_dev(E) can_raw(E) can(E) iptable_filter(E) xt_tcpudp(E) lzo_rle(E) lzo_compress(E) zram(E) overlay(E) ramoops(E) reed_solomon(E) loop(E) nvgpu(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_asrc(E) snd_soc_tegra186_dspk(E) ghash_ce(E) snd_soc_tegra186_arad(E) r8168(E) snd_soc_tegra210_iqc(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_afc(E) snd_hda_codec_hdmi(E) snd_soc_tegra210_dmic(E) sha2_ce(E) ahci(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) snd_hda_tegra(E) binfmt_misc(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_adsp(E) sha256_arm64(E) snd_soc_tegra210_i2s(E) libahci(E) snd_soc_tegra210_mixer(E)
Jul  9 11:55:38 ubuntu kernel: [251005.352198]  snd_soc_tegra_machine_driver(E) sha1_ce(E) ucsi_ccg(E) snd_soc_tegra_pcm(E) snd_hda_codec(E) snd_soc_tegra210_sfc(E) libata(E) snd_soc_tegra_utils(E) nvadsp(E) typec_ucsi(E) snd_hda_core(E) snd_soc_simple_card_utils(E) pwm_fan(E) snd_soc_spdif_tx(E) snd_soc_tegra210_ahub(E) typec(E) snd_soc_cs42l42(E) tegra_bpmp_thermal(E) ina3221(E) nct1008(E) tegra210_adma(E) userspace_alert(E) spi_tegra114(E) nvidia(OE) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
Jul  9 11:55:38 ubuntu kernel: [251005.352234] CPU: 0 PID: 224501 Comm: DA5742631_R Tainted: G           OE     5.10.104-tegra #6
Jul  9 11:55:38 ubuntu kernel: [251005.352235] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 3.1-32827747 03/19/2023
Jul  9 11:55:38 ubuntu kernel: [251005.352238] pstate: 60400089 (nZCv daIf +PAN -UAO -TCO BTYPE=--)
Jul  9 11:55:38 ubuntu kernel: [251005.352240] pc : tegra234_cbb_isr+0x130/0x170
Jul  9 11:55:38 ubuntu kernel: [251005.352241] lr : tegra234_cbb_isr+0x10c/0x170
Jul  9 11:55:38 ubuntu kernel: [251005.352242] sp : ffff800010003e10
Jul  9 11:55:38 ubuntu kernel: [251005.352243] x29: ffff800010003e10 x28: ffff39c89d208e80 
Jul  9 11:55:38 ubuntu kernel: [251005.352245] x27: 0000000000000001 x26: 0000000000000080 
Jul  9 11:55:38 ubuntu kernel: [251005.352247] x25: ffffd6d7077d5900 x24: ffffd6d70812ce40 
Jul  9 11:55:38 ubuntu kernel: [251005.352249] x23: ffffd6d707ab7000 x22: 0000000000000022 
Jul  9 11:55:38 ubuntu kernel: [251005.352251] x21: ffffd6d707f4f1a0 x20: 0000000000000002 
Jul  9 11:55:38 ubuntu kernel: [251005.352253] x19: ffffd6d707f4f190 x18: 0000000000000010 
Jul  9 11:55:38 ubuntu kernel: [251005.352255] x17: 0000000000000000 x16: ffffd6d7060b3210 
Jul  9 11:55:38 ubuntu kernel: [251005.352257] x15: ffff39c89d2093f0 x14: ffffffffffffffff 
Jul  9 11:55:38 ubuntu kernel: [251005.352259] x13: ffff800090003917 x12: ffff800010003920 
Jul  9 11:55:38 ubuntu kernel: [251005.352261] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f 
Jul  9 11:55:38 ubuntu kernel: [251005.352263] x9 : ffff800010003c30 x8 : 2a2a2a2a2a2a2a2a 
Jul  9 11:55:38 ubuntu kernel: [251005.352265] x7 : 2a2a2a2a2a2a2a09 x6 : c0000000ffffefff 
Jul  9 11:55:38 ubuntu kernel: [251005.352267] x5 : ffff39ceaeb1f958 x4 : ffffd6d707dd7be8 
Jul  9 11:55:38 ubuntu kernel: [251005.352268] x3 : 0000000000000000 x2 : ffffd6d70624e170 
Jul  9 11:55:38 ubuntu kernel: [251005.352270] x1 : ffff39c89d208e80 x0 : 0000000100010000 
Jul  9 11:55:38 ubuntu kernel: [251005.352273] Call trace:
Jul  9 11:55:38 ubuntu kernel: [251005.352275]  tegra234_cbb_isr+0x130/0x170
Jul  9 11:55:38 ubuntu kernel: [251005.352280]  __handle_irq_event_percpu+0x68/0x2a0
Jul  9 11:55:38 ubuntu kernel: [251005.352282]  handle_irq_event_percpu+0x40/0xa0
Jul  9 11:55:38 ubuntu kernel: [251005.352284]  handle_irq_event+0x50/0xf0
Jul  9 11:55:38 ubuntu kernel: [251005.352286]  handle_fasteoi_irq+0xc0/0x170
Jul  9 11:55:38 ubuntu kernel: [251005.352288]  generic_handle_irq+0x40/0x60
Jul  9 11:55:38 ubuntu kernel: [251005.352290]  __handle_domain_irq+0x70/0xd0
Jul  9 11:55:38 ubuntu kernel: [251005.352293]  gic_handle_irq+0x68/0x134
Jul  9 11:55:38 ubuntu kernel: [251005.352294]  el1_irq+0xd0/0x180
Jul  9 11:55:38 ubuntu kernel: [251005.352297]  el0_svc_common.constprop.0+0x48/0x1d0
Jul  9 11:55:38 ubuntu kernel: [251005.352299]  do_el0_svc+0x38/0xb0
Jul  9 11:55:38 ubuntu kernel: [251005.352303]  el0_svc+0x1c/0x30
Jul  9 11:55:38 ubuntu kernel: [251005.352305]  el0_sync_handler+0xa8/0xb0
Jul  9 11:55:38 ubuntu kernel: [251005.352306]  el0_sync+0x16c/0x180
Jul  9 11:55:38 ubuntu kernel: [251005.352307] ---[ end trace ffaa7128db814e10 ]---
Jul  9 11:55:38 ubuntu kernel: [251005.419293] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:38 ubuntu kernel: [251005.421617] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.423435] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.425745] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.427158] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.429452] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.431155] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.433290] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.439169] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.449602] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.459150] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.469083] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.479139] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.490224] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.500891] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.510396] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.520019] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.531147] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.541676] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.551137] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.561925] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.572370] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.582370] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.592610] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.602858] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.612779] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.623144] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.633539] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.643142] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.655137] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed
Jul  9 11:55:39 ubuntu kernel: [251005.664574] nvgpu: 17000000.ga10b        nvgpu_engine_fb_queue_push:401  [ERR]  falcon id-0, queue id-1, failed

syslog.7.10.log (66.8 MB)
kernel_error_11_55.txt (130.2 KB)

3

Any specific repro steps? Failure rate?
Can you try with newer JetPack SW?

5

复现概率比较低，10台设备左右，半年就出现2次。

主要复现概率很低，我问问客户愿不愿意用jp6.2的版本试试

6

Hi,

Recommend upgrading to JP6.2, as some GPU fixes that are not merged in 5.1.2.

Thanks

7

jp5.1.5的版本有解决此问题吗
或者jp5.X的有解决此问题的吗

9

Hi,

Please try JP 5.1.6 instead from Jetson Linux | NVIDIA Developer

Thanks

10

收到，我们在jp5.1.6上试试

11

There is no update from you for a period, assuming this is not an issue anymore.
Hence, we are closing this topic. If need further support, please open a new one.
Thanks ~0812

Hi,

Wait for update.

Thanks