Gsp wait for init timedout with AGX ORIN Jetpack R36.4.4

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Hi

We get GSP panic issue when using AGX ORIN Jetpack R36.4.4

Apr 16 10:48:28 chassis kernel: [ 5263.860510] nvethernet 6800000.ethernet: config.flags = 0x0, tx_type = 0x1,rx_filter = 0xc
Apr 16 10:59:43 chassis kernel: [ 5939.127371] nvethernet 6800000.ethernet: config.flags = 0x0, tx_type = 0x1,rx_filter = 0xc
Apr 16 11:00:05 chassis kernel: [ 5961.365189] nvgpu: 17000000.gpu gsp_sched_wait_for_init:110 [ERR] gsp wait for init timedout
Apr 16 11:00:05 chassis kernel: [ 5961.365283] nvgpu: 17000000.gpu nvgpu_gsp_sched_bootstrap_hs:209 [ERR] gsp wait for basic init failed
Apr 16 11:00:05 chassis kernel: [ 5961.365344] nvgpu: 17000000.gpu nvgpu_finalize_poweron:1095 [ERR] Failed initialization for: &nvgpu_gsp_sched_bootstrap_hs
Apr 16 11:00:05 chassis kernel: [ 5961.526290] nvethernet 6800000.ethernet: config.flags = 0x0, tx_type = 0x1,rx_filter = 0xc
Apr 16 11:00:22 chassis kernel: [ 5978.353166] thermal cooling_device12: hot-surface-alert cooling state: 1 → 0
Apr 16 11:00:28 chassis kernel: [ 5984.576704] rcu: INFO: rcu_preempt self-detected stall on CPU
Apr 16 11:00:28 chassis kernel: [ 5984.576711] rcu: 0-…: (5249 ticks this GP) idle=63f/1/0x4000000000000002 softirq=1057721/1082275 fqs=2328
Apr 16 11:00:28 chassis kernel: [ 5984.576717] (t=5250 jiffies g=1303105 q=309526)
Apr 16 11:00:28 chassis kernel: [ 5984.576720] Task dump for CPU 0:
Apr 16 11:00:28 chassis kernel: [ 5984.576721] task:irq/208-gk20a_s state:R running task stack: 0 pid: 361 ppid: 2 flags:0x0000000a
Apr 16 11:00:28 chassis kernel: [ 5984.576726] Call trace:
Apr 16 11:00:28 chassis kernel: [ 5984.576727] dump_backtrace+0x0/0x1d0
Apr 16 11:00:28 chassis kernel: [ 5984.576738] show_stack+0x34/0x50
Apr 16 11:00:28 chassis kernel: [ 5984.576741] sched_show_task+0x17c/0x1b0
Apr 16 11:00:28 chassis kernel: [ 5984.576746] dump_cpu_task+0x50/0x68
Apr 16 11:00:28 chassis kernel: [ 5984.576750] rcu_dump_cpu_stacks+0xf4/0x140
Apr 16 11:00:28 chassis kernel: [ 5984.576753] rcu_sched_clock_irq+0xc9c/0xef0
Apr 16 11:00:28 chassis kernel: [ 5984.576755] update_process_times+0xa8/0xf0
Apr 16 11:00:28 chassis kernel: [ 5984.576759] tick_sched_handle+0x4c/0x80
Apr 16 11:00:28 chassis kernel: [ 5984.576760] tick_sched_timer+0x58/0xc0
Apr 16 11:00:28 chassis kernel: [ 5984.576762] __hrtimer_run_queues+0x184/0x360
Apr 16 11:00:28 chassis kernel: [ 5984.576764] hrtimer_interrupt+0xf4/0x240
Apr 16 11:00:28 chassis kernel: [ 5984.576766] arch_timer_handler_phys+0x44/0x70
Apr 16 11:00:28 chassis kernel: [ 5984.576770] handle_percpu_devid_irq+0x94/0x260
Apr 16 11:00:28 chassis kernel: [ 5984.576773] handle_domain_irq+0x74/0xc0
Apr 16 11:00:28 chassis kernel: [ 5984.576774] gic_handle_irq+0x68/0x170
Apr 16 11:00:28 chassis kernel: [ 5984.576778] call_on_irq_stack+0x20/0x50
Apr 16 11:00:28 chassis kernel: [ 5984.576780] do_interrupt_handler+0x78/0xa0
Apr 16 11:00:28 chassis kernel: [ 5984.576782] el1_interrupt+0x30/0x70
Apr 16 11:00:28 chassis kernel: [ 5984.576785] el1h_64_irq_handler+0x18/0x30
Apr 16 11:00:28 chassis kernel: [ 5984.576787] el1h_64_irq+0x7c/0x80
Apr 16 11:00:28 chassis kernel: [ 5984.576788] queued_spin_lock_slowpath+0x264/0x3a0
Apr 16 11:00:28 chassis kernel: [ 5984.576791] _raw_spin_lock+0x9c/0xc0
Apr 16 11:00:28 chassis kernel: [ 5984.576792] __mutex_lock.constprop.0+0xa4/0x600
Apr 16 11:00:28 chassis kernel: [ 5984.576795] __mutex_lock_slowpath+0x2c/0x50
Apr 16 11:00:28 chassis kernel: [ 5984.576797] mutex_lock+0x84/0xa0
Apr 16 11:00:28 chassis kernel: [ 5984.576800] nvgpu_gsp_isr_mutex_acquire+0x28/0x40 [nvgpu]
Apr 16 11:00:28 chassis kernel: [ 5984.576890] ga10b_gsp_isr+0xc0/0x410 [nvgpu]
Apr 16 11:00:28 chassis kernel: [ 5984.576954] nvgpu_gsp_sched_isr+0x34/0x50 [nvgpu]
Apr 16 11:00:28 chassis kernel: [ 5984.577015] nvgpu_gsp_isr+0x38/0x60 [nvgpu]
Apr 16 11:00:28 chassis kernel: [ 5984.577075] ga10b_intr_isr_stall+0x660/0x970 [nvgpu]
Apr 16 11:00:28 chassis kernel: [ 5984.577134] nvgpu_cic_mon_intr_stall_handle+0x48/0xa0 [nvgpu]
Apr 16 11:00:28 chassis kernel: [ 5984.577193] __traceiter_gk20a_cde_finished_ctx_cb+0x1098/0x1310 [nvgpu]
Apr 16 11:00:28 chassis kernel: [ 5984.577251] irq_thread_fn+0x38/0xd0
Apr 16 11:00:28 chassis kernel: [ 5984.577254] irq_thread+0x1ac/0x320
Apr 16 11:00:28 chassis kernel: [ 5984.577256] kthread+0x184/0x1a0
Apr 16 11:00:28 chassis kernel: [ 5984.577257] ret_from_fork+0x10/0x20

bootlog_33.log (143.9 KB)

3

How this issue happened? Any reproduce steps? On devkit?

5

Actually I don’t know how to reproduce it. We only encountered this issue once

Yes，it is devkit

We use agx orin as the chassis controller for our robot.
That day, the tester informed me that she couldn’t log into the agx orin via the Ethernet tool SSH robot@xxx.xxx.xxx.xxx.
I tried to connect this agx orin through serial port and enter the username and password, but the terminal didn’t respond.
I had to restart the device and then retrieve the logs.

6

I notice the thermal log appeared in your log which indicates the environment seems over 70c.

What is the exact operating situation there? Have you checked tegrastats to see the exact temperature of your enviromennt?

8

chassis_monitor.zip (4.4 MB)

When error happened，the tegrastat log below

[2026-04-16 11:01:01] === [2026-04-16 11:00:55] 系统性能数据 ===
[2026-04-16 11:01:01] CPU: all_used:67.5% user:29.0% kernel:38.5% idle:32.5% iowait:0.0%
[2026-04-16 11:01:01] Load: 1min:56.17 5min:40.80 15min:34.92
[2026-04-16 11:01:01] Memory: total:62839MB used:26301MB free:960MB avail:34052MB use:41.9%
[2026-04-16 11:01:01] Swap: total:31419MB used:13MB free:31406MB use:0.0%
[2026-04-16 11:01:01] Processes: total:420 zombie:0
[2026-04-16 11:01:01] eno1 bandwidth: RX:34.09MB/s TX:0.17MB/s
[2026-04-16 11:01:01] eno1 status: eno1 UP 3c:6d:66:b1:2a:54 <BROADCAST,MULTICAST,UP,LOWER_UP>
[2026-04-16 11:01:01] tegrastats: tegrastats: Timeout/Error
[2026-04-16 11:01:01] — WHITELIST processes (pid ppid comm user_cpu kernel_cpu total_cpu mem cmdline) —
[2026-04-16 11:01:01] 5793 5612 slam_shm_online_wrapper 116.02 50.18 166.20 18.3% /opt/robby/robby-slam/install/slam/lib/slam/slam_shm_online_wrapper --ros-args --params-file /tmp/launch_params_0xjpahaw --params-file /tmp/launch_params_9uze8n92
[2026-04-16 11:01:01] 3065 1 RobbyOutPost 38.99 53.86 92.85 5.4% ./bin/RobbyOutPost RobbyR2_CHASSIS_ORIN_FUNCTION
[2026-04-16 11:01:01] 3066 1 perception_node 51.78 36.92 88.70 12.1% /opt/robby/robby-perception/lib/perception/perception_node --ros-args -r __node:=perception_node --params-file /opt/robby/robby-perception/share/perception/config/config_for_evt3.yaml
[2026-04-16 11:01:01] 2475 1792 RobbySensorCenter 39.95 22.53 62.48 1.9% /opt/robby/robby-sensor-center/bin/RobbySensorCenter /opt/robby/robby-sensor-center/config/chassis_function/sensor_center/sensor_global_cam_orb_jpeg.json
[2026-04-16 11:01:01] 2546 1792 RobbySensorCenter 12.31 14.06 26.37 0.3% /opt/robby/robby-sensor-center/bin/RobbySensorCenter /opt/robby/robby-sensor-center/config/chassis_function/sensor_center/sensor_global_lidar.json
[2026-04-16 11:01:01] 4376 4132 map_node 0.48 0.32 0.80 0.1% /opt/robby/robby-navi/robby_navi/lib/robby_navi/map_node --ros-args -r __node:=map_manager --params-file /opt/robby/robby-navi/robby_navi/share/robby_navi/config/robby_navi_config.yaml
[2026-04-16 11:01:01] 4373 4132 navi_node 0.32 0.48 0.80 0.1% /opt/robby/robby-navi/robby_navi/lib/robby_navi/navi_node --ros-args -r __node:=navi_node --params-file /opt/robby/robby-navi/robby_navi/share/robby_navi/config/robby_navi_config.yaml
[2026-04-16 11:01:01] 4380 4132 data_node 0.16 0.16 0.32 0.1% /opt/robby/robby-navi/robby_navi/lib/robby_navi/data_node --ros-args -r __node:=object_manager --params-file /opt/robby/robby-navi/robby_navi/share/robby_navi/config/robby_navi_config.yaml
[2026-04-16 11:01:01] 5164 4751 global_planner_service 0.00 0.16 0.16 0.2% /opt/robby/robby-planning-and-control/nav_planner_app/lib/nav_planner_app/global_planner_service --ros-args -r __node:=global_planner_service -r __ns:=/ --params-file /tmp/launch_params_ke4gleb7 --params-file /tmp/launch_params_xnk4g75r --params-file /tmp/launch_params_kfd4e3lp --params-file /tmp/launch_params_b6giqpy1
[2026-04-16 11:01:01] ========================================
[2026-04-16 11:01:01]

9

There is no update from you for a period, assuming this is not an issue anymore.
Hence, we are closing this topic. If need further support, please open a new one.
Thanks ~0506

please help try to figure out the method that could reproduce issue.