GPU freeze issue on jetson agx orin

[user160627]

am using jetson agx orin for my robotics development am running 3 parallel algorithms in 3 terminal which will using GPU on the agx orin board , when iam running the algorithm pc or gpu gets freeze sometimes, i checked with kernel logs and found the below fails on the gpu attaching the log below

[ +16.364213] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013384] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012540] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013293] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012596] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012935] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013175] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[Jul 8 13:34] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013412] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012994] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012764] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012586] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012613] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013093] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013288] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +7.458744] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013828] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011838] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011974] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012092] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011806] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011981] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011968] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +2.398575] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012259] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011908] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011880] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +4.922443] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012826] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011906] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012802] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +7.495666] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.015038] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012283] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012567] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013891] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012688] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012712] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013400] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[Jul 8 13:35] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014883] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012393] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011914] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012944] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013232] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012459] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012128] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +3.661255] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013782] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012649] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012112] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013335] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013235] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012154] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012502] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013994] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013371] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011865] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011945] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011951] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012078] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011927] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012059] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011967] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012010] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011951] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011957] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012027] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012047] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012017] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011995] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011990] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012012] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012004] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011987] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012012] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011974] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012006] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012002] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011978] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011986] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012880] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012410] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012387] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012401] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013068] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012667] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012206] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011983] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012046] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011968] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011997] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011964] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012066] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011933] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011973] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011988] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012031] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012050] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012023] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011982] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011971] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011987] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012019] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012004] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012143] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011851] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011994] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012052] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011901] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012038] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012139] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011905] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011983] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012063] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012706] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012417] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012401] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012215] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012402] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012560] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012727] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012314] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012255] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011953] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012000] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012028] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012365] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012413] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012399] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012399] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012436] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013065] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012680] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012261] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012723] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012414] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012492] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012960] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.002968] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.010170] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.007443] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000024] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.012745] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.009427] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.224086] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012794] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012397] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012233] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012373] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014094] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012438] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012456] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013138] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012290] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012185] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012047] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011994] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011952] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012418] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012408] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012678] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012401] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012932] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012991] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012319] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011914] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011995] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012060] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011938] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011967] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012079] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011957] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012026] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011985] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011973] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012024] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012145] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011892] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011964] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012013] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011955] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012835] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011971] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012401] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012580] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013100] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012897] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012226] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012691] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012530] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012405] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012406] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013089] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012648] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012207] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012317] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012758] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012060] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011685] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012407] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012387] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012396] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012988] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012793] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012216] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011927] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012204] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011879] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012174] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012330] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012289] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012390] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012396] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012402] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013124] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012415] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012267] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012209] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012204] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011910] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011939] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012299] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012407] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012412] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012404] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012409] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012988] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012838] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012213] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012025] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012021] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012020] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011951] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011997] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.000563] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.012722] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.007928] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.002499] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.009383] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.007681] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000009] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000014] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000001] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000008] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000001] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000008] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000010] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000010] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.015017] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.584065] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012562] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012678] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012210] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012347] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012704] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012986] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +31.642042] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012418] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012800] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013057] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013393] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013746] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012741] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012528] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +7.511238] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014141] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012719] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011906] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011925] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011751] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011891] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013635] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +1.150706] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013898] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012842] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011834] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011963] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012423] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012306] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012901] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +6.100175] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012108] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013079] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012330] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012212] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011952] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013197] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012036] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[Jul 8 13:36] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013650] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011933] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012079] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012445] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012473] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013314] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013075] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +12.783244] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013299] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.001103] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.013094] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.006317] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000015] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.011991] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.010387] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.012522] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.055255] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012832] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012498] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013043] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +16.476800] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013875] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012464] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012049] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012525] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012545] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012979] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014288] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +7.300676] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014338] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012496] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012203] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012803] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014296] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013317] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +7.406468] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013356] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012322] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012169] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012748] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011898] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013789] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +7.356898] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013060] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012132] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012094] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011925] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012981] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013584] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013874] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[Jul 8 13:37] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013735] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013570] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012675] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012535] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014348] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011829] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012985] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +4.021282] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012271] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011966] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013592] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013058] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012959] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +10.316838] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +33.372797] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013888] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012244] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012332] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012262] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012071] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011745] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014453] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012087] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012998] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012440] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012405] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013086] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012758] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012294] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011966] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012012] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012832] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012294] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012812] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013069] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012732] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013018] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013012] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013020] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012911] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012221] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011990] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012044] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012064] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011950] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012015] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011940] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012561] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013015] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012948] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013048] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012222] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012907] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012679] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013333] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013076] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012210] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011970] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012077] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012075] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011954] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011977] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012018] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012062] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011908] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011948] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011982] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012075] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011997] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012005] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011999] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012008] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012611] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012458] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012416] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012415] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013146] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012721] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012299] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011878] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012108] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012000] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011985] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011936] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013006] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012425] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012409] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013019] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012906] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012304] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011976] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012009] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012086] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011879] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012022] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012036] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012574] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012421] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012420] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012408] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013244] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012627] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012256] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011964] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012437] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012418] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.000272] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.012142] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.008016] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000011] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000008] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000008] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000001] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000004] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.010444] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.002288] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.005990] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.002414] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.001218] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000046] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.009921] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.014367] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.008934] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.012146] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.229810] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012849] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012849] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.015722] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +6.065906] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012883] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012717] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012225] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012769] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011929] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011982] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.016173] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[Jul 8 13:38] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014639] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011898] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011768] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012681] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011902] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013222] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014080] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +7.394359] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.014742] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011896] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011909] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013306] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012912] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012390] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013092] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +5.431843] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.015018] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012755] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011842] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011729] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012484] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011964] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013290] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012427] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013016] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +9.706086] usb 1-4.3: USB disconnect, device number 6
[ +1.788261] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013996] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012761] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012308] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012652] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012432] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +1.136395] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013650] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012402] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011974] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011899] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012804] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012304] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013718] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013163] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012578] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012420] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012939] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012875] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012256] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011943] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011970] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012001] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012165] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011866] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011977] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012054] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012001] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012002] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011957] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011984] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012019] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012023] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012029] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011940] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011998] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012613] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011964] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012401] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012418] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012415] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012100] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012410] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012505] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012990] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012209] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011965] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012005] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012166] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011845] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011974] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012023] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012065] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011869] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011982] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012043] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012274] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011819] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011947] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011985] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011983] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012033] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011998] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012015] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011983] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012115] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011885] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011978] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012024] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012047] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011940] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011965] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012046] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012027] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011973] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012018] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011989] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012363] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011905] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012416] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011855] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012059] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012456] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012390] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012403] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012899] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012872] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012173] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012174] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012109] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012053] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011888] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011960] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012037] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012260] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012549] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012412] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012398] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.008931] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.003476] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.008514] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000033] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000012] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.012122] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.002284] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.007534] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.090327] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012746] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012547] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012220] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012127] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012988] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012790] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013059] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012437] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012157] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012015] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012044] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012005] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012016] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012007] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011972] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011996] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011986] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011950] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012037] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012001] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011997] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012003] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012001] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012035] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011953] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012022] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012001] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011983] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012022] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011945] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012052] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011948] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012006] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012055] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012007] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012155] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011861] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011961] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011975] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012065] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012011] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011963] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012016] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012033] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011915] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011955] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012023] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012046] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012017] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012001] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011987] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011913] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012020] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012075] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011981] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012232] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011774] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011994] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011980] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012016] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011988] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011995] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012075] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012026] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011940] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011999] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012036] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011940] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012013] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011951] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012013] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012011] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011986] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012039] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012021] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011968] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011982] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011970] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012055] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012027] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012326] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012669] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012421] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012413] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012606] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012406] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012018] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012667] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012235] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012160] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.001776] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.010575] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.008215] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000006] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000003] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000005] nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available
[ +0.000007] nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4
[ +0.000002] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.155255] nvgpu: 17000000.ga10b nvgpu_pmu_perfmon_get_samples_rpc:678 [ERR] Failed to execute RPC, status=0xfffffff5
[ +0.000514] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.023822] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012969] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.012953] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013081] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.013528] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed
[ +0.011975] nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed

[carolyuu]

*** Please note that this reply is generated by LLM automatically ***

The provided log output appears to be from a NVIDIA Jetson AGX Orin device, specifically the GPU driver. The log is filled with error messages indicating that the GPU is experiencing issues with queue pushes and RPC (Remote Procedure Call) executions.

The most common error message is:

nvgpu: 17000000.ga10b nvgpu_engine_fb_queue_push:401 [ERR] falcon id-0, queue id-1, failed

This error message suggests that the GPU is having trouble pushing data to a queue, specifically the falcon id-0, queue id-1 queue. The falcon likely refers to a specific hardware component or engine within the GPU.

Another common error message is:

nvgpu: 17000000.ga10b nvgpu_pmu_seq_acquire:131 [ERR] no free sequence available

This error message indicates that the GPU’s performance monitoring unit (PMU) is unable to acquire a free sequence, which is likely a resource required for the GPU to function properly.

The log also contains error messages related to RPC executions, such as:

nvgpu: 17000000.ga10b nvgpu_pmu_rpc_execute:713 [ERR] Failed to execute RPC status=0xfffffff5, func=0x4

This error message suggests that the GPU is having trouble executing RPCs, which are likely used for communication between different components of the GPU or between the GPU and other system components.

Overall, the log output suggests that the GPU is experiencing a range of issues related to queue management, performance monitoring, and RPC execution. These issues may be causing the system to freeze or become unresponsive.

To troubleshoot this issue, you may want to try the following:

1. Check the system’s power management settings to ensure that the GPU is not being throttled or limited in any way.
2. Verify that the GPU driver is up-to-date and functioning properly.
3. Check the system’s logs for any other error messages that may be related to the GPU or system hardware.
4. Consider running a stress test or benchmarking tool to see if the issue is reproducible and to gather more information about the system’s performance.
5. If the issue persists, you may want to try resetting the GPU or system to its default settings or seeking further assistance from NVIDIA support or a system administrator.

In comparison to other NVIDIA Jetson devices, the AGX Orin is a more powerful and complex device, with a more advanced GPU architecture. As such, it may be more prone to issues related to queue management and RPC execution. However, the specific error messages and issues reported in the log output are not unique to the AGX Orin and may be applicable to other NVIDIA Jetson devices as well.

In terms of tools and resources that may be helpful in troubleshooting this issue, you may want to consider the following:

• NVIDIA’s Jetson AGX Orin documentation and support resources
• The NVIDIA GPU driver documentation and support resources
• System monitoring and debugging tools, such as nvidia-smi and syslog
• Stress testing and benchmarking tools, such as glmark2 and sysbench
• Online forums and communities, such as the NVIDIA Jetson forums and Reddit’s r/JetsonNano community.

[AastaLLL]

Hi,

Which JetPack version do you use?
To debug further, could you share the reproducible steps with us so we can investigate it locally?

Thanks.

[user160627]

1. am using jetpack 5.1.2
2. Am running three ros2 nodes which runs parallely(all have separate torch models running in different fps) which get the data from a common camera node (here am using a camera that runs on 30 fps)
   and finally these all 3 nodes will send the processed image outcome to a final node where i will do some cpu operations and display in opencv window

am using following tools and libraries

1. Python 3.8
2. Opencv 4.10.0
3. pytorch 2.0.0+nv23.05 - which is build from jetson embedded downloads and installed
   4.ros2 foxy

any other things needed let me know

we are in production level stage so please consider and check ASAP

[AastaLLL]

There is no update from you for a period, assuming this is not an issue anymore.
Hence, we are closing this topic. If need further support, please open a new one.
Thanks ~0730

Hi,

Could you share a reproducible source so we can check it further?

Thanks.