我的GPU是 RTX 6000 PRO 96G工作站版 2张
我是用的 gpu_burn 做的压测，分别跑了10分钟，1小时，10小时，GPU负载拉满（温度：85 - 90，显存：85G，使用率：100%），测了一晚上，没掉驱动。
但是我们跑训练的时候，程序一启动就掉，功耗、温度都还没上来，就掉一个。而且每次都掉这个 GPU1: 0000:91:00.0
GPU0还能正常用。
root@huamod-algorithm:/home/wangrui# nvidia-smi
Unable to determine the device handle for GPU1: 0000:91:00.0: Unknown Error
Tue May 19 11:15:03 2026
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 |
±----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX PRO 6000 Blac… On | 00000000:61:00.0 Off | 0 |
| 30% 50C P1 147W / 600W | 860MiB / 97887MiB | 30% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 91290 C /app/.venv/bin/python 850MiB |
±----------------------------------------------------------------------------------------+
掉驱动的时候，系统日志如下：
2026-05-19T10:59:46.956451+08:00 huamod-algorithm kernel: pcieport 0000:90:01.1: pciehp: Slot(21): Link Down
2026-05-19T10:59:47.215438+08:00 huamod-algorithm kernel: snd_hda_codec_hdmi hdaudioC1D0: HDMI: invalid ELD buf size -1
2026-05-19T10:59:47.215467+08:00 huamod-algorithm kernel: message repeated 15 times: [ snd_hda_codec_hdmi hdaudioC1D0: HDMI: invalid ELD buf size -1]
2026-05-19T10:59:47.215468+08:00 huamod-algorithm kernel: snd_hda_intel 0000:91:00.1: GPU sound probed, but not operational: please add a quirk to driver_denylist
2026-05-19T10:59:47.415428+08:00 huamod-algorithm kernel: [drm] [nvidia-drm] [GPU ID 0x00009100] Removing device
2026-05-19T10:59:47.415436+08:00 huamod-algorithm kernel: [drm] [nvidia-drm] [GPU ID 0x00009100] Unloading driver
2026-05-19T10:59:47.628195+08:00 huamod-algorithm kernel: NVRM: GPU at PCI:0000:91:00: GPU-6a8685da-2567-f4f7-6348-7e07109daf53
2026-05-19T10:59:47.628202+08:00 huamod-algorithm kernel: NVRM: GPU Board Serial Number: 1790226112000
2026-05-19T10:59:47.628203+08:00 huamod-algorithm kernel: NVRM: Xid (PCI:0000:91:00): 79, GPU has fallen off the bus.
2026-05-19T10:59:47.628204+08:00 huamod-algorithm kernel: NVRM: GPU 0000:91:00.0: GPU has fallen off the bus.
2026-05-19T10:59:47.628205+08:00 huamod-algorithm kernel: NVRM: GPU 0000:91:00.0: GPU serial number is 1790226112000.
2026-05-19T10:59:47.628206+08:00 huamod-algorithm kernel: NVRM: krcRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.
2026-05-19T10:59:47.628207+08:00 huamod-algorithm kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
2026-05-19T10:59:47.628215+08:00 huamod-algorithm kernel: message repeated 7 times: [ NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!]
2026-05-19T10:59:47.628216+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167292!
2026-05-19T10:59:47.628221+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628223+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167293!
2026-05-19T10:59:47.628224+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628226+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167294!
2026-05-19T10:59:47.628227+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628228+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167295!
2026-05-19T10:59:47.628239+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628241+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167296!
2026-05-19T10:59:47.628242+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628243+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167297!
2026-05-19T10:59:47.628244+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628245+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167298!
2026-05-19T10:59:47.628248+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628249+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167299!
2026-05-19T10:59:47.628250+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628251+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167300!
2026-05-19T10:59:47.628252+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628254+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167301!
2026-05-19T10:59:47.628255+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628256+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167302!
2026-05-19T10:59:47.628256+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628257+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167303!
2026-05-19T10:59:47.628258+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628260+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167304!
2026-05-19T10:59:47.628261+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628262+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167305!
2026-05-19T10:59:47.628263+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628264+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167306!
2026-05-19T10:59:47.628266+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628267+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167307!
2026-05-19T10:59:47.628268+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628268+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167308!
2026-05-19T10:59:47.628269+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628271+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167309!
2026-05-19T10:59:47.628273+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628273+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167310!
2026-05-19T10:59:47.628274+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628275+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167311!
2026-05-19T10:59:47.628276+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628278+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167312!
2026-05-19T10:59:47.628323+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628325+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167313!
2026-05-19T10:59:47.628326+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628328+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167314!
2026-05-19T10:59:47.628329+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628331+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167315!
2026-05-19T10:59:47.628332+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628334+08:00 huamod-algorithm kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 167316!
2026-05-19T10:59:47.628335+08:00 huamod-algorithm kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
2026-05-19T10:59:47.628337+08:00 huamod-algorithm kernel: NVRM: RmLogGpuCrash: RmLogGpuCrash: failed to save GPU crash data
2026-05-19T10:59:47.628340+08:00 huamod-algorithm kernel: NVRM: _kgspLogRpcSanityCheckFailure: GPU1 sanity check failed 0xf waiting for RPC response from GSP. Expected function 76 (GSP_RM_CONTROL) sequence 167291 (0x2080a61d 0xc).
2026-05-19T10:59:47.628341+08:00 huamod-algorithm kernel: NVRM: GPU1 GSP RPC buffer contains function 78 (DUMP_PROTOBUF_COMPONENT) sequence 167316 and data 0x0000000000000000 0x0000000000000000.
2026-05-19T10:59:47.628342+08:00 huamod-algorithm kernel: NVRM: GPU1 RPC history (CPU → GSP):
2026-05-19T10:59:47.628344+08:00 huamod-algorithm kernel: NVRM: entry function sequence data0 data1 ts_start ts_end duration actively_polling
2026-05-19T10:59:47.628345+08:00 huamod-algorithm kernel: NVRM: 0 76 GSP_RM_CONTROL 167291 0x000000002080a61d 0x000000000000000c 0x00065222df3806e8 0x0000000000000000 y
2026-05-19T10:59:47.628346+08:00 huamod-algorithm kernel: NVRM: -1 76 GSP_RM_CONTROL 167290 0x000000002080a61d 0x000000000000000c 0x00065222df28be0c 0x00065222df28bec1 181us
2026-05-19T10:59:47.628349+08:00 huamod-algorithm kernel: NVRM: -2 76 GSP_RM_CONTROL 167289 0x000000002080a61d 0x000000000000000c 0x00065222df197580 0x00065222df197631 177us
2026-05-19T10:59:47.628351+08:00 huamod-algorithm kernel: NVRM: -3 76 GSP_RM_CONTROL 167288 0x000000002080a61d 0x000000000000000c 0x00065222df0a2cdf 0x00065222df0a2d89 170us
2026-05-19T10:59:47.628352+08:00 huamod-algorithm kernel: NVRM: -4 76 GSP_RM_CONTROL 167287 0x0000000020800a4c 0x0000000000000004 0x00065222defae39c 0x00065222defae432 150us
2026-05-19T10:59:47.628353+08:00 huamod-algorithm kernel: NVRM: -5 76 GSP_RM_CONTROL 167286 0x000000000073010c 0x0000000000000010 0x00065222defae23e 0x00065222defae2f8 186us
2026-05-19T10:59:47.628354+08:00 huamod-algorithm kernel: NVRM: -6 76 GSP_RM_CONTROL 167285 0x000000000073010c 0x0000000000000010 0x00065222defae172 0x00065222defae22d 187us
2026-05-19T10:59:47.628356+08:00 huamod-algorithm kernel: NVRM: -7 76 GSP_RM_CONTROL 167284 0x000000000073010c 0x0000000000000010 0x00065222defae0a8 0x00065222defae161 185us
2026-05-19T10:59:47.628358+08:00 huamod-algorithm kernel: NVRM: GPU1 RPC event history (CPU ← GSP):
2026-05-19T10:59:47.628359+08:00 huamod-algorithm kernel: NVRM: entry function sequence data0 data1 ts_start ts_end duration during_incomplete_rpc
2026-05-19T10:59:47.628360+08:00 huamod-algorithm kernel: NVRM: 0 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x00065217a0160eb5 0x00065217a0160eb5
2026-05-19T10:59:47.628361+08:00 huamod-algorithm kernel: NVRM: -1 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x00065217a015fa8d 0x00065217a015fa8d
2026-05-19T10:59:47.628362+08:00 huamod-algorithm kernel: NVRM: -2 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x00065217a015f9b3 0x00065217a015f9b3
2026-05-19T10:59:47.628365+08:00 huamod-algorithm kernel: NVRM: -3 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x00065217a015f632 0x00065217a015f632
2026-05-19T10:59:47.628366+08:00 huamod-algorithm kernel: NVRM: -4 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x00065217a014c597 0x00065217a014c597
2026-05-19T10:59:47.628367+08:00 huamod-algorithm kernel: NVRM: -5 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x00065217a014c50f 0x00065217a014c50f
2026-05-19T10:59:47.628368+08:00 huamod-algorithm kernel: NVRM: -6 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000000 0x0000000000000000 0x00065217a014b7bf 0x00065217a014b7bf
2026-05-19T10:59:47.628369+08:00 huamod-algorithm kernel: NVRM: -7 4124 GSP_LOCKDOWN_NOTICE 0 0x0000000000000001 0x0000000000000000 0x00065217a0149fc5 0x00065217a0149fc5
2026-05-19T10:59:47.628372+08:00 huamod-algorithm kernel: CPU: 45 PID: 2813837 Comm: nvitop Tainted: G OE 6.8.0-111-generic #111-Ubuntu
2026-05-19T10:59:47.628373+08:00 huamod-algorithm kernel: Hardware name: Giga Computing MZ73-LM0-000/MZ73-LM0-000, BIOS R15_F37 08/08/2025
2026-05-19T10:59:47.628375+08:00 huamod-algorithm kernel: Call Trace:
2026-05-19T10:59:47.628376+08:00 huamod-algorithm kernel:
2026-05-19T10:59:47.628376+08:00 huamod-algorithm kernel: dump_stack_lvl+0x76/0xa0
2026-05-19T10:59:47.628377+08:00 huamod-algorithm kernel: dump_stack+0x10/0x20
2026-05-19T10:59:47.628379+08:00 huamod-algorithm kernel: os_dump_stack+0xe/0x20 [nvidia]
2026-05-19T10:59:47.628380+08:00 huamod-algorithm kernel: _kgspRpcRecvPoll+0x714/0x1220 [nvidia]
2026-05-19T10:59:47.628381+08:00 huamod-algorithm kernel: ? os_get_system_time+0x3e/0x90 [nvidia]
2026-05-19T10:59:47.628382+08:00 huamod-algorithm kernel: ? _kgspRpcSendMessage+0xd3/0x1a0 [nvidia]
2026-05-19T10:59:47.628383+08:00 huamod-algorithm kernel: _issueRpcAndWait+0xd5/0x920 [nvidia]
2026-05-19T10:59:47.628385+08:00 huamod-algorithm kernel: ? osGetCurrentThread+0x26/0x60 [nvidia]
2026-05-19T10:59:47.628385+08:00 huamod-algorithm kernel: ? rmDeviceGpuLockIsOwner+0x29/0x90 [nvidia]
2026-05-19T10:59:47.628388+08:00 huamod-algorithm kernel: ? os_mem_set+0x14/0x20 [nvidia]
2026-05-19T10:59:47.628389+08:00 huamod-algorithm kernel: rpcRmApiControl_GSP+0x76f/0x940 [nvidia]
2026-05-19T10:59:47.628390+08:00 huamod-algorithm kernel: RmGssLegacyRpcCmd+0x190/0x340 [nvidia]
2026-05-19T10:59:47.628402+08:00 huamod-algorithm kernel: ? RmDeprecatedVidHeapControl+0x80/0x80 [nvidia]
2026-05-19T10:59:47.628403+08:00 huamod-algorithm kernel: _nv04ControlWithSecInfo+0x47/0xa0 [nvidia]
2026-05-19T10:59:47.628404+08:00 huamod-algorithm kernel: ? rmapiControlWithSecInfoTls+0xf0/0xf0 [nvidia]
2026-05-19T10:59:47.628409+08:00 huamod-algorithm kernel: ? _rmAllocForDeprecatedApi+0x30/0x30 [nvidia]
2026-05-19T10:59:47.628409+08:00 huamod-algorithm kernel: ? _rmControlForDeprecatedApi+0x30/0x30 [nvidia]
2026-05-19T10:59:47.628410+08:00 huamod-algorithm kernel: ? _rmFreeForDeprecatedApi+0x20/0x20 [nvidia]
2026-05-19T10:59:47.628411+08:00 huamod-algorithm kernel: ? RmCopyUserForDeprecatedApi+0xe0/0xe0 [nvidia]
2026-05-19T10:59:47.628412+08:00 huamod-algorithm kernel: ? _rmMapMemoryForDeprecatedApi+0x30/0x30 [nvidia]
2026-05-19T10:59:47.628412+08:00 huamod-algorithm kernel: ? _rmAllocMemForDeprecatedApi+0x10/0x10 [nvidia]
2026-05-19T10:59:47.628413+08:00 huamod-algorithm kernel: RmIoctl+0x934/0xda0 [nvidia]
2026-05-19T10:59:47.628415+08:00 huamod-algorithm kernel: ? portSyncSpinlockAcquire+0x18/0x30 [nvidia]
2026-05-19T10:59:47.628416+08:00 huamod-algorithm kernel: ? os_release_spinlock+0x1a/0x30 [nvidia]
2026-05-19T10:59:47.628417+08:00 huamod-algorithm kernel: rm_ioctl+0x66/0x4f0 [nvidia]
2026-05-19T10:59:47.628418+08:00 huamod-algorithm kernel: ? _nv04ControlWithSecInfo+0x8d/0xa0 [nvidia]
2026-05-19T10:59:47.628419+08:00 huamod-algorithm kernel: nvidia_unlocked_ioctl+0x80a/0xb80 [nvidia]
2026-05-19T10:59:47.628420+08:00 huamod-algorithm kernel: ? syscall_exit_to_user_mode+0x43/0x1e0
2026-05-19T10:59:47.628423+08:00 huamod-algorithm kernel: __x64_sys_ioctl+0xa0/0xf0
2026-05-19T10:59:47.628425+08:00 huamod-algorithm kernel: x64_sys_call+0x12a3/0x25a0
2026-05-19T10:59:47.628426+08:00 huamod-algorithm kernel: do_syscall_64+0x7f/0x180
2026-05-19T10:59:47.628427+08:00 huamod-algorithm kernel: ? portSyncSpinlockAcquire+0x18/0x30 [nvidia]
2026-05-19T10:59:47.628429+08:00 huamod-algorithm kernel: ? check_heap_object+0x186/0x1e0
2026-05-19T10:59:47.628429+08:00 huamod-algorithm kernel: ? __task_pid_nr_ns+0x62/0xf0
2026-05-19T10:59:47.628431+08:00 huamod-algorithm kernel: ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
2026-05-19T10:59:47.628432+08:00 huamod-algorithm kernel: ? syscall_exit_to_user_mode+0x43/0x1e0
2026-05-19T10:59:47.628433+08:00 huamod-algorithm kernel: ? do_syscall_64+0x8c/0x180
2026-05-19T10:59:47.628434+08:00 huamod-algorithm kernel: ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
2026-05-19T10:59:47.628435+08:00 huamod-algorithm kernel: ? syscall_exit_to_user_mode+0x43/0x1e0
2026-05-19T10:59:47.628435+08:00 huamod-algorithm kernel: ? __task_pid_nr_ns+0x62/0xf0
2026-05-19T10:59:47.628436+08:00 huamod-algorithm kernel: ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
2026-05-19T10:59:47.628438+08:00 huamod-algorithm kernel: ? syscall_exit_to_user_mode+0x43/0x1e0
2026-05-19T10:59:47.628439+08:00 huamod-algorithm kernel: ? do_syscall_64+0x8c/0x180
2026-05-19T10:59:47.628440+08:00 huamod-algorithm kernel: ? update_process_times+0x8e/0xb0
2026-05-19T10:59:47.628441+08:00 huamod-algorithm kernel: ? __note_gp_changes+0x1c7/0x220
2026-05-19T10:59:47.628441+08:00 huamod-algorithm kernel: ? note_gp_changes+0x8f/0xa0
2026-05-19T10:59:47.628442+08:00 huamod-algorithm kernel: ? rseq_get_rseq_cs+0x22/0x280
2026-05-19T10:59:47.628445+08:00 huamod-algorithm kernel: ? rseq_ip_fixup+0x90/0x170
2026-05-19T10:59:47.628445+08:00 huamod-algorithm kernel: ? rcu_core_si+0xe/0x20
2026-05-19T10:59:47.628457+08:00 huamod-algorithm kernel: ? arch_exit_to_user_mode_prepare.isra.0+0x1a/0xe0
2026-05-19T10:59:47.628458+08:00 huamod-algorithm kernel: ? irqentry_exit_to_user_mode+0x38/0x1e0
2026-05-19T10:59:47.628458+08:00 huamod-algorithm kernel: ? irqentry_exit+0x43/0x50
2026-05-19T10:59:47.628459+08:00 huamod-algorithm kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80
2026-05-19T10:59:47.628460+08:00 huamod-algorithm kernel: RIP: 0033:0x7be161f24e1d
2026-05-19T10:59:47.628463+08:00 huamod-algorithm kernel: Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
2026-05-19T10:59:47.628464+08:00 huamod-algorithm kernel: RSP: 002b:00007be1599fc910 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
2026-05-19T10:59:47.628465+08:00 huamod-algorithm kernel: RAX: ffffffffffffffda RBX: 0000000000000020 RCX: 00007be161f24e1d
2026-05-19T10:59:47.628466+08:00 huamod-algorithm kernel: RDX: 00007be1599fcab0 RSI: 00000000c020462a RDI: 0000000000000009
2026-05-19T10:59:47.628466+08:00 huamod-algorithm kernel: RBP: 00007be1599fc960 R08: 00007be1599fcab0 R09: 00007be1599fcacc
2026-05-19T10:59:47.628467+08:00 huamod-algorithm kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007be1599fcab0
2026-05-19T10:59:47.628469+08:00 huamod-algorithm kernel: R13: 0000000000000009 R14: 00000000c020462a R15: 00007be1599fc980
2026-05-19T10:59:47.628470+08:00 huamod-algorithm kernel:
2026-05-19T10:59:47.628471+08:00 huamod-algorithm kernel: NVRM: tmrGetTimeEx_GH100: NVRM-RC: Consistently Bad TimeLo value ffffffff
2026-05-19T10:59:47.628472+08:00 huamod-algorithm kernel: NVRM: _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 0 heartbeat 4294967295 heartbeatWithOffsetMs 984666518 diff 3310300778 timeout 5200
2026-05-19T10:59:47.628473+08:00 huamod-algorithm kernel: NVRM: _kgspRpcRecvPoll: GSP RM heartbeat timed out
2026-05-19T10:59:47.628474+08:00 huamod-algorithm kernel: NVRM: tmrGetTimeEx_GH100: NVRM-RC: Consistently Bad TimeLo value ffffffff
2026-05-19T10:59:47.628476+08:00 huamod-algorithm kernel: NVRM: _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 0 heartbeat 4294967295 heartbeatWithOffsetMs 984666518 diff 3310300778 timeout 5200
2026-05-19T10:59:47.628477+08:00 huamod-algorithm kernel: NVRM: _kgspRpcRecvPoll: LibOS heartbeat timed out
2026-05-19T10:59:47.629427+08:00 huamod-algorithm kernel: NVRM: GPU at PCI:0000:61:00: GPU-22d18bcc-4df5-a2cd-e498-0ed9f545897f
2026-05-19T10:59:47.629431+08:00 huamod-algorithm kernel: NVRM: GPU Board Serial Number: 1790226111939
2026-05-19T10:59:47.629432+08:00 huamod-algorithm kernel: NVRM: Xid (PCI:0000:61:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)
2026-05-19T10:59:47.629433+08:00 huamod-algorithm kernel: NVRM: Xid (PCI:0000:91:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)
2026-05-19T10:59:48.886420+08:00 huamod-algorithm kernel: sched: RT throttling activated
2026-05-19T10:59:52.416441+08:00 huamod-algorithm kernel: nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
2026-05-19T10:59:57.416419+08:00 huamod-algorithm kernel: nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
2026-05-19T11:00:01.115424+08:00 huamod-algorithm kernel: NVRM: GPU1 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
2026-05-19T11:00:01.115449+08:00 huamod-algorithm kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
2026-05-19T11:00:01.115450+08:00 huamod-algorithm kernel: NVRM: GPU1 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
2026-05-19T11:00:01.115451+08:00 huamod-algorithm kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
2026-05-19T11:00:01.115462+08:00 huamod-algorithm kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
2026-05-19T11:00:01.115463+08:00 huamod-algorithm kernel: NVRM: GPU1 nvCheckFailedNoLog: Check failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ gpu_user_shared_data.c:503
2026-05-19T11:00:01.115463+08:00 huamod-algorithm kernel: NVRM: GPU1 nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from _gpushareddataSendDataPollRpc(pGpu, polledDataMask, pollingIntervalMs) @ gpu_user_shared_data.c:618
2026-05-19T11:00:01.115464+08:00 huamod-algorithm kernel: NVRM: GPU1 nvCheckFailedNoLog: Check failed: _handlePollMaskHelper(pGpu, NV_FALSE, NV_TRUE) == NV_OK @ gpu_user_shared_data.c:151
2026-05-19T11:00:02.416420+08:00 huamod-algorithm kernel: nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
2026-05-19T11:00:10.416969+08:00 huamod-algorithm systemd[1]: Starting sysstat-collect.service - system activity accounting tool…
2026-05-19T11:00:10.425385+08:00 huamod-algorithm systemd[1]: sysstat-collect.service: Deactivated successfully.
2026-05-19T11:00:10.425605+08:00 huamod-algorithm systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
2026-05-19T11:01:57.418425+08:00 huamod-algorithm kernel: message repeated 23 times: [ nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040]
2026-05-19T11:02:00.713433+08:00 huamod-algorithm kernel: nvitop[2822063]: segfault at 0 ip 00007a943e275bde sp 00007ffeb379c160 error 4 in libnvidia-ml.so.595.71.05[7a943e21f000+179000] likely on CPU 54 (core 32, socket 1)
2026-05-19T11:02:00.713456+08:00 huamod-algorithm kernel: Code: 38 e5 0b 00 59 5e 48 8d 65 e0 44 89 e0 5b 41 5c 41 5d 41 5e 5d c3 66 0f 1f 44 00 00 55 48 89 e5 41 54 53 48 89 f3 48 83 ec 10 <80> 3f 01 0f 84 91 00 00 00 48 83 7f 20 00 0f 84 86 00 00 00 8b 47
2026-05-19T11:02:00.786516+08:00 huamod-algorithm systemd[2401991]: Created slice session.slice - User Core Session Slice.
2026-05-19T11:02:00.808649+08:00 huamod-algorithm systemd[2401991]: Starting dbus.service - D-Bus User Message Bus…
2026-05-19T11:02:00.814077+08:00 huamod-algorithm dbus-daemon[2822070]: [session uid=1004 pid=2822070] AppArmor D-Bus mediation is enabled
2026-05-19T11:02:00.814254+08:00 huamod-algorithm systemd[2401991]: Started dbus.service - D-Bus User Message Bus.
2026-05-19T11:02:02.418425+08:00 huamod-algorithm kernel: nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
2026-05-19T11:05:01.658764+08:00 huamod-algorithm CRON[2831753]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2026-05-19T11:05:27.912781+08:00 huamod-algorithm systemd[1]: tmp-containerd\x2dmount1466862280.mount: Deactivated successfully.
2026-05-19T11:05:28.028128+08:00 huamod-algorithm systemd[1]: tmp-containerd\x2dmount133133779.mount: Deactivated successfully.
2026-05-19T11:06:20.485291+08:00 huamod-algorithm systemd[1]: var-lib-docker-tmp-buildkit\x2dmount2023254067.mount: Deactivated successfully.
2026-05-19T11:06:20.488924+08:00 huamod-algorithm systemd[1]: var-lib-docker-tmp-buildkit\x2dmount1697781622.mount: Deactivated successfully.
2026-05-19T11:06:20.518988+08:00 huamod-algorithm dockerd[82866]: time=“2026-05-19T11:06:20.518741285+08:00” level=warning msg=“forcibly turning on oci-mediatype mode for attestations” span=“exporting to image” spanID=060f18006c8500fd traceID=731b6674483fa291e6bc655e56a2aafe
2026-05-19T11:07:27.424425+08:00 huamod-algorithm kernel: message repeated 65 times: [ nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040]
2026-05-19T11:07:27.897451+08:00 huamod-algorithm kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND
2026-05-19T11:07:32.424426+08:00 huamod-algorithm kernel: nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
2026-05-19T11:10:00.409907+08:00 huamod-algorithm systemd[1]: Starting sysstat-collect.service - system activity accounting tool…
2026-05-19T11:10:00.419831+08:00 huamod-algorithm systemd[1]: sysstat-collect.service: Deactivated successfully.
2026-05-19T11:10:00.420023+08:00 huamod-algorithm systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
2026-05-19T11:12:47.434429+08:00 huamod-algorithm kernel: message repeated 63 times: [ nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040]
2026-05-19T11:12:47.897442+08:00 huamod-algorithm kernel: workqueue: drm_fb_helper_damage_work hogged CPU for >10000us 128 times, consider switching to WQ_UNBOUND
2026-05-19T11:12:52.434424+08:00 huamod-algorithm kernel: nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
2026-05-19T11:15:01.666921+08:00 huamod-algorithm CRON[2863980]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2026-05-19T11:16:37.588580+08:00 huamod-algorithm dockerd[82866]: time=“2026-05-19T11:16:37.588095598+08:00” level=error msg=“failed to validate image signature” error=“resolving signature chain for image sha256:622e78a1d02c0f90ed900e3985d6c975d8e2dc9ee5e61643aed587dcf9129f42: expected image index descriptor, got application/vnd.docker.distribution.manifest.list.v2+json”
2026-05-19T11:17:01.671825+08:00 huamod-algorithm CRON[2886372]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
2026-05-19T11:17:26.738373+08:00 huamod-algorithm systemd[1]: var-lib-docker-tmp-buildkit\x2dmount389010970.mount: Deactivated successfully.
2026-05-19T11:17:26.738840+08:00 huamod-algorithm systemd[1]: var-lib-docker-tmp-buildkit\x2dmount2207227484.mount: Deactivated successfully.
2026-05-19T11:17:22.439428+08:00 huamod-algorithm kernel: message repeated 54 times: [ nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040]
2026-05-19T11:17:26.802459+08:00 huamod-algorithm kernel: docker0: port 5(vethca912d6) entered blocking state
2026-05-19T11:17:26.802486+08:00 huamod-algorithm kernel: docker0: port 5(vethca912d6) entered disabled state
2026-05-19T11:17:26.802488+08:00 huamod-algorithm kernel: vethca912d6: entered allmulticast mode
2026-05-19T11:17:26.802489+08:00 huamod-algorithm kernel: vethca912d6: entered promiscuous mode
2026-05-19T11:17:26.807429+08:00 huamod-algorithm systemd-networkd[3633]: vethca912d6: Link UP
2026-05-19T11:17:26.900427+08:00 huamod-algorithm kernel: eth0: renamed from veth35e8081
2026-05-19T11:17:26.910327+08:00 huamod-algorithm systemd-networkd[3633]: vethca912d6: Gained carrier
2026-05-19T11:17:26.910426+08:00 huamod-algorithm kernel: docker0: port 5(vethca912d6) entered blocking state
2026-05-19T11:17:26.910432+08:00 huamod-algorithm kernel: docker0: port 5(vethca912d6) entered forwarding state
2026-05-19T11:17:27.439427+08:00 huamod-algorithm kernel: nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
2026-05-19T11:17:28.682961+08:00 huamod-algorithm systemd-networkd[3633]: vethca912d6: Gained IPv6LL
2026-05-19T11:18:45.879840+08:00 huamod-algorithm systemd[1]: var-lib-docker-buildkit-executor-m1uxdaffxqfik3uzr5v5qdgrg-rootfs.mount: Deactivated successfully.
2026-05-19T11:18:45.880288+08:00 huamod-algorithm systemd[1]: var-lib-docker-buildkit-containerd\x2doverlayfs-cachemounts-buildkit2306390335.mount: Deactivated successfully.
2026-05-19T11:18:45.892127+08:00 huamod-algorithm systemd-networkd[3633]: vethca912d6: Lost carrier
2026-05-19T11:18:42.440430+08:00 huamod-algorithm kernel: message repeated 15 times: [ nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040]
2026-05-19T11:18:45.892424+08:00 huamod-algorithm kernel: docker0: port 5(vethca912d6) entered disabled state
2026-05-19T11:18:45.892456+08:00 huamod-algorithm kernel: veth35e8081: renamed from eth0
2026-05-19T11:18:45.931966+08:00 huamod-algorithm systemd-networkd[3633]: vethca912d6: Link DOWN
2026-05-19T11:18:45.932412+08:00 huamod-algorithm kernel: docker0: port 5(vethca912d6) entered disabled state
2026-05-19T11:18:45.933418+08:00 huamod-algorithm kernel: vethca912d6 (unregistering): left allmulticast mode
2026-05-19T11:18:45.933424+08:00 huamod-algorithm kernel: vethca912d6 (unregistering): left promiscuous mode
2026-05-19T11:18:45.933425+08:00 huamod-algorithm kernel: docker0: port 5(vethca912d6) entered disabled state
2026-05-19T11:18:45.975936+08:00 huamod-algorithm systemd[1]: run-docker-netns-7c5ff4c328fa.mount: Deactivated successfully.
2026-05-19T11:18:46.609118+08:00 huamod-algorithm dockerd[82866]: time=“2026-05-19T11:18:46.608667243+08:00” level=warning msg=“forcibly turning on oci-mediatype mode for attestations” span=“exporting to image” spanID=b48166461f6adbfb traceID=1a65c4ac13ed2ceef3a0358b742e8d1d
2026-05-19T11:18:46.880355+08:00 huamod-algorithm systemd[1]: var-lib-docker-buildkit-containerd\x2doverlayfs-cachemounts-buildkit3015411010.mount: Deactivated successfully.
2026-05-19T11:18:46.880525+08:00 huamod-algorithm systemd[1]: var-lib-docker-tmp-buildkit\x2dmount3832337429.mount: Deactivated successfully.
2026-05-19T11:18:47.440429+08:00 huamod-algorithm kernel: nvidia-modeset: ERROR: GPU:1: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
2026-05-19T11:19:47.655874+08:00 huamod-algorithm systemd[1]: tmp-containerd\x2dmount3564235935.mount: Deactivated successfully.
2026-05-19T11:19:47.725643+08:00 huamod-algorithm systemd[1]: tmp-containerd\x2dmount305800565.mount: Deactivated successfully.
2026-05-19T11:20:03.494946+08:00 huamod-algorithm systemd[1]: Starting sysstat-collect.service - system activity accounting tool…
2026-05-19T11:20:03.502560+08:00 huamod-algorithm systemd[1]: sysstat-collect.service: Deactivated successfully.