GPU 0000:00:02.0: RmInitAdapter failed! (0x24:0x72:1556)

I am using an RTX4000 SFF Ada 20GB graphics card on i9 13900, with a driver of 550.107.02. However, I occasionally experience GPU loading failure when booting up, with only two logs reported
Sep 1 20:59:35 ubuntu kernel: [ 38.758464] NVRM: GPU 0000:00:02.0: RmInitAdapter failed! (0x24:0x72:1556)
Sep 1 20:59:35 ubuntu kernel: [ 38.786414] NVRM: GPU 0000:00:02.0: rm_init_adapter failed, device minor number 0
We have 20 machines, and only two of them have experienced this issue, so the probability of it happening is very small
I read open-gpu-kerne-modules-550 and found that the error code means,
RM_INIT_GPU_STATE_INIT_FAILED
NV_STATUS_CODE(NV_ERR_MEMORY_ERROR, 0x00000072, “Generic memory error”)
And I have confirmed that line 1556 should be at the following code location
if (pRmApi->Alloc(
pRmApi,
nv->rmapi.hClient,
nv->rmapi.hDevice,
&nv->rmapi.hSubDevice,
NV20_SUBDEVICE_0,
&subDeviceParams,
sizeof(subDeviceParams)) ! = NV_OK)
{
1556 goto fail;
}
But I see that the RM related code is closed source, and I want to know what caused the goto fail
Thank you