I am currently trying to use an industrial orin on jetpack 5.1.4. It appears that when I put the GPU under heavy load we are getting a crash in the low level driver resulting in the device becoming usable until a reboot has been performed.
I have seen that the system gets into this error state at a faster rate if we are on a power profile that unlocks more performance. E.g. if we are on the 60W profile it happens quicker than if running the 35W profile.
Any advise on how we can understand what the root cause if this will be appreciated. Along with any additional diagnostic steps. I have attached the dmesg log from the UART USB port. I have also been seeing instances where we get a double memory fault detected
Thank you for the information. We are currently on kernel 5.10.216-tegra. This is before that regression occurred. According to the linked post. Would this patch still apply? As its for a newer jetpack version
Also I have noticed that sometimes I get a double bit ECC fault on the unit. We are using an industrial orin.
From the UART log, the main failure appears to be a GPU context switch timeout:
Host pfifo ctxsw timeout error
ctxsw timeout error: active engine id =0, tsg=3, info: awaiting ack ms=3100
Since you are also seeing double-bit ECC faults, I would check whether an ECC event is reported immediately before these GPU timeouts. That could help determine whether the context-switch failure is the root cause or a consequence of a memory-related fault.
Best regards,
Fabian Munoz
Embedded SW Engineer at RidgeRun
I have inspected the system and I cannot see any ECC faults being reported. I am currently trying a newer version of Jetpack to see if the problem is still present. If so then I will attempt to make a working example that shows the fault. Thank you for the suggestion!