Crash driver CUDA of GeForce RTX 4090

Gaming and Visualization Technologies General Topics and Other SDKs
1

I have two Geforce RTX 4090 on one motherboard. I’m running two processes for training Tensorflow models on different maps. After a while I get an error on the first video card (GPU:0). I see an error in the Windows system log - nvlddmkm (\Device\Video13. Error occurred on GPUID : 100). Also i see in Windows Application logs next error - ucrtbase.dll. How to fix this error?
Driver Version : 537.34
CUDA Version : 12.2
TensorFlow Version: 2.10.0
OS: Win10x64 Pro

2

The error was reproduced again. Here is the window with the exception. Can anyone help me?

Ошибка Cuda
Ошибка Cuda1936×1080 180 KB