# GPU Lost when using Tensorflow Training

**URL:** <https://forums.developer.nvidia.com/t/gpu-lost-when-using-tensorflow-training/71226>\
**Category:** Linux\
**Created:** [March 6, 2019, 2:24pm UTC](https://forums.developer.nvidia.com/t/gpu-lost-when-using-tensorflow-training/71226 "2019-03-06T14:24:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![schr2718](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@schr2718](https://forums.developer.nvidia.com/u/schr2718)\
**Post date:** [March 6, 2019, 2:24pm UTC](https://forums.developer.nvidia.com/t/gpu-lost-when-using-tensorflow-training/71226/1 "2019-03-06T14:24:28Z")

</div>

Dear all,

I’m using two Tesla V100 cards in a Tyan FT77A-B7059 GPU server (PCIe 3.0, updated BIOS 1.05 which at least supports K80). While a test with gpu\_burn shows that the 250W electrical power is provided without problems, and airflow is sufficient to cool it to 82 degC, I run into trouble as soon as I train NN with Tensorflow (e.g. [DeepLearningFrameworks/Tensorflow\_MultiGPU.ipynb at master · ilkarman/DeepLearningFrameworks · GitHub](https://github.com/ilkarman/DeepLearningFrameworks/blob/master/notebooks/Tensorflow_MultiGPU.ipynb)):

nvidia-smi sporadically reports “GPU lost” / Xid 79 (see also attached nvidia-bug-log):

Mar 1 16:11:40 ml-comp kernel: NVRM: Xid (PCI:0000:89:00): 79, GPU has fallen off the bus.

I software reboot is not sufficient to bring back the card, but a full power down is needed. I tried the usual hints to change or replug power and PCI slots, but without success. Thermal problems shouldn’t be responsible, as only a temperature around 60-70 degC are reported during the incidence and gpu\_burn worked flawlessly.

I’m using CentOS 7.6, with Nvidia Driver 410.79, and Tensorflow 1.19 in a Docker Container using nvidia-docker2 runtime.

Thanks for any suggestion in advance! H.

UPDATE: limiting the power to 100W apparently avoids the problem as first test show…