Titan X reboot automatically on Ubuntu

Hi everyone,

I encountered a really terrible thing recently which almost made me crazy.

I try to use DIGIT for deep learning framework with two Titan X equipped in my computer. However, when I run the training program, the GPU reboot automatically. Almost every time!!!

I chosen the LeNet model (published on 1998) for classification.

Before program crashed, I saw the memory utlised rate is 95%. Is it normal? Titan X has 12GB global memory, while I bet the model is not so large.

Did anyone encounter similar problem or can figure out what happened? I don’t think this is a software problem.

BTW, I don’t use 2-way SLI to connect two GPUs together. Is 2-way SLI required for building multi-GPUs system via DIGIT?

Thanks!

What do you mean with “the GPU reboot automatically”? Does your entire system reboot? That could be a Power Supply issue. Running 2 Titan X draws a lot of juice when they are working at full load. What kind of Power Supply do you use?

Sorry, I may use some words causing misunderstanding. “the GPU reboot automatically” means only GPU shutdown and restart itself instead of whole system. I am not sure whether the entire system reboot. If I access the machine via ssh, the pipe will be broken.

I use power supply which can provide 500W power inside the machine.

Thanks!

500 W isn’t even really enough for one card let alone a Titan X, especially if its a cheap low efficiency noname Chinese power supply.

Get a 1000W PSU from a decent manufacturer like beQuiet!, Corsair etc. I’m 99% certain that this will fix your problem.

Thanks for tips!

I tried to use only one Titan X, but still can’t solve this problem. Do you know the exact power needed by one Titan X?

I will try to use other power supply.

http://www.geforce.com/hardware/desktop-gpus/geforce-gtx-titan-x/specifications

1000 W PSUs are generally recommended for SLI and gives you enough headroom if you want to overclock and add more devices.

http://www.realhardtechx.com/index_archivos/Page362.htm

A single Titan X can draw up to 320 W alone.
http://www.extremetech.com/extreme/201346-nvidia-geforce-gtx-titan-x-reviewed-crushing-the-single-gpu-market/4

Thank you for your help. I am trying larger power supply now!

Don’t only use a larger PSU also make sure to use one with high efficiency.