Jetson Xavier crashes when running CNN.

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
#1

Hi, I’m running a recently Jetpack-flashed Xavier. Further I’ve downloaded Keras(2.1.6) with pip3, tensorflow-gpu (1.13.1+nv19.03). I’m then running a demo of a convolutional neural network (MaskRCNN) and when the script calls for the results of the network it works as it should until it reads in terminal:

date time: I tensorflow/stream_executor/sdo_loader.cc:153] sucessfully opened CUDA library libcublas.so.10.0 locally.

… then nothing happens for a brief second or so and then the Xavier shuts down.

It happened the same way multiple times. Trying to isolate the problem, I compiled some CUDA samples and they all ran well, I reinstalled CUDA just to be sure and the crash still happened. So my bet is that the fault is with tensorflow.

Other modifications done: installed a Samsung 970 Pro SSD harddisk and a Intel AC 7265 wifi-card.

Thanks in advance!

#2

Hi,

Could you help to run the model with CPU mode to see if works?

config = tf.ConfigProto(
        device_count = {'GPU': 0}
    )
sess = tf.Session(config=config)

Thanks.

#3

Hi,

I tried, but it still crashed.

Although I changed the nvpmodel from MAXN to 15W (-m 0->2) and then it ran the whole program through without crashing.

The problem might be power starvation or a broken core? The Xavier is powered by a Dell D6000 dock/hub which should deliver 19,5V and 6,7A.

Best regards!

#4

Hi,

As you said, a possible reason is the power starvation.
Could you try to unplug the hard disk and test the inference with MAXN mode again?

Thanks.