TLT fail to init a cudnn with a RTX GPU, cannot fix without source

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

while running "tlt-evaluate detectnet_v2 " ,failed with following output:

2020-04-28 10:13:55.945423: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2020-04-28 10:13:56.900110: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-04-28 10:13:56.901303: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

We can fix it with this code
tf.config.experimental.set_memory_growth(gpu, True)

but I cannot find source for tlt tools

3

Could you please

$ export TF_FORCE_GPU_ALLOW_GROWTH=true

and retry?

1 Like
4

Thanks, FIXED