Conversion of KITTI to TFRecords failing due to ImportError: /usr/lib/x86_64-linux-gnu/libcuda.so.1: file too short

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I am trying to convert my training dataset from KITTI to TFRecords using the tlt-dataset-convert
tool and I am getting the following error:

$ tlt-dataset-convert -d tfrecords_kitti_train.txt -o tfrecords/kitti_trainval/kitti_trainval
Traceback (most recent call last):
  File "/usr/local/bin/tlt-dataset-convert", line 6, in <module>
    from iva.detectnet_v2.scripts.dataset_convert import main
  File "./detectnet_v2/scripts/dataset_convert.py", line 14, in <module>
  File "./detectnet_v2/dataio/build_converter.py", line 13, in <module>
  File "./detectnet_v2/dataio/kitti_converter_lib.py", line 16, in <module>
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: /usr/lib/x86_64-linux-gnu/libcuda.so.1: file too short


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

Can anyone suggest what may be causing this issue and/or how to solve or work around it? Thanks in advance…

2

Hi monocongo,
Can you trigger detectnet_v2 Jupyter notebook inside TLT docker to cross check?
In TLT released notebook, they use KITTI dataset.
To narrow down your issue,you can run the notebook to see if “tlt-dataset-convert” runs well.

3

Hi monocongo,

Please make sure you are using NVIDIA docker 2 and use nvidia-docker run(not docker run) to start the docker.

Thanks
Zhimeng

4

Thanks for the suggestions/help. My issue was that I was running an out-of-date version of Docker rather than 19.03 which includes the NVIDIA runtime. Updating the docker package fixed this.