Failed to load the native TensorFlow runtime.

Trying to convert kitti dataset to tfrecords.

command used:

!tlt-dataset-convert -d spec_files/convert.txt -o /tf_records

or

!tlt-dataset-convert -d /workspace/spec_files/convert.txt -o /workspace/tf_records/

and many other path combinations.

The spec_file being used:

kitti_config {
  root_directory_path: "/workspace/dataset/KITTI_original/training"
  image_dir_name: "image_2"
  label_dir_name: "label_2"
  image_extension: ".png"
  partition_mode: "random"
  num_partitions:2
  val_split: 15
  num_shards: 10
}
image_directory_path: "/workspace/dataset/KITTI_original/training"

Error:

Traceback (most recent call last):
  File "/usr/local/bin/tlt-dataset-convert", line 6, in <module>
    from iva.detectnet_v2.scripts.dataset_convert import main
  File "./detectnet_v2/scripts/dataset_convert.py", line 14, in <module>
  File "./detectnet_v2/dataio/build_converter.py", line 13, in <module>
  File "./detectnet_v2/dataio/kitti_converter_lib.py", line 16, in <module>
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

trying to follow this notebook, https://github.com/NVIDIA-AI-IOT/tlt-iva-examples/blob/master/TLT_notebook.ipynb

kindly look into it.

Environment details

DGX version

grep VERSION /etc/dgx-release
DGX_SWBUILD_VERSION="3.1.4"
DGX_OTA_VERSION="3.1.7"
DGX_OTA_VERSION="3.1.8"
PATH	/opt/nvidia/tools:/opt/ngccli:/usr/local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CUDA_VERSION	10.0.130
CUDA_PKG_VERSION	10-0=10.0.130-1
LD_LIBRARY_PATH	/usr/local/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NVIDIA_VISIBLE_DEVICES	all
NVIDIA_DRIVER_CAPABILITIES	compute,utility
NVIDIA_REQUIRE_CUDA	cuda>=10.0 brand=tesla,driver>=384,driver<385 brand=tesla,driver>=410,driver<411
NCCL_VERSION	2.4.2
LIBRARY_PATH	/usr/local/cuda/lib64/stubs
MOFED_VERSION	3.4-1.0.0.0
TRT_LIB_PATH	/usr/lib/x86_64-linux-gnu
TRT_INCLUDE_PATH	/usr/include/x86_64-linux-gnu
PYTHONPATH	/workspace/examples:/opt/nvidia/third_party/nvml:/opt/nvidia:

I found the solution here! sorry for raising new topic.
https://devtalk.nvidia.com/default/topic/1065003/transfer-learning-toolkit/conversion-of-kitti-to-tfrecords-failing-due-to-importerror-usr-lib-x86_64-linux-gnu-libcuda-so-1-file-too-short/post/5392892/#5392892