for more information, I found that all my tfrecords files are empty, but creating the tfrecords using the tlt-dataset-convert runs fine.
Converting Tfrecords for kitti trainval dataset
2021-08-23 02:08:53.074311: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Using TensorFlow backend.
2021-08-23 02:08:55,280 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2021-08-23 02:08:55,284 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Creating output directory /workspace/tlt-experiments/data/data/tfrecords/kitti_trainval
2021-08-23 02:08:55,308 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 0 Val: 0
2021-08-23 02:08:55,308 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2021-08-23 02:08:55,308 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
2021-08-23 02:08:55,308 - tensorflow - WARNING - From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
2021-08-23 02:08:55,322 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
2021-08-23 02:08:55,329 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
2021-08-23 02:08:55,337 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
2021-08-23 02:08:55,344 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
2021-08-23 02:08:55,344 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
2021-08-23 02:08:55,352 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
2021-08-23 02:08:55,359 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
2021-08-23 02:08:55,366 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
2021-08-23 02:08:55,373 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
2021-08-23 02:08:55,373 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2021-08-23 02:08:55,373 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
2021-08-23 02:08:55,374 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map.
Label in GT: Label in tfrecords file
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
2021-08-23 02:08:55,374 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.