Error when using tlt-dataset-convert

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I am trying to train a SSD model using TLT using the notebook: /workspace/examples/ssd/ssd.ipynb.

When I try to generate TFrecords using the command:
!tlt-dataset-convert -d $SPECS_DIR/ssd_tfrecords_kitti_trainval.txt \ -o $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval/kitti_trainval
the following output comes:
`Using TensorFlow backend.
2020-07-20 14:15:56,069 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2020-07-20 14:15:56,083 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 6434 Val: 1047
2020-07-20 14:15:56,083 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2020-07-20 14:15:56,084 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:266: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2020-07-20 14:15:56,166 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
2020-07-20 14:15:56,246 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
2020-07-20 14:15:56,324 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
2020-07-20 14:15:56,404 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4
2020-07-20 14:15:56,484 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5
2020-07-20 14:15:56,564 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6
2020-07-20 14:15:56,643 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7
2020-07-20 14:15:56,720 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8
2020-07-20 14:15:56,800 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9
2020-07-20 14:15:56,884 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
cyclist: 223
van: 394
tram: 65
car: 3925
misc: 135
pedestrian: 657
truck: 173
person_sitting: 29
dontcare: 1619

2020-07-20 14:15:56,885 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
2020-07-20 14:15:57,372 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
2020-07-20 14:15:57,863 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
2020-07-20 14:15:58,348 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
2020-07-20 14:15:58,841 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4
2020-07-20 14:15:59,331 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5
2020-07-20 14:15:59,820 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6
2020-07-20 14:16:00,315 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7
2020-07-20 14:16:00,801 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8
2020-07-20 14:16:01,286 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
2020-07-20 14:16:01,775 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
cyclist: 1404
van: 2520
tram: 446
car: 24817
misc: 838
pedestrian: 3830
truck: 921
person_sitting: 193
dontcare: 9676

2020-07-20 14:16:01,775 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2020-07-20 14:16:01,775 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
cyclist: 1627
van: 2914
tram: 511
car: 28742
misc: 973
pedestrian: 4487
truck: 1094
person_sitting: 222
dontcare: 11295

2020-07-20 14:16:01,775 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map.
Label in GT: Label in tfrecords file
Cyclist: cyclist
Van: van
Tram: tram
Car: car
Misc: misc
Pedestrian: pedestrian
Truck: truck
Person_sitting: person_sitting
DontCare: dontcare
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2020-07-20 14:16:01,775 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.`

When I check the output directory, all the files that are supposed to be present are created(kitti_trainval-fold-000-of-002-shard-00000-of-00010 … kitti_trainval-fold-001-of-002-shard-00009-of-00010). But each file contains the following error message:

Screenshot from 2020-07-20 20-41-08
Screenshot from 2020-07-20 20-41-081517×155 21.2 KB

3

You mentioned that “I check the output directory”. Could you share the command how did you check?
More, you are generate the tfrecords for public KITTI dataset, right?

4

I opened the file using jupyter.
Yes, I was trying to generate TFrecords for KITTI.

Moreover, I think, the problem was only in displaying the file on jupyter & not in creating the TFrecords files. Because:

  1. when I downloaded(using jupyter) one of the tfrecords file and viewed its properties, it was shown as a binary file.
  2. And, the size of each TFrecords file is unique.
  3. There was no problem in training. I successfully trained the model for 2 epochs.

Thanks for your prompt reply!!