Empty TFRecords Being created From to detectnet_v2 dataset convert

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware RTX
• Network Type : Detectnet_v2
• TLT Version dockers:
nvidia/tao/tao-toolkit-tf:
v3.21.11-tf1.15.5-py3:
• Training spec file
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

Issue : getting empty TFRecords.

Log:

tao detectnet_v2 dataset_convert                   -d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt                   -o $DATA_DIR/tfrecords/kitti_trainval/kitti_trainval
2022-02-12 17:37:03,802 [INFO] root: Registry: ['nvcr.io']
2022-02-12 17:37:03,849 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.4-py3
2022-02-12 17:37:03,926 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/sitevisit/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2022-02-12 12:07:08,804 [INFO] iva.detectnet_v2.dataio.build_converter: Instantiating a kitti converter
2022-02-12 12:07:08,804 [INFO] root: Instantiating a kitti converter
2022-02-12 12:07:08,804 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Creating output directory /workspace/tao-experiments/data/tfrecords/kitti_trainval
2022-02-12 12:07:08,804 [INFO] root: Generating partitions
2022-02-12 12:07:08,805 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Num images in
Train: 0	Val: 0
2022-02-12 12:07:08,805 [INFO] root: Num images in
Train: 0	Val: 0
2022-02-12 12:07:08,805 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-02-12 12:07:08,805 [INFO] root: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-02-12 12:07:08,805 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Cumulative object statistics
2022-02-12 12:07:08,805 [INFO] root: Cumulative object statistics
2022-02-12 12:07:08,805 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:

2022-02-12 12:07:08,805 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Class map. 
Label in GT: Label in tfrecords file 
2022-02-12 12:07:08,805 [INFO] root: Class map. 
Label in GT: Label in tfrecords file 
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-02-12 12:07:08,805 [INFO] root: For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-02-12 12:07:08,805 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Tfrecords generation complete.
2022-02-12 12:07:08,805 [INFO] root: TFRecords generation complete.
2022-02-12 17:37:09,610 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

I resized my images to the size 1280 x 960 as I thought it might be a size related error, but didn’t help.
Same command and combination of spec file with different image-label set produces valid tfrecords. What am I doing wrong here?

2

There are 0 images.

Please check if the path is correct.
Please note that the path should be the a path inside the docker. Please check the ~/.tao_mounts.json

3

Morgan, I have checked those things. Mounts is fine.

When I place a different dataset in the same path, it creates the TFRecords.

Is there something specific to image or label files that could make it so that images are not considered/detected?

4

You can check the spec file.
tao detectnet_v2 run cat $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt

More, please check if there is access issue for your dataset.

5

I’ve checked the spec file and it checks out. What’s more, when I use the same spec file with a different dataset, it is able to generate TFRecords.

I didn’t change the path in my spec file, I just placed my new dataset at the same location and the TFRecords turn up blank.

6

Can you share the detectnet_v2_tfrecords_kitti_trainval.txt ?

7

Dear Morgan,
I managed to solve the issue myself. It was a mismatch in the image extensions.
Thanks

8

Great. Thanks for the info.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.