Empty TFRecords Being created From to detectnet_v2 dataset convert

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

• Hardware: RTX 3050Ti
• Network Type: Detectnet_v2
• TLT Version :

Configuration of the TAO Toolkit Instance
dockers: ['nvidia/tao/tao-toolkit-tf', 'nvidia/tao/tao-toolkit-pyt', 'nvidia/tao/tao-toolkit-lm']
format_version: 2.0
toolkit_version: 3.22.05
published_date: 05/25/2022

Issue : obtaining empty TFRecords!!!

 !tao detectnet_v2 dataset_convert \
                  -d /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_tfrecords_kitti_trainval.txt \
                  -o /home/diva/Desktop/cv_samples_v1.4.1/data/tfrecords/kitti_trainval/

2022-10-19 15:03:32,755 [INFO] root: Registry: ['nvcr.io']
2022-10-19 15:03:32,797 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.4-py3
2022-10-19 15:03:32,804 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/diva/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Using TensorFlow backend.
2022-10-19 13:03:37,554 [INFO] iva.detectnet_v2.dataio.build_converter: Instantiating a kitti converter
2022-10-19 13:03:37,554 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Creating output directory /home/diva/Desktop/cv_samples_v1.4.1/data/tfrecords/kitti_trainval
2022-10-19 13:03:37,567 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Num images in
Train: 6434	Val: 1047
2022-10-19 13:03:37,567 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-10-19 13:03:37,571 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 0
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:169: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

2022-10-19 13:03:37,571 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:169: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:315: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2022-10-19 13:03:37,637 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 1
2022-10-19 13:03:37,701 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 2
2022-10-19 13:03:37,764 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 3
2022-10-19 13:03:37,828 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 4
2022-10-19 13:03:37,890 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 5
2022-10-19 13:03:37,951 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 6
2022-10-19 13:03:38,014 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 7
2022-10-19 13:03:38,078 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 8
2022-10-19 13:03:38,142 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 9
2022-10-19 13:03:38,208 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'car': 3976
b'misc': 130
b'dontcare': 1544
b'pedestrian': 539
b'van': 431
b'cyclist': 229
b'tram': 77
b'truck': 162
b'person_sitting': 42

2022-10-19 13:03:38,208 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 0
2022-10-19 13:03:38,595 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 1
2022-10-19 13:03:38,986 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 2
2022-10-19 13:03:39,372 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 3
2022-10-19 13:03:39,763 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 4
2022-10-19 13:03:40,150 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 5
2022-10-19 13:03:40,545 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 6
2022-10-19 13:03:40,933 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 7
2022-10-19 13:03:41,324 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 8
2022-10-19 13:03:41,720 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 9
2022-10-19 13:03:42,111 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'car': 24766
b'tram': 434
b'dontcare': 9751
b'van': 2483
b'truck': 932
b'pedestrian': 3948
b'cyclist': 1398
b'person_sitting': 180
b'misc': 843

2022-10-19 13:03:42,111 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Cumulative object statistics
2022-10-19 13:03:42,111 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'car': 28742
b'misc': 973
b'dontcare': 11295
b'pedestrian': 4487
b'van': 2914
b'cyclist': 1627
b'tram': 511
b'truck': 1094
b'person_sitting': 222

2022-10-19 13:03:42,111 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Class map. 
Label in GT: Label in tfrecords file 
b'Car': b'car'
b'Misc': b'misc'
b'DontCare': b'dontcare'
b'Pedestrian': b'pedestrian'
b'Van': b'van'
b'Cyclist': b'cyclist'
b'Tram': b'tram'
b'Truck': b'truck'
b'Person_sitting': b'person_sitting'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-10-19 13:03:42,111 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Tfrecords generation complete.
2022-10-19 15:03:42,740 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

This issue is similar to Empty TFRecords Being created From to detectnet_v2 dataset convert, but the proposed solution does not fit my issue!!
Could you please provide some directions to explore!

2

Refer to Error when training detectnet_v2 resnet34 on tfrecord file - #7 by Morganh in order to map local folder to docker correctly.

3

Thank you for the reference,

!tao detectnet_v2 dataset_convert \
                  -d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt \
                  -o $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval

!cat ~/.tao_mounts.json

{
    "Mounts": [
        {
            "source": "/home/diva/Desktop/cv_samples_v1.4.1/detectnet_v2/data",
            "destination": "/workspace/tao-experiments/data"
        },
        {
            "source": "/home/diva/Desktop/cv_samples_v1.4.1/detectnet_v2/specs",
            "destination": "/workspace/tao-experiments/specs"
        }
    ]
}

I am not sure what I am mistaken here.

4

Because the folder /home/diva is not available inside the docker.
According to tao_mounts.json, you already set to something like /workspace/tao-experiments.

For ease, please set destination to the same as source.

{
    "Mounts": [
        {
            "source": "/home/diva/Desktop/cv_samples_v1.4.1/detectnet_v2/data",
            "destination": "/home/diva/Desktop/cv_samples_v1.4.1/detectnet_v2/data"
        },
        {
            "source": "/home/diva/Desktop/cv_samples_v1.4.1/detectnet_v2/specs",
            "destination": "/home/diva/Desktop/cv_samples_v1.4.1/detectnet_v2/specs"
        }
    ]
}
5

The issue is solved. I do confirm that it was related to the mapping. Thank you @Morganh for your guidance.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.