• Hardware A6000
• Network Type efficientdet_tf1, efficientdet_tf2
• TLT Version TAO 5.0.0
format_version: 3.0
toolkit_version: 5.0.0
published_date: 07/14/2023
I am testing Object Detection models after TAO 5.0 update. I successfully trained Deformable DETR in coco format with my custom dataset. Similarly, I tried to train Efficientdet model with the same training data, but the following error occurred during dataset_convert to tfrecords.
I followed the efficientdet notebook from tao-getting-started
The cell where the error occurred is shown below.
# convert training data to TFRecords
!tao model efficientdet_tf2 dataset_convert -e $SPECS_DIR/spec_train.yaml \
dataset_convert.results_dir=$DATA_DOWNLOAD_DIR
The error is as follows
2023-07-31 22:22:58,832 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
2023-07-31 22:22:58,928 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf2.11.0
2023-07-31 22:22:58,978 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 275: Printing tty value True
2023-07-31 13:23:00.044540: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[1690809782.736606] [dgx-a100:44 :f] vfs_fuse.c:424 UCX WARN failed to connect to vfs socket '������������������������������': Invalid argument
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-ui7ofeui because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
<frozen cv.efficientdet.scripts.dataset_convert>:345: UserWarning:
'spec_train.yaml' is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
Dataset_convert results will be saved at: /workspace/tao-experiments/data
Log file already exists at /workspace/tao-experiments/data/status.json
Starting efficientdet data conversion.
INFO:tensorflow:writing to output path: /workspace/tao-experiments/data/train
writing to output path: /workspace/tao-experiments/data/train
INFO:tensorflow:Building bounding box index.
Building bounding box index.
INFO:tensorflow:0 images are missing bboxes.
0 images are missing bboxes.
segmentation groundtruth is missing in object: 5.
Error executing job with overrides: ['dataset_convert.results_dir=/workspace/tao-experiments/data']
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "<frozen cv.efficientdet.scripts.dataset_convert>", line 218, in _pool_create_tf_example
File "<frozen cv.efficientdet.scripts.dataset_convert>", line 134, in create_tf_example
ValueError: segmentation groundtruth is missing in object: 5.
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 368, in <lambda>
lambda: hydra.run(
File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 110, in run
_ = ret.return_value
File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "<frozen cv.efficientdet.scripts.dataset_convert>", line 341, in main
File "<frozen common.decorators>", line 88, in _func
File "<frozen common.decorators>", line 61, in _func
File "<frozen cv.efficientdet.scripts.dataset_convert>", line 325, in run_conversion
File "<frozen cv.efficientdet.scripts.dataset_convert>", line 282, in _create_tf_record_from_coco_annotations
File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "<frozen cv.efficientdet.scripts.dataset_convert>", line 218, in _pool_create_tf_example
File "<frozen cv.efficientdet.scripts.dataset_convert>", line 134, in create_tf_example
ValueError: segmentation groundtruth is missing in object: 5.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "</usr/local/lib/python3.8/dist-packages/nvidia_tao_tf2/cv/efficientdet/scripts/dataset_convert.py>", line 3, in <module>
File "<frozen cv.efficientdet.scripts.dataset_convert>", line 345, in <module>
File "<frozen common.hydra.hydra_runner>", line 99, in wrapper
File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 367, in _run_hydra
run_and_report(
File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 251, in run_and_report
assert mdl is not None
AssertionError
The error seems to require segmentation data from the annotation. But my dataset is Object Detection, so I only have information about the bboxes. Why does efficientdet ask for segmentation data?
Below is an excerpt from my annotation json file.
{
"annotations": [
{
"id": 5,
"image_id": 4,
"category_id": 1,
"bbox": [
57,
37,
48,
46
],
"area": 0,
"iscrowd": 0
},
{
"id": 6,
"image_id": 4,
"category_id": 2,
"bbox": [
72.15,
3.08,
29.24,
33.54
],
"area": 0,
"iscrowd": 0
}
]
}
Below is the dataset_convert config portion of spec_train.json.
dataset_convert:
image_dir: '/workspace/tao-experiments/data/raw-data/image/train/'
annotations_file: '/workspace/tao-experiments/data/raw-data/annotations/train.json'
results_dir: '/workspace/tao-experiments/data'
tag: 'train'
num_shards: 256
include_masks: True