• Network Type: Detectnet_v2
• Format_version: 2.0
• toolkit_version: 3.22.05
• published_date: 05/25/2022
• Training spec file:
detectnet_v2_train_resnet10_coco.txt (16.5 KB)
Hi there,
I want to train detectNet_v2 with resnet-10 arch. with a custom dataset. My dataset contains empty labels images, here is my dataset converter message:
Converting Tfrecords for coco trainval dataset
2022-12-14 14:01:11,739 [INFO] root: Registry: ['nvcr.io']
2022-12-14 14:01:11,809 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.4-py3
2022-12-14 14:01:11,816 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/root/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
2022-12-14 14:01:17,122 [INFO] iva.detectnet_v2.dataio.build_converter: Instantiating a coco converter
2022-12-14 14:01:17,122 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Creating output directory /workspace/tao-experiments/data/tfrecords/coco_trainval
loading annotations into memory...
Done (t=0.20s)
creating index...
index created!
2022-12-14 14:01:17,336 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 0
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/coco_converter_lib.py:136: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
2022-12-14 14:01:17,336 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/coco_converter_lib.py:136: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
2022-12-14 14:01:17,420 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 1
2022-12-14 14:01:17,501 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 2
2022-12-14 14:01:17,582 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 3
2022-12-14 14:01:17,664 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 4
2022-12-14 14:01:17,745 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 5
2022-12-14 14:01:17,826 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 6
2022-12-14 14:01:17,907 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 7
2022-12-14 14:01:17,990 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 8
2022-12-14 14:01:18,071 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 9
2022-12-14 14:01:18,154 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 10
2022-12-14 14:01:18,235 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 11
2022-12-14 14:01:18,317 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 12
2022-12-14 14:01:18,398 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 13
2022-12-14 14:01:18,478 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 14
2022-12-14 14:01:18,559 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 15
2022-12-14 14:01:18,641 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 16
2022-12-14 14:01:18,722 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 17
2022-12-14 14:01:18,804 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 18
2022-12-14 14:01:18,884 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 19
2022-12-14 14:01:18,965 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 20
2022-12-14 14:01:19,046 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 21
2022-12-14 14:01:19,126 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 22
2022-12-14 14:01:19,207 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 23
2022-12-14 14:01:19,288 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 24
2022-12-14 14:01:19,370 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 25
2022-12-14 14:01:19,452 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 26
2022-12-14 14:01:19,533 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 27
2022-12-14 14:01:19,614 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 28
2022-12-14 14:01:19,695 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 29
2022-12-14 14:01:19,777 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 30
2022-12-14 14:01:19,858 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 31
2022-12-14 14:01:19,947 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b'tsa': 3361
b'peo': 1766
b'sgn': 5319
b'jp6': 4982
b'vad': 1574
b'tel': 6844
b'sld': 218
b'tdb': 2145
b'xng': 2184
b'vov': 803
b'sig': 672
b'von': 4210
b'ash': 1375
2022-12-14 14:01:19,947 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Cumulative object statistics
2022-12-14 14:01:19,947 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b'tsa': 3361
b'peo': 1766
b'sgn': 5319
b'jp6': 4982
b'vad': 1574
b'tel': 6844
b'sld': 218
b'tdb': 2145
b'xng': 2184
b'vov': 803
b'sig': 672
b'von': 4210
b'ash': 1375
2022-12-14 14:01:19,947 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Class map.
Label in GT: Label in tfrecords file
TSA: tsa
PEO: peo
SGN: sgn
JP6: jp6
VAD: vad
TEL: tel
SLD: sld
TDB: tdb
XNG: xng
VOV: vov
SIG: sig
VON: von
ASH: ash
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
2022-12-14 14:01:19,947 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Tfrecords generation complete.
2022-12-14 14:01:19,947 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing the log_warning.json
2022-12-14 14:01:19,947 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: There were errors in the labels. Details are logged at /workspace/tao-experiments/data/tfrecords/coco_trainval/coco_trainval_waring.json
2022-12-14 14:01:20,849 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
I get this error:
There were errors in the labels. Details are logged at /workspace/tao-experiments/data/tfrecords/coco_trainval/coco_trainval_waring.json
coco_trainval_waring.json contains :
{
"6658": [
384,
306,
390,
306
]
}
therefore I get an error while running train command.
I get this error:
2022-12-14 14:54:42,732 [INFO] root: Registry: ['nvcr.io']
2022-12-14 14:54:42,803 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.4-py3
2022-12-14 14:54:42,810 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/root/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.
2022-12-14 14:54:48,031 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
2022-12-14 14:54:48,130 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
2022-12-14 14:54:48,610 [INFO] iva.common.logging.logging: Log file already exists at /workspace/tao-experiments/USER_EXPERIMENT_detectnet_v2/experiment_dir_unpruned/status.json
2022-12-14 14:54:48,610 [INFO] root: Starting DetectNet_v2 Training job
2022-12-14 14:54:48,610 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet10_coco.txt.
2022-12-14 14:54:48,611 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet10_coco.txt
2022-12-14 14:54:48,626 [INFO] root: Training gridbox model.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2022-12-14 14:54:48,626 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 917, in <module>
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 906, in <module>
File "<decorator-gen-2>", line 2, in main
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/utilities/timer.py", line 46, in wrapped_fn
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 893, in main
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 757, in run_experiment
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 626, in train_gridbox
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataloader/build_dataloader.py", line 273, in build_dataloader
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataloader/drivenet_dataloader.py", line 491, in __init__
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataloader/drivenet_dataloader.py", line 548, in _construct_data_sources
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataloader/drivenet_dataloader.py", line 395, in __init__
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataloader/drivenet_dataloader.py", line 395, in <listcomp>
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataloader/drivenet_dataloader.py", line 394, in <genexpr>
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/lib/io/tf_record.py", line 181, in tf_record_iterator
reader.GetNext()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/pywrap_tensorflow_internal.py", line 1034, in GetNext
return _pywrap_tensorflow_internal.PyRecordReader_GetNext(self)
tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 0
2022-12-14 14:54:49,685 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
When I delete the warning.json file in tfrecords folder and run the training command again I get this error:
2022-12-14 14:55:52,930 [INFO] root: Registry: ['nvcr.io']
2022-12-14 14:55:53,001 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.4-py3
2022-12-14 14:55:53,008 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/root/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.
2022-12-14 14:55:58,258 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
2022-12-14 14:55:58,358 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
2022-12-14 14:55:58,813 [INFO] iva.common.logging.logging: Log file already exists at /workspace/tao-experiments/USER_EXPERIMENT_detectnet_v2/experiment_dir_unpruned/status.json
2022-12-14 14:55:58,813 [INFO] root: Starting DetectNet_v2 Training job
2022-12-14 14:55:58,814 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet10_coco.txt.
2022-12-14 14:55:58,814 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet10_coco.txt
2022-12-14 14:55:58,828 [INFO] root: Training gridbox model.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2022-12-14 14:55:58,828 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2022-12-14 14:55:59,627 [INFO] root: Sampling mode of the dataloader was set to user_defined.
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 917, in <module>
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 906, in <module>
File "<decorator-gen-2>", line 2, in main
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/utilities/timer.py", line 46, in wrapped_fn
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 893, in main
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 757, in run_experiment
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py", line 638, in train_gridbox
File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py", line 26, in build_cost_auto_weight_hook
ValueError: steps_per_epoch must be > 0
2022-12-14 14:56:00,663 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
Does the TAO toolkit accept empty label images (background images) or do I have to use non-empty label images?