Why couldn’t I convert my dataset to tfrecord for retail

malek206 · June 8, 2023, 1:42pm

Why couldn’t I convert my dataset to tfrecords 
 and retrain it using tao v4.0.2 with the retail-detector-binary.etlt model? What are the errors that appear in the following images?

Is this model suitable for detecting shelves and crowded products with high density?

Convert 1

Convert 2

Retrain

Space file retrain
data:
loader:
prefetch_size: 4
shuffle_file: True
max_instances_per_image: 100
image_size: ‘416x416’
num_classes: 46
train_tfrecords:
- "/home/jupyter/TAOTensorFlow/aa/train/.tfrecord’
val_tfrecords:
- '/home/jupyter/TAOTensorFlow/aa/valid/.tfrecord’
val_json_file: ‘/datasets/coco/annotations/instances_val2017.json’
train:
checkpoint:
“/home/jupyter/TAOTensorFlow/efficientdet-d5_046.tlt”
num_examples_per_epoch: 10000
batch_size: 8
model:
name: ‘efficientdet-d5’
evaluate:
batch_size: 8
num_samples: 5000
max_detections_per_image: 100
model_path: “/home/jupyter/TAOTensorFlow/efficientdet-d5_046.tlt”
export:
max_batch_size: 8
dynamic_batch_size: True
min_score_thresh: 0.4
model_path: “”
output_path: “”
inference:
model_path: “”
image_dir: “”
output_dir: “”
dump_label: False
batch_size: 1
prune:
model_path: “”
normalizer: ‘max’
output_path: “”
equalization_criterion: ‘union’
granularity: 8
threshold: 0.5
min_num_filters: 16
excluded_layers:
key: ‘nvidia_tlt’
results_dir: ‘/home/jupyter/TAOTensorFlow/Xout’

Space file convert datasets

dataset_convert: image_dir: ‘/workspace/tao-experiments/data/raw-data/train2017/’ annotations_file: ‘/workspace/tao-experiments/data/raw-data/annotations/instances_train2017.json’ output_dir: ‘/workspace/tao-experiments/data’ tag: ‘train’ num_shards: 256 include_masks: True

Morganh · June 9, 2023, 3:21am

According to your log, you are running efficientdet_tf2 network.

Its user guide is in EfficientDet (TF2) - NVIDIA Docs

For notebook, you can download it and run the efficientdet_tf2 notebook.

wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/tao/tao-getting-started/versions/4.0.2/zip -O getting_started_v4.0.2.zip
unzip -u getting_started_v4.0.2.zip  -d ./getting_started_v4.0.2 && rm -rf getting_started_v4.0.2.zip && cd ./getting_started_v4.0.2

Firstly, please check if you can run it successfully.

Also, only .tlt model can be set as the pretrained model. The .etlt model cannot be used to run training.

Please check if you still meet error when run dataset convert in the notebook.

malek206 · June 10, 2023, 3:41pm

Used nvcr.io/nvidia/tao/tao-toolkit:4.0.0-tf2.9.1 on vertex-ai

Convert.yaml

dataset_convert:
image_dir: ‘/home/jupyter/Data/train/’
annotations_file: ‘/home/jupyter/Data/train/_annotations.coco1.json’
output_dir: ‘/home/jupyter/Data’
tag: ‘train’
num_shards: 256
include_masks: True

Error

efficientdet_tf2 dataset_convert -e $SPECS_DIR/convert_train.yaml

Traceback (most recent call last): File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 211, in run_and_report return func() File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 368, in lambda: hydra.run( File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py”, line 87, in run cfg = self.compose_config( File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py”, line 564, in compose_config cfg = self.config_loader.load_configuration( File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py”, line 146, in load_configuration return self._load_configuration_impl( File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py”, line 233, in _load_configuration_impl self._process_config_searchpath(config_name, parsed_overrides, caching_repo) File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py”, line 162, in _process_config_searchpath loaded = repo.load_config(config_path=config_name) File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_repository.py”, line 337, in load_config ret = self.delegate.load_config(config_path=config_path) File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_repository.py”, line 91, in load_config ret = source.load_config(config_path=config_path) File "/usr/local/lib/python3.8/dist-packages/hydra/internal/core_plugins/file_config_source.py", line 31, in load_config cfg = OmegaConf.load(f) File “/usr/local/lib/python3.8/dist-packages/omegaconf/omegaconf.py”, line 186, in load obj = yaml.load(file, Loader=get_yaml_loader()) File “/usr/local/lib/python3.8/dist-packages/yaml/init.py”, line 81, in load return loader.get_single_data() File “/usr/local/lib/python3.8/dist-packages/yaml/constructor.py”, line 49, in get_single_data node = self.get_single_node() File “/usr/local/lib/python3.8/dist-packages/yaml/composer.py”, line 36, in get_single_node document = self.compose_document() File “/usr/local/lib/python3.8/dist-packages/yaml/composer.py”, line 55, in compose_document node = self.compose_node(None, None) File “/usr/local/lib/python3.8/dist-packages/yaml/composer.py”, line 84, in compose_node node = self.compose_mapping_node(anchor) File “/usr/local/lib/python3.8/dist-packages/yaml/composer.py”, line 133, in compose_mapping_node item_value = self.compose_node(node, item_key) File “/usr/local/lib/python3.8/dist-packages/yaml/composer.py”, line 84, in compose_node node = self.compose_mapping_node(anchor) File “/usr/local/lib/python3.8/dist-packages/yaml/composer.py”, line 127, in compose_mapping_node while not self.check_event(MappingEndEvent): File “/usr/local/lib/python3.8/dist-packages/yaml/parser.py”, line 98, in check_event self.current_event = self.state() File “/usr/local/lib/python3.8/dist-packages/yaml/parser.py”, line 438, in parse_block_mapping_key raise ParserError(“while parsing a block mapping”, self.marks[-1], yaml.parser.ParserError: while parsing a block mapping in “/home/jupyter/efficientdet_tf2/specs/convert_train.yaml”, line 2, column 3 expected , but found ‘’ in “/home/jupyter/efficientdet_tf2/specs/convert_train.yaml”, line 4, column 16 Sending telemetry data. Telemetry data couldn’t be sent, but the command ran successfully. [Error]: <urlopen error [Errno -2] Name or service not known> Execution status: FAIL

Morganh · June 11, 2023, 9:37am

Could you please save the full log to a txt file upload via the button?
Thanks.

malek206 · June 14, 2023, 4:56pm

The problem has been resolved after using this configuration space_retrain.ymal
The file is in the attachments
But the checkpoint not saved and finle file

This error : dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 500 batches). You may need to use the repeat() function when building your dataset. [Errno 2] No such file or directory: ‘R/weights/efficientdet-d5_001.tlt’ Error executing job with overrides: Traceback (most recent call last): File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”

spec_retrain.yaml (2.0 KB)

Morganh · June 15, 2023, 4:06pm

Could you elaborate the latest issue? More, could you upload the full log?

malek206 · June 24, 2023, 5:29pm

the problem:
No such file or directory: ‘R/weights/efficientdet-

the solution:
.tao_mounts.json
{
“Mounts”: [
{
“source”: “/home/jupyter/”,
“destination”: “/workspace/tao-experiments”
},
{
“source”: “/home/jupyter/efficientdet_tf2/specs”,
“destination”: “/home/jupyter/efficientdet_tf2/specs”
}
],
“DockerOptions”: {
“user”: “1000:1001”,
“network”: “host”
}
}

Thank you for helping us

Morganh · June 25, 2023, 6:34am

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Could you upload the full log? Thanks.

system · July 24, 2023, 6:11am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.