Retail Object Detection with EfficientDet - encryption key error

Accelerated Computing Intelligent Video Analytics TAO Toolkit
,
1

• Network Type:efficientdet-d5
• TLT Version: 5.5

Hi there,

I am trying to use Efficientdet-d5 pretrained model from Retail Object Detection Retail Object Detection | NVIDIA NGC.
But there is an issue when trying to load the file. TAO is not recognising the encryption key ‘nvidia_tlt’
With the following training spec:

train:
    checkpoint: "/workspace/tao-experiments/models/trainable_retail_object_detection_v1.1/efficientdet-d5_090.tlt"
….
encryption_key: 'nvidia_tlt'

I am getting the following error

File "<frozen cv.efficientdet.model.efficientdet_module>", line 58, in __init__
 File "<frozen cv.efficientdet.model.efficientdet_module>", line 117, in _load_pretrained_weights
 File "<frozen cv.efficientdet.utils.helper>", line 74, in decode_eff
 File "<frozen eff.core.file>", line 268, in get_handle
 File "<frozen eff.core.file>", line 209, in decrypt
 File "<frozen eff.core.file>", line -1, in check_decryption
PermissionError: The provided passphrase is invalid

I have tried the trainable model from v1.0 and it can load it correctly with the same encryption key.

Is there anything wrong on how the weights were encrypted?

Thanks

2

Can you confirm that you are using TAO5.5 docker to run training?
If yes, the training spec file is not needed to contain “encryption_key”. Because since TAO 5.0 version, the encryption is deprecated.

Reference spec file is in tao_tutorials/notebooks/tao_launcher_starter_kit/retail_object_detection/specs/train.yaml at main · NVIDIA/tao_tutorials · GitHub.

Suggest to use newer version of pretrained model which has .pth as extension.

3

Sorry for the delay, just getting back to this training.

Yes I can confirm I am using 5.5

docker run --rm --gpus all -v /mnt/rod_efs/:/workspace/tao-experiments nvcr.io/nvidia/tao/tao-toolkit:5.5.0-tf2 efficientdet_tf2 train -e /workspace/tao-experiments/tao/specs/train42.yaml results_dir=/workspace/tao-experiments/tao/results/training42/ num_gpus=4

=======================
=== TAO Toolkit TF2 ===
=======================

NVIDIA Release 5.5.0-TF2 (build 87841059)
TAO Toolkit Version 5.5.0

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

Any idea?

I have also tried with the pre-trained weights from trainable_binary_v1.0/efficientdet-d5_038.tlt but I am getting a graph execution error. See the logs attached.
efficentdet_d5_training_log.txt (232.0 KB)

Reference spec file is in tao_tutorials/notebooks/tao_launcher_starter_kit/retail_object_detection/specs/train.yaml at main · NVIDIA/tao_tutorials · GitHub.

Suggest to use newer version of pretrained model which has .pth as extension.

The config you have posted is using DINO, not EfficientDet. I am trying to train an test a smaller model than DINO.

4

Could you please run with
default notebook tao_tutorials/notebooks/tao_launcher_starter_kit/efficientdet_tf2/efficientdet.ipynb at main · NVIDIA/tao_tutorials · GitHub along with its default dataset? Is it successful?

More, for the key, could you please try nvidia_tao also?

5

I have changed the encryption key to nvidia-tao and the training could start.

You said:

If yes, the training spec file is not needed to contain “encryption_key”. Because since TAO 5.0 version, the encryption is deprecated.

For that particular pre-trained model it seems like you still need to specify it. Is it related to how the model was saved and or the TAO version?

I could start the training but I am getting the same execution error that I’ve posted above using pre-trained from trainable_binary_v1.0. I am attaching the logs to my reply.
Any ideas?
Thanks
efficentdet_d5_training_log2.txt (245.6 KB)

6

Looks like there is out-of-memory. Could you set a smaller batch-size and retry?

More, as requested previously, in TAO 5.5 docker, could you please run with default notebook tao_tutorials/notebooks/tao_launcher_starter_kit/efficientdet_tf2/efficientdet.ipynb at main · NVIDIA/tao_tutorials · GitHub along with its default dataset? Is it successful?

Yes, you can use old TAO 4.x version. nvcr.io/nvidia/tao/tao-toolkit:4.0.0-tf2.9.1. Also, please use 4.0 version notebooks/specs/doc.