BYOM model trainer with Tao 5.5 toolkit

Hello, While using the TAO Toolkit to train the classification_tf2 model using byom approach, I am having following error. This error is not present if I train the model starting from a saved model weight file from the efficientnet_b0 file. However, when I try to have the byom pathway, I am having an error at the launch of
!tao model classification_tf2 train -e $SPECS_DIR/spec.yaml

cannot import name ‘conv_utils’ from ‘keras.utils’ (/usr/local/lib/python3.10/dist-packages/keras/utils/init.py)
Error executing job with overrides: Traceback (most recent call last):
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 213, in run_and_report
return func()
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py”, line 453, in
lambda: hydra.run(
File “/usr/local/lib/python3.10/dist-packages/clearml/binding/hydra_bind.py”, line 91, in _patched_hydra_run
return PatchHydra._original_hydra_run(self, config_name, task_function, overrides, *args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py”, line 132, in run
_ = ret.return_value
File “/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py”, line 260, in return_value
raise self._return_value
File “/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py”, line 186, in run_job
ret.return_value = task_function(task_cfg)
File “/usr/local/lib/python3.10/dist-packages/clearml/binding/hydra_bind.py”, line 230, in _patched_task_function
return task_function(a_config, *a_args, **a_kwargs)
File “”, line 217, in main
File “”, line 93, in _func
File “”, line 60, in _func
File “”, line 178, in run_experiment
File “”, line 42, in init
File “”, line 86, in _build_models
File “”, line 579, in get_model
File “”, line 118, in get_byom
File “”, line 345, in decode_tltb
File “”, line 322, in deserialize_custom_layers
File “”, line 9, in
ImportError: cannot import name ‘conv_utils’ from ‘keras.utils’ (/usr/local/lib/python3.10/dist-packages/keras/utils/init.py)

For full error log, kindly find the attached file.
byom_tao_launch_error.txt (6.3 KB)

Could you please share the spec file?

Sure, here is the spec file I am using.
spec.yaml.txt (1.4 KB)
I have installed the latest tao_byom and followed the byom converter instructions for the latest tao 5.5.

I am using the notebooks and scripts in the github repo GitHub - STMicroelectronics/stm32ai-tao: Nvidia TAO (Train, Adapt, Optimize) with STM32Cube.AI Developer Cloud

This is a 3rd-party github which is not from TAO release.
Please use official GitHub - NVIDIA-AI-IOT/tao_byom_examples: Examples of converting different open-source deep learning models to TAO compatible format through TAO BYOM package. and tao_tutorials/notebooks/tao_launcher_starter_kit/classification_tf2/byom_voc at main · NVIDIA/tao_tutorials · GitHub.

I know but the error that I have is coming from the training. I have this error even when I use the tao provided notebooks.

Hi,
To narrow down, could you please use TAO5.0 docker to retry? You can use docker run.
$ docker run --runtime=nvidia -it -v /your/localpath:/docker/path nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf2.11.0 /bin/bash

Thanks.

Hi Morganh,
There are not issues if I use TAO 5.0 or 5.3. The issue is only coming if I use 5.5 version.

OK, please run with 5.0 or 5.3 version. The 5.5 might bring this regression during software stack. Suggest to use 5.0 or 5.3 since BYOM might be deprecated in future.

OK Thank you!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.