License Plate Recognition

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

• GeForce GTX 1660 Ti Mobile
• DeepStream 6.1
• Ubuntu 20.04
• TensorRT 8.2.5.1
• GStreamer 1.16.2
• NVIDIA driver 510.47.03
• CUDA 11.6 Update 1

I want to train LPR model as described in Creating a Real-Time License Plate Detection and Recognition App.

I have an error:

(launcher) codeinside@CI-1442:~$ tao lprnet train -e /workspace/tao-experiments/lprnet/tutorial_spec.txt -r /workspace/tao-experiments/lprnet/ -k nvidia_tao -m /workspace/tao-experiments/lprnet/us_lprnet_baseline18_trainable.tlt
/home/codeinside/anaconda3/envs/launcher/lib/python3.6/site-packages/tlt/__init__.py:20: DeprecationWarning: 
The `nvidia-tlt` package will be deprecated soon. Going forward please migrate to using the `nvidia-tao` package.

  warnings.warn(message, DeprecationWarning)
2022-06-16 16:59:17,642 [INFO] root: Registry: ['nvcr.io']
2022-06-16 16:59:17,693 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
2022-06-16 16:59:17,777 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/codeinside/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:57: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2022-06-16 13:59:23,240 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:57: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2022-06-16 13:59:23,240 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:61: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

2022-06-16 13:59:23,501 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:61: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

2022-06-16 13:59:23,501 [INFO] iva.lprnet.utils.spec_loader: Merging specification from /workspace/tao-experiments/lprnet/tutorial_spec.txt
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py", line 279, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py", line 275, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py", line 64, in run_experiment
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/utils/spec_loader.py", line 127, in load_experiment_spec
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/utils/spec_loader.py", line 107, in load_proto
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/utils/spec_loader.py", line 93, in _load_from_file
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 725, in Merge
    allow_unknown_field=allow_unknown_field)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 793, in MergeLines
    return parser.MergeLines(lines, message)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 818, in MergeLines
    self._ParseOrMerge(lines, message)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 837, in _ParseOrMerge
    self._MergeField(tokenizer, message)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 967, in _MergeField
    merger(tokenizer, message, field)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 1042, in _MergeMessageField
    self._MergeField(tokenizer, sub_message)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 934, in _MergeField
    (message_descriptor.full_name, name))
google.protobuf.text_format.ParseError: 57:5 : Message type "AugmentationConfig" has no field named "transform_prob".
2022-06-16 16:59:24,132 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

If I delete fields “transform_prob” and “rotate_degree” from “augmentation_config”, I have problem:

(launcher) codeinside@CI-1442:~$ tao lprnet train -e /workspace/tao-experiments/lprnet/tutorial_spec.txt -r /workspace/tao-experiments/lprnet/ -k nvidia_tao -m /workspace/tao-experiments/lprnet/us_lprnet_baseline18_trainable.tlt
/home/codeinside/anaconda3/envs/launcher/lib/python3.6/site-packages/tlt/__init__.py:20: DeprecationWarning: 
The `nvidia-tlt` package will be deprecated soon. Going forward please migrate to using the `nvidia-tao` package.

  warnings.warn(message, DeprecationWarning)
2022-06-16 17:06:06,424 [INFO] root: Registry: ['nvcr.io']
2022-06-16 17:06:06,476 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
2022-06-16 17:06:06,563 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/codeinside/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:57: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2022-06-16 14:06:11,768 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:57: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2022-06-16 14:06:11,768 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:61: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

2022-06-16 14:06:12,031 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:61: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

2022-06-16 14:06:12,031 [INFO] iva.lprnet.utils.spec_loader: Merging specification from /workspace/tao-experiments/lprnet/tutorial_spec.txt
2022-06-16 14:06:12,033 [INFO] __main__: Loading pretrained weights. This may take a while...
2022-06-16 17:06:13,098 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

And it stoped without training, what’s wrong?

3

Moving into TAO forum.

5

Please use latest TAO version.

Since the blog is released more than 1 year ago, it is for 3.0-dp-py3 release.
Currently, the latest docker is TAO 22.05.

6

I installed tao by pip3 install nvidia-tao and it is using nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
How I can change container?

7

This stopping should be related to one new issue. Refer to the workaround in Chmod: cannot access ‘/opt/ngccli/ngc’: No such file or directory - #2 by Morganh

8

Now it uses TAO 22.05, but I have the same problem:

codeinside@CI-1442:~$ tao lprnet train -e /workspace/tao-experiments/lprnet/tutorial_spec.txt -r /workspace/tao-experiments/lprnet/ -k nvidia_tao -m /workspace/tao-experiments/lprnet/us_lprnet_baseline18_trainable.tlt
2022-06-17 11:28:43,037 [INFO] root: Registry: ['nvcr.io']
2022-06-17 11:28:43,089 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3
2022-06-17 11:28:43,144 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/codeinside/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:64: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:64: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

INFO: Log file already exists at /workspace/tao-experiments/lprnet/status.json
INFO: Merging specification from /workspace/tao-experiments/lprnet/tutorial_spec.txt
INFO: 57:5 : Message type "AugmentationConfig" has no field named "transform_prob".
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py", line 340, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py", line 336, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py", line 323, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py", line 79, in run_experiment
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/utils/spec_loader.py", line 129, in load_experiment_spec
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/utils/spec_loader.py", line 109, in load_proto
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/utils/spec_loader.py", line 95, in _load_from_file
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 725, in Merge
    allow_unknown_field=allow_unknown_field)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 793, in MergeLines
    return parser.MergeLines(lines, message)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 818, in MergeLines
    self._ParseOrMerge(lines, message)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 837, in _ParseOrMerge
    self._MergeField(tokenizer, message)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 967, in _MergeField
    merger(tokenizer, message, field)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 1042, in _MergeMessageField
    self._MergeField(tokenizer, sub_message)
  File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 934, in _MergeField
    (message_descriptor.full_name, name))
google.protobuf.text_format.ParseError: 57:5 : Message type "AugmentationConfig" has no field named "transform_prob".
2022-06-17 11:28:50,351 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
9

Similar to Error: google.protobuf.text_format.ParseError: 57:5 : Message type "AugmentationConfig" has no field named "transform_prob" - #3 by Morganh,
please follow LPRNet — TAO Toolkit 3.22.05 documentation to modify the augmentation part of your training spec file.

10

I modified spec file and use it Chmod: cannot access ‘/opt/ngccli/ngc’: No such file or directory - #2 by Morganh, but I have a similar problem:

codeinside@CI-1442:~$ tao lprnet train -e /workspace/tao-experiments/lprnet/tutorial_spec.txt -r /workspace/tao-experiments/lprnet/ -k nvidia_tao -m /workspace/tao-experiments/lprnet/us_lprnet_baseline18_trainable.tlt
2022-06-17 13:53:40,104 [INFO] root: Registry: ['nvcr.io']
2022-06-17 13:53:40,154 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-jfriw9ks because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:64: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:64: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

INFO: Merging specification from /workspace/tao-experiments/lprnet/tutorial_spec.txt
INFO: Loading pretrained weights. This may take a while...
INFO: Training was interrupted
INFO: Training was interrupted.
2022-06-17 13:53:49,831 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
11

Please login the docker to debug.
$ tao lprnet run /bin/bash

then
# lprnet train xxx

12

Result:

codeinside@CI-1442:~$ tao lprnet run /bin/bash
2022-06-17 15:10:51,085 [INFO] root: Registry: ['nvcr.io']
2022-06-17 15:10:51,135 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3
groups: cannot find name for group ID 1000
I have no name!@673ddc86ea5d:/workspace$ lprnet train -e /workspace/tao-experiments/lprnet/tutorial_spec.txt -r /workspace/tao-experiments/lprnet/ -k nvidia_tao -m /workspace/tao-experiments/lprnet/us_lprnet_baseline18_trainable.tlt
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-6qlj9eck because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:64: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/train.py:64: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

INFO: Log file already exists at /workspace/tao-experiments/lprnet/status.json
INFO: Merging specification from /workspace/tao-experiments/lprnet/tutorial_spec.txt
INFO: Loading pretrained weights. This may take a while...
INFO: Training was interrupted
INFO: Training was interrupted.
I have no name!@673ddc86ea5d:/workspace$

Log file:

{"date": "6/17/2022", "time": "12:10:44", "status": "FAILURE", "verbosity": "INFO", "message": "Training was interrupted"}
{"date": "6/17/2022", "time": "12:11:9", "status": "FAILURE", "verbosity": "INFO", "message": "Training was interrupted"}
13

Please add docker options.
Refer to TAO Toolkit Launcher - NVIDIA Docs

14

My .tao_mounts.json file:

{
    "Mounts": [
        {
            "source": "/home/codeinside/tao-experiments",
            "destination": "/workspace/tao-experiments"
        },
        {
            "source": "/home/codeinside/openalpr",
            "destination": "/workspace/openalpr"
        }
    ],
    "Envs": [
        {
            "variable": "CUDA_DEVICE_ORDER",
            "value": "PCI_BUS_ID"
        }
    ],
    "DockerOptions": {
        "shm_size": "16G",
        "ulimits": {
            "memlock": -1,
            "stack": 67108864
        },
        "user": "1000:1000",
        "ports": {
            "8888": 8888
        }
    }
}
15

Please change the key
nvidia_tao
to
nvidia_tlt
and retry.

16

This solved my problem, thank you

17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.