• Hardware: DesktopPC (Ubuntu 20)
• Network Type: Detectnet_v2
Trying to execute jupyter notebooks from video example from here TAO Toolkit Quick Start Guide
(cv_samples_v1.3.0), and stuck with error on “4. Run TAO training”
error: “ValueError: invalid literal for int() with base 0: ‘0.0500000007451’”
I am trying to execute
“!tao model detectnet_v2 train -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt
-r $USER_EXPERIMENT_DIR/experiment_dir_unpruned
-k $KEY
-n resnet18_detector
–gpus $NUM_GPUS”
and this what output with error I got:
2023-12-22 19:00:38,087 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2023-12-22 19:00:38,168 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2023-12-22 19:00:38,178 [TAO Toolkit] [WARNING] nvidia_tao_cli.components.docker_handler.docker_handler 288:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/nvidia/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
2023-12-22 19:00:38,178 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
2023-12-22 17:00:40.164065: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2023-12-22 17:00:40,210 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2023-12-22 17:00:41,811 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-22 17:00:41,852 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-22 17:00:41,857 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-22 17:00:43,595 [TAO Toolkit] [INFO] matplotlib.font_manager 1633: generated new fontManager
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-22 17:00:45,801 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-22 17:00:45,840 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-22 17:00:45,843 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2023-12-22 17:00:47,327 [TAO Toolkit] [INFO] root 2102: Starting DetectNet_v2 Training job
2023-12-22 17:00:47,327 [TAO Toolkit] [INFO] main 817: Loading experiment spec at /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt.
2023-12-22 17:00:47,328 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.detectnet_v2.spec_handler.spec_loader 113: Merging specification from /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt
2023-12-22 17:00:47,329 [TAO Toolkit] [INFO] root 2102: 61:29 : ’ dbscan_min_samples: 0.0500000007451’: Couldn’t parse integer: 0.0500000007451
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1702, in _ParseAbstractInteger
return int(text, 0)
ValueError: invalid literal for int() with base 0: ‘0.0500000007451’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1652, in _ConsumeInteger
result = ParseInteger(tokenizer.token, is_signed=is_signed, is_long=is_long)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1674, in ParseInteger
result = _ParseAbstractInteger(text)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1704, in _ParseAbstractInteger
raise ValueError(‘Couldn't parse integer: %s’ % orig_text)
ValueError: Couldn’t parse integer: 0.0500000007451
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/scripts/train.py”, line 1067, in
raise e
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/scripts/train.py”, line 1046, in
main()
File “/usr/local/lib/python3.8/dist-packages/decorator.py”, line 232, in fun
return caller(func, *(extras + args), **kw)
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/utilities/timer.py”, line 46, in wrapped_fn
return_args = fn(*args, **kwargs)
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/scripts/train.py”, line 1024, in main
run_experiment(
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/scripts/train.py”, line 821, in run_experiment
experiment_spec = load_experiment_spec(
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/spec_handler/spec_loader.py”, line 136, in load_experiment_spec
experiment_spec = load_proto(spec_path, experiment_spec, default_spec_path,
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/spec_handler/spec_loader.py”, line 114, in load_proto
_load_from_file(spec_path, proto_buffer)
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/spec_handler/spec_loader.py”, line 100, in _load_from_file
merge_text_proto(f.read(), pb2)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 719, in Merge
return MergeLines(
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 793, in MergeLines
return parser.MergeLines(lines, message)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 818, in MergeLines
self._ParseOrMerge(lines, message)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 837, in _ParseOrMerge
self._MergeField(tokenizer, message)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 967, in _MergeField
merger(tokenizer, message, field)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1042, in _MergeMessageField
self._MergeField(tokenizer, sub_message)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 967, in _MergeField
merger(tokenizer, message, field)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1042, in _MergeMessageField
self._MergeField(tokenizer, sub_message)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 967, in _MergeField
merger(tokenizer, message, field)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1042, in _MergeMessageField
self._MergeField(tokenizer, sub_message)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 967, in _MergeField
merger(tokenizer, message, field)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1042, in _MergeMessageField
self._MergeField(tokenizer, sub_message)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 967, in _MergeField
merger(tokenizer, message, field)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1076, in _MergeScalarField
value = _ConsumeInt32(tokenizer)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1573, in _ConsumeInt32
return _ConsumeInteger(tokenizer, is_signed=True, is_long=False)
File “/usr/local/lib/python3.8/dist-packages/google/protobuf/text_format.py”, line 1654, in _ConsumeInteger
raise tokenizer.ParseError(str(e))
google.protobuf.text_format.ParseError: 61:29 : ’ dbscan_min_samples: 0.0500000007451’: Couldn’t parse integer: 0.0500000007451
Execution status: FAIL
2023-12-22 19:00:52,473 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.