Hello,
I get the error of tao inference when I run the peoplesemsegnet:
ValueError: could not broadcast input array from shape (4700160) into shape (1566720)
The reference of setting is
https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/peoplesemsegnet
And, my spec file is
unet_peoplesemsegnet.txt (1.2 KB)
I wonder that it’s because the input image size and the model input size are different.
If that is true, how can I fix this problem?
Thanks!
Could you please share the full command and full log?
Here are the command and the log:
!tao unet inference --gpu_index=$GPU_INDEX -e $SPECS_DIR/unet_peoplesemsegnet_vanilla_unet_dynamic.txt \
-m $USER_EXPERIMENT_DIR/export/trt.fp32.tlt.peoplesemsegnet.engine \
-o $USER_EXPERIMENT_DIR/itri_experiment_peoplesemsegnet_1108/ \
-k tlt_encode
2022-11-30 09:23:15,235 [INFO] root: Registry: [‘nvcr.io’]
2022-11-30 09:23:15,328 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-s1ucsyln because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn’t match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py:51: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.2022-11-30 01:23:21,605 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py:51: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py:51: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
2022-11-30 01:23:21,605 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py:51: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
2022-11-30 01:23:21,606 [INFO] iva.common.logging.logging: Log file already exists at /workspace/tao-experiments/unet_tao/itri_experiment_peoplesemsegnet_1108/status.json
2022-11-30 01:23:21,606 [INFO] root: Starting UNet Inference
2022-11-30 01:23:21,606 [INFO] main: Loading experiment spec at /workspace/tao-experiments/unet/specs/unet_peoplesemsegnet_vanilla_unet_dynamic.txt.
2022-11-30 01:23:21,607 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/unet/specs/unet_peoplesemsegnet_vanilla_unet_dynamic.txt
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.2022-11-30 01:23:21,608 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2022-11-30 01:23:21,609 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1
2022-11-30 01:23:21,609 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0Phase test: Total 8500 files.
[11/30/2022-01:23:22] [TRT] [W] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[11/30/2022-01:23:22] [TRT] [E] 3: [executionContext.cpp::setBindingDimensions::946] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::setBindingDimensions::946, condition: profileMaxDims.d[i] >= dimensions.d[i]. Supplied binding dimension [3,3,544,960] for bindings[0] exceed min ~ max range at index 0, maximum dimension in profile is 1, minimum dimension in profile is 1, but supplied dimension is 3.
)
[11/30/2022-01:23:22] [TRT] [W] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[11/30/2022-01:23:22] [TRT] [E] 3: [executionContext.cpp::setBindingDimensions::946] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::setBindingDimensions::946, condition: profileMaxDims.d[i] >= dimensions.d[i]. Supplied binding dimension [3,3,544,960] for bindings[0] exceed min ~ max range at index 0, maximum dimension in profile is 1, minimum dimension in profile is 1, but supplied dimension is 3.
)
0%| | 0/2834 [00:00<?, ?it/s]WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:477: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.2022-11-30 01:23:22,969 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:477: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.
Traceback (most recent call last):
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py”, line 476, in
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py”, line 471, in main
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py”, line 382, in run_experiment
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py”, line 328, in infer_unet
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py”, line 242, in run_inference_trt
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/inference_trt.py”, line 135, in infer
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/inference_trt.py”, line 124, in _inference_folder
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/inference_trt.py”, line 82, in _predict_batch
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/inferencer/trt_inferencer.py”, line 123, in infer_batch
File “<array_function internals>”, line 6, in copyto
ValueError: could not broadcast input array from shape (4700160) into shape (1566720)PyCUDA ERROR: The context stack was not empty upon module cleanup.
A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.2022-11-30 09:23:25,731 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
Can you share the unet_peoplesemsegnet_vanilla_unet_dynamic.txt ? Is it the same as above https://forums.developer.nvidia.com/uploads/short-url/2u7g9oZCIhv1lOKEPWDVf64ycfH.txt ?
How did you generate the trt.fp32.tlt.peoplesemsegnet.engine ? Using tao-converter or using “tao unet export”?
Here is the file :
unet_peoplesemsegnet_vanilla_unet_dynamic.txt (1.2 KB)
And, the converter command is:
!tao converter $USER_EXPERIMENT_DIR/export/peoplesemsegnet_vanilla_unet_dynamic_etlt_fp32.etlt \
-k tlt_encode \
-t fp32 \
-p input_1:0,1x3x544x960,1x3x544x960,1x3x544x960 \
-e $USER_EXPERIMENT_DIR/export/trt.fp32.tlt.peoplesemsegnet.engine
In your training spec file, the images have "model_input_channels: 3 ".
So, please generate engine with
-p input_1:0,1x3x544x960,1x3x544x960,3x3x544x960
Thanks!!!
Hello,
By running the tao converter command above, the inference command can run without error.
However, the the inference process would stop at around 80%, and the output files (vis_overlay_tlt & mask_labels_tlt) are empty.
Below are my command and log:
!tao unet inference --gpu_index=$GPU_INDEX -e $SPECS_DIR/unet_peoplesemsegnet_vanilla_unet_dynamic.txt \
-m $USER_EXPERIMENT_DIR/export/trt.fp32.tlt.peoplesemsegnet.engine \
-o $USER_EXPERIMENT_DIR/itri_experiment_peoplesemsegnet_1108/ \
-k tlt_encode
2022-12-08 10:08:20,262 [INFO] root: Registry: [‘nvcr.io’]
2022-12-08 10:08:20,349 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-vep4ztzs because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn’t match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py:51: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.
2022-12-08 02:08:28,960 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py:51: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py:51: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
2022-12-08 02:08:28,961 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py:51: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
2022-12-08 02:08:28,962 [INFO] iva.common.logging.logging: Log file already exists at /workspace/tao-experiments/unet_tao/itri_experiment_peoplesemsegnet_1108/status.json
2022-12-08 02:08:28,962 [INFO] root: Starting UNet Inference
2022-12-08 02:08:28,962 [INFO] main: Loading experiment spec at /workspace/tao-experiments/unet/specs/unet_peoplesemsegnet_vanilla_unet_dynamic.txt.
2022-12-08 02:08:28,962 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/unet/specs/unet_peoplesemsegnet_vanilla_unet_dynamic.txt
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2022-12-08 02:08:28,964 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2022-12-08 02:08:28,966 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1
2022-12-08 02:08:28,966 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0
Phase test: Total 8500 files.
[12/08/2022-02:08:29] [TRT] [W] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[12/08/2022-02:08:31] [TRT] [W] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0%| | 0/2834 [00:00<?, ?it/s]WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:477: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.
2022-12-08 02:08:31,848 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:477: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.
81%|█████████████████████████████▊ | 2288/2834 [3:58:09<56:49, 6.25s/it]2022-12-08 14:07:28,696 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
Please create a new topic since the original issue is gone.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.