Segformer Polygon output

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc) Dual A6000
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Segformer
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

Hi

2 questions:

  1. I am able to inference with a trained tlt using the segformer network. How can we get that actual numerical values for the segmentations? With the MRCNN models you get a json output with the actual polygons as well. Output without numerical values is severely limiting.

  2. Please note that when using the

TAO_Deploy segformer inference

command with a TRT engine model, the input images seem to have to be PNG rather than a JPEG - this is presumably because JPEG does not support an alpha channel? This does not appear to happen with the normal tao segformer inference command.

Thank you.

For item1, it should be an enhancement work for segformer. You can use generate the polygon based on the result images.
For item2, is there any error log for running with JPEG image?

Hi

Thank you for the response and apologies for delay in responding.

For item1, are you saying there is a function that will generate the polygons or is this part of an update/enhancement?

For item 2, the error is

OSError: cannot write mode P as JPEG

I think whatever standard libs are used (presumably something like Pillow) there’s not a check to see if the image is a jpeg and handle it accordingly.

Thank you.

Cheers.

Hi,
For item1, currently, Segformer does not support generating polygons as Mask_rcnn did. Maybe you can refer to this(Polygon coordinates for the mask of each detected object · Issue #2360 · matterport/Mask_RCNN · GitHub) to have a try to draw the polygons.

For item2, could you share the full command and full log? Thanks a lot.

Hi Morgan

Here is the log:

2023-07-07 15:18:05,439 [INFO] root: Registry: [‘nvcr.io’]
2023-07-07 15:18:05,453 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:4.0.0-deploy
2023-07-07 15:18:05,603 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/super/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
python /usr/local/lib/python3.8/dist-packages/nvidia_tao_deploy/cv/segformer/scripts/inference.py --config-path /specs --config-name test_isbi.yaml encryption_key=nvidia_tao model_path=/results/isbi_experiment/export/window_model_export.engine output_dir=/results/isbi_experiment/trt_infer/
2023-07-07 19:18:07,093 [DEBUG] hydra.core.utils: Setting JobRuntime:name=UNKNOWN_NAME
2023-07-07 19:18:07,093 [DEBUG] hydra.core.utils: Setting JobRuntime:name=frozen_cv.segformer.scripts
sys:1: UserWarning:
‘test_isbi.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
:86: UserWarning:
‘test_isbi.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions.
/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
[07/07/2023-19:18:07] [TRT] [W] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[07/07/2023-19:18:08] [TRT] [W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See CUDA_MODULE_LOADING in CUDA C++ Programming Guide
[07/07/2023-19:18:08] [TRT] [W] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
[07/07/2023-19:18:08] [TRT] [W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See CUDA_MODULE_LOADING in CUDA C++ Programming Guide
Producing predictions: 0%| | 0/2 [00:00<?, ?it/s]
Error executing job with overrides: [‘encryption_key=nvidia_tao’, ‘model_path=/results/isbi_experiment/export/window_model_export.engine’, ‘output_dir=/results/isbi_experiment/trt_infer/’]
An error occurred during Hydra’s exception formatting:
AssertionError()
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/PIL/JpegImagePlugin.py”, line 629, in _save
rawmode = RAWMODE[im.mode]
KeyError: ‘P’

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “</usr/local/lib/python3.8/dist-packages/nvidia_tao_deploy/cv/segformer/scripts/inference.py>”, line 3, in
File “”, line 147, in
File “”, line 86, in wrapper
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 389, in _run_hydra
_run_app(
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 452, in _run_app
run_and_report(
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 296, in run_and_report
raise ex
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 213, in run_and_report
return func()
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 453, in
lambda: hydra.run(
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py”, line 132, in run
_ = ret.return_value
File “/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py”, line 260, in return_value
raise self._return_value
File “/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py”, line 186, in run_job
ret.return_value = task_function(task_cfg)
File “”, line 116, in main
File “/usr/local/lib/python3.8/dist-packages/PIL/Image.py”, line 2240, in save
save_handler(self, fp, filename)
File “/usr/local/lib/python3.8/dist-packages/PIL/JpegImagePlugin.py”, line 631, in _save
raise OSError(f"cannot write mode {im.mode} as JPEG") from e
OSError: cannot write mode P as JPEG
Exception ignored in: <function SegformerInferencer.del at 0x7febdeea2550>
Traceback (most recent call last):
File “”, line 116, in del
NameError: name ‘armor_enter_vax_001219’ is not defined
Sending telemetry data.
Telemetry data couldn’t be sent, but the command ran successfully.
[Error]: <urlopen error [Errno -2] Name or service not known>
Execution status: FAIL
2023-07-07 15:18:09,803 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

With the command:

Inference with generated TensorRT engine

!tao-deploy segformer inference -e $SPECS_DIR/test_isbi.yaml
encryption_key=$KEY
model_path=$RESULTS_DIR/isbi_experiment/export/window_model_export.engine
output_dir=$RESULTS_DIR/isbi_experiment/trt_infer/

When I run the inference command with the engine file and an input image is a jpg then it fails with the above error. With a PNG it does not. Also the image overlay is of a much lower resolution with the engine than with a tlt. Maybe there’s some INT8 implications?

Thank you.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Could you please do some preprocessing against your jpg images?
Refer to python - Getting "cannot write mode P as JPEG" while operating on JPG image - Stack Overflow

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.