SegFormer Pretrained Models - Different Architectures

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Although I can find pre-trained Segformer models on NGC Catalog, the model architectures are different from the paper (B0-B5).
In the training configuration schema, these are the options for the backbones:
-mit_b0
-mit_b1
-mit_b2
-mit_b3
-mit_b4
-mit_b5
-fan_tiny_8_p4_hybrid
-fan_large_16_p4_hybrid
-fan_small_12_p4_hybrid
-fan_base_16_p4_hybrid

The models in the catalog are FAN models: Pre-trained SegFormer ImageNet weights | NVIDIA NGC

What’s the correspondence of these models to the ones described in the paper? (For instance, is the tiny FAN model equivalent to B0?)
Where can I find the pre-trained models for mit_b(0-5) backbones?

2

You can find the mit_b5 pretrained model in CitySemSegformer | NVIDIA NGC
The fan_tiny is not equivalent to B0.

3

I was able to reach the pretrained models for mit_b0 to b5 from the GitHub repo: GitHub - NVlabs/SegFormer: Official PyTorch implementation of SegFormer
Now, the link on the repo isn’t available.

The ones that I downloaded earlier (mit_b0 and b1) from this repo was working with tao 4.0.0-pyt, however it throws out an error with the current version 5.0.0-pyt.

Are you planning to share some pretrained models for these backbones, or is this FAN structure a permanent update on the architecture of SegFormer?

4

For the GitHub repo: GitHub - NVlabs/SegFormer: Official PyTorch implementation of SegFormer, please try to download from ( google drive | onedrive ).

5

As I indicated earlier, I receive the following error (in tao v4, there was no problem) when I choose the backbone option as ‘mit_b0’ and use the mit_b0.pth file in the link that you have provided:

:245: UserWarning: Log file already exists at /workspace/tao-experiments/SegFormer_TAO/results/train/status.json
2023-09-11 07:17:51,363 - mmseg - INFO - Start logging for Training
2023-09-11 07:17:51,474 - mmseg - INFO - Label Id 0: Train Id 0
2023-09-11 07:17:51,474 - mmseg - INFO - Label Id 1: Train Id 1
2023-09-11 07:17:51,474 - mmseg - INFO - Label Id 2: Train Id 2
2023-09-11 07:17:51,474 - mmseg - INFO - Label Id 3: Train Id 3
2023-09-11 07:17:51,474 - mmseg - INFO - Label Id 4: Train Id 4
2023-09-11 07:17:51,474 - mmseg - INFO - Label Id 5: Train Id 5
2023-09-11 07:17:51,474 - mmseg - INFO - Completed Data Module Construction
Concatenating datasets
2023-09-11 07:17:51,606 - mmseg - INFO - Loaded 10140 images
Error executing job with overrides: [‘results_dir=/workspace/tao-experiments/SegFormer_TAO/results’]
An error occurred during Hydra’s exception formatting:
AssertionError()
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 254, in run_and_report
assert mdl is not None
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “</usr/local/lib/python3.8/dist-packages/nvidia_tao_pytorch/cv/segformer/scripts/train.py>”, line 3, in
File “”, line 176, in
File “”, line 107, in wrapper
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 389, in _run_hydra
_run_app(
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 452, in _run_app
run_and_report(
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 296, in run_and_report
raise ex
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 213, in run_and_report
return func()
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py”, line 453, in
lambda: hydra.run(
File “/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py”, line 132, in run
_ = ret.return_value
File “/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py”, line 260, in return_value
raise self._return_value
File “/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py”, line 186, in run_job
ret.return_value = task_function(task_cfg)
File “”, line 172, in main
File “”, line 162, in main
File “”, line 124, in run_experiment
File “”, line 85, in build_segmentor
File “”, line 51, in build
File “/usr/local/lib/python3.8/dist-packages/mmcv/utils/registry.py”, line 72, in build_from_cfg
raise type(e)(f’{obj_cls.name}: {e}')
KeyError: “EncoderDecoder: ‘state_dict’”

The mit_b0 backbone introduced in tao 5.0.0-pyt doesn’t match with the ones in GitHub repo.
Where else can I find these backbones?

6

Will sync with internal team if there are B0/B1/B2/B3/B4 pretrained models.

1 Like