It is similar to topic Tao Classifier Mobilenetv2 very low accuracy compared to effecientnet b0 & Resnet - #15 by tane.vanderboon . But I cannot reproduce.
That user fix the issue when use 3.21.08 docker. Please check if it also works on your side.
TAO Toolkit for Computer Vision | NVIDIA NGC
docker pull nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.08-py3