TAO - Custom Mask RCNN - Dataset Convert error

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc) A6000
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Mask_rcnn
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here) v3.22.05-tf1.15.5-py3
• Training spec file(If have, please share here) N/A - Convert task
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

Hi

I’m running a custom TAO Mask RCNN experiment (SpaceNet) and attempting to run the dataset_convert task and getting the following error:

Traceback (most recent call last):
File “/usr/lib/python3.6/multiprocessing/pool.py”, line 119, in worker
result = (True, func(*args, **kwds))
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/dataset_convert.py”, line 190, in _pool_create_tf_example
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/dataset_convert.py”, line 70, in create_tf_example
File “/usr/local/lib/python3.6/dist-packages/PIL/Image.py”, line 3031, in open
“cannot identify image file %r” % (filename if filename else fp)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7fdf57a84d58>
“”"

This looks like a PIL error where it is unable to read a particular image (all images are tifs). I have created the mount points as follows:

import os

%env KEY=nvidia_tlt
%env NUM_GPUS=1
%env TAO_USER_EXPERIMENT_DIRECTORY=/workspace/tao-experiments/mask_rcnn
%env TAO_DATA_DIRECTORY=/data
%env TAO_COCO_FILE=/coco/jsonout.json

%env LOCAL_PROJECT_DIRECTORY=experiments

os.environ[“LOCAL_DATA_DIRECTORY”] = ‘/media/super/data2/SpaceNet/AOI_2_Vegas_Train/RGB-PanSharpen’
os.environ[“LOCAL_COCO_FILE”] = ‘/media/super/data2/SpaceNet/AOI_2_Vegas_Train/coco/jsonout.json’
os.environ[“LOCAL_TFRECORDS_DIRECTORY”] = ‘/home/super/AIProgramming/mask_rcnn/experiments/mask_rcnn/TFRecords’
os.environ[“LOCAL_EXPERIMENT_DIRECTORY”] = os.path.join(
os.getenv(“LOCAL_PROJECT_DIRECTORY”, os.getcwd()),
“mask_rcnn”
)

os.environ[“LOCAL_SPECS_DIRECTORY”] = os.path.join(
os.getenv(“NOTEBOOK_ROOT”, os.getcwd()),
“specs”
)
%env TAO_SPECS_DIRECTORY=/workspace/tao-experiments/mask_rcnn/specs
%env TAO_TFRECORDS_DIRECTORY = /workspace/tao-experiments/mask_rcnn/TFRecords

!ls -rlt $LOCAL_SPECS_DIRIRECTORY
!ls -rlt $LOCAL_DATA_DIRECTORY

{
“Mounts”: [
{
“source”: “experiments”,
“destination”: “/workspace/tao-experiments”
},
{
“source”: “/home/super/AIProgramming/mask_rcnn/specs”,
“destination”: “/workspace/tao-experiments/mask_rcnn/specs”
},
{
“source”: “/media/super/data2/SpaceNet/AOI_2_Vegas_Train/RGB-PanSharpen”,
“destination”: “/data”
},
{
“source”: “/media/super/data2/SpaceNet/AOI_2_Vegas_Train/coco/jsonout.json”,
“destination”: “/coco/jsonout.json”
},
{
“source”: “/home/super/AIProgramming/mask_rcnn/experiments/mask_rcnn/TFRecords”,
“destination”: “/workspace/tao-experiments/mask_rcnn/TFRecords”
}
]
}

The command I’m issuing is:

!tao mask_rcnn dataset_convert -i $TAO_DATA_DIRECTORY -a $TAO_COCO_FILE -o $TAO_TFRECORDS_DIRECTORY

I think this is correct as an earlier version identified that an image could not be found. If the issue is indeed a bad image, how do I identify which one and the Trace does not identify the image?

Cheers

3

Could you try to open a terminal and check if the image can be identified?

$ tao mask_rcnn run /bin/bash

`#`  python
>> from PIL import Image
>> Image.open("the_path_of_your_tifs_image")
4

Thank you @Morganh - it appears the tif files were multi-layered and could not be read. I saved them as regular tifs and that worked. However, a warning to others, TFRecords will be formed with tif files but TensorFlow requires jpg (and maybe png - but not sure). After doing all of that I got the model to run and obtained excellent results.

I’m assuming MRCNN models can be ran on a Jetson Orin with Deepstream?

If anyone needs help with SpaceNet datasets and TAO Mask RCNN you can let me know.

Cheers

5

Either jpg or png file can work.

Yes, the TAO mask_rcnn model can be ran on any dgpu machine or Jetpack device with Deepstream.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.