TAO - Custom Mask RCNN - Dataset Convert error

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc) A6000
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Mask_rcnn
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here) v3.22.05-tf1.15.5-py3
• Training spec file(If have, please share here) N/A - Convert task
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)


I’m running a custom TAO Mask RCNN experiment (SpaceNet) and attempting to run the dataset_convert task and getting the following error:

Traceback (most recent call last):
File “/usr/lib/python3.6/multiprocessing/pool.py”, line 119, in worker
result = (True, func(*args, **kwds))
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/dataset_convert.py”, line 190, in _pool_create_tf_example
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/dataset_convert.py”, line 70, in create_tf_example
File “/usr/local/lib/python3.6/dist-packages/PIL/Image.py”, line 3031, in open
“cannot identify image file %r” % (filename if filename else fp)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7fdf57a84d58>

This looks like a PIL error where it is unable to read a particular image (all images are tifs). I have created the mount points as follows:

import os

%env KEY=nvidia_tlt
%env NUM_GPUS=1
%env TAO_USER_EXPERIMENT_DIRECTORY=/workspace/tao-experiments/mask_rcnn
%env TAO_COCO_FILE=/coco/jsonout.json


os.environ[“LOCAL_DATA_DIRECTORY”] = ‘/media/super/data2/SpaceNet/AOI_2_Vegas_Train/RGB-PanSharpen’
os.environ[“LOCAL_COCO_FILE”] = ‘/media/super/data2/SpaceNet/AOI_2_Vegas_Train/coco/jsonout.json’
os.environ[“LOCAL_TFRECORDS_DIRECTORY”] = ‘/home/super/AIProgramming/mask_rcnn/experiments/mask_rcnn/TFRecords’
os.environ[“LOCAL_EXPERIMENT_DIRECTORY”] = os.path.join(
os.getenv(“LOCAL_PROJECT_DIRECTORY”, os.getcwd()),

os.environ[“LOCAL_SPECS_DIRECTORY”] = os.path.join(
os.getenv(“NOTEBOOK_ROOT”, os.getcwd()),
%env TAO_SPECS_DIRECTORY=/workspace/tao-experiments/mask_rcnn/specs
%env TAO_TFRECORDS_DIRECTORY = /workspace/tao-experiments/mask_rcnn/TFRecords


“Mounts”: [
“source”: “experiments”,
“destination”: “/workspace/tao-experiments”
“source”: “/home/super/AIProgramming/mask_rcnn/specs”,
“destination”: “/workspace/tao-experiments/mask_rcnn/specs”
“source”: “/media/super/data2/SpaceNet/AOI_2_Vegas_Train/RGB-PanSharpen”,
“destination”: “/data”
“source”: “/media/super/data2/SpaceNet/AOI_2_Vegas_Train/coco/jsonout.json”,
“destination”: “/coco/jsonout.json”
“source”: “/home/super/AIProgramming/mask_rcnn/experiments/mask_rcnn/TFRecords”,
“destination”: “/workspace/tao-experiments/mask_rcnn/TFRecords”

The command I’m issuing is:

!tao mask_rcnn dataset_convert -i $TAO_DATA_DIRECTORY -a $TAO_COCO_FILE -o $TAO_TFRECORDS_DIRECTORY

I think this is correct as an earlier version identified that an image could not be found. If the issue is indeed a bad image, how do I identify which one and the Trace does not identify the image?


Could you try to open a terminal and check if the image can be identified?

$ tao mask_rcnn run /bin/bash

`#`  python
>> from PIL import Image
>> Image.open("the_path_of_your_tifs_image")

Thank you @Morganh - it appears the tif files were multi-layered and could not be read. I saved them as regular tifs and that worked. However, a warning to others, TFRecords will be formed with tif files but TensorFlow requires jpg (and maybe png - but not sure). After doing all of that I got the model to run and obtained excellent results.

I’m assuming MRCNN models can be ran on a Jetson Orin with Deepstream?

If anyone needs help with SpaceNet datasets and TAO Mask RCNN you can let me know.


Either jpg or png file can work.

Yes, the TAO mask_rcnn model can be ran on any dgpu machine or Jetpack device with Deepstream.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.