PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO

[pddarrell]

Please provide the following information when requesting support.

• Hardware (RGX 3080)
• Network Type (Classification)
• TLT Version format_version: 2.0
toolkit_version: 3.22.05
published_date: 05/25/2022()
• Training spec file
• How to reproduce the issue ?

I am experiencing an error similar to this topic:

Error when executing tao classification, cannot identify image file

• Hardware AWS EC2 instance • Network Type: Classification, efficientNetB1 • TAO Version 3.0-21.08 I’ve used the following command as always: !tao classification evaluate -e $SPECS_DIR/classification_spec.cfg -k $KEY | tee “eval_output.txt” And then these error appears (I put the full log): Found 75995 images belonging to 133 classes. 2021-12-10 00:13:47,281 [INFO] main: Processing dataset (evaluation): /workspace/tao-experiments/data/split/TEST Traceback (most recent call last): File “…

“When I run tao classification train” the container stops and I get the following error:

There are 2 classes and these are the files:
Uploading: Screenshot from 2022-09-24 16-01-55.png…
Uploading: Screenshot from 2022-09-24 16-02-25.png…
I have checked the files thumbnails for each and all the files are .jpg image files e.g:
Uploading: Screenshot from 2022-09-24 16-03-18.png…

Please advise

[yingliu]

As mentioned in the included topic some images seem to be broken. You can google this error message to find ways to locate the images. You may also try to remove all the formatted&splitted datasets and execute the “prepare dataset” again.

[Morganh]

Please refer to the debug method and solution in TAO - Custom Mask RCNN - Dataset Convert error - #3 by Morganh and TAO - Custom Mask RCNN - Dataset Convert error - #4 by IainA

[pddarrell]

Hi again.

I am struggling to re-purpose your solution to the mask-RCNN topic, where you suggested using the following to check if an image can be identified:

I have tried the following and I am getting a syntax error:

Please give me some very simple instructions on how to check the 1004 .jpgs which are in two folders inside the “train” directory.

Thank you

[Morganh]

Hi,
Why did you run “tao tao_vod run /bin/bash” ? There is no tao_vod.

Can you use "tao mask_rcnn run /bin/bash " ?

[pddarrell]

I was running it because that is the notebook I am using.

I have solved the problem.

It turns out my machine was keeping copies of the files with their original names as hidden files. I discovered this by using the script found here:

PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object

I then just had to run “rm ._*” to remove the rogue files.

Thank you

[Morganh]

Glad to know the issue is fixed.

[pddarrell]

It turns out the issue may not be completely solved.

I can run “tao classification train” on the 2-class, 1004 image database.
it completes 1 epoch but at image 1003 of the 2nd epoch it throws the “UnidentifiedImageError”:

I ran “$ ls -la” in the “train” folder and found the following hidden files, in addition to the two dataset folders.
The contents of “.” and “…” are as follows:

part of the result of “cat .DS_Store” is as follows:

cat ._.DS_Store gives:

1. Are these what are causing the error?
2. Do I fix this in tao_mounts?

Please advise

[Morganh]

@pddarrell

.DS_Store

You can try to delete .DS_Store.

[pddarrell]

Hi, that did not fix it:

Is there any reason not to delete ._.DS_store too?

[Morganh]

@Morganh

.DS_Store

Why there is ._.DS_store file in your training data folder?
You need to check it.
If it is not necessary, please delete it.

[pddarrell]

@Morganh

Why there is ._.DS_store file in your training data folder?

The database was created on a Mac, which, I have now learnt, will put this hidden file in any folder it creates. I have deleted it, but the same error persists.

Does the 1003/1004 have any significance?

[Morganh]

I think there still be something wrong in some files under your training data folder.
Please check further.

[pddarrell]

Every file opens as an image.
I have tried running a PIL-based script, which revealed the hidden copies with the original names, but now doesn’t flag anything.
The next thing to try would be to save each .jpg as a .png and retry. Do you have any other suggestion?

[Morganh]

Suggest you to check your training data folder via bisection method.

[pddarrell]

I have checked using the bisection method (extensively, though not exhaustively) and tao classification train seems to have a problem with every image. I can run you through the exact procedure I followed if you need that, but it is very repetitive. What criterion is it using for identification? These are all .jpgs that open and have no visible artefacts. Is it possible there is another explanation for this behaviour?

May I also repeat that these datasets run on a Jetson Nano and were used that way in earlier tests.

As a sanity check I will try some other images in two classes to see if they will work.

To keep things as simple as possible I cleared the folders from my “train” directory.
I created 2 new folders, one called “cat” and the other “dog”.
I put the image above in each.
When I run “tao classification train” I get the same UIError.

I don’t think its the .jpgs

Please advise

[Morganh]

@pddarrell

I created 2 new folders, one called “cat” and the other “dog”.
I put the image above in each.

So, do you mean you can still meet “PIL.UnidentifiedImageError” even with very few images?
Can you share some of them?

[pddarrell]

@Morganh

So, do you mean you can still meet “PIL.UnidentifiedImageError” even with very few images?
Can you share some of them?

Here is the single image from the “cat” folder"

It was also the single image in the “dog” folder.

I have now used the utility jpeginfo to verify all files.
The image above is [OK].
Here is a printout of part of the results of the contents of the “damage” folder:

Here is a printout of part of the results of the contents of the “healthy” folder:

Every file is [OK]
Here is an example from the “damage” folder:

Here is an example from the “healthy” folder:

(these images are confidential)

I do not think the .jpgs are the problem here.

However the “damage” image does not currently show in the response when published. Here is a screen grab of it in the preview window.

Please advise

[Morganh]

So, you have run two kinds of training dataset, right?
For “cat” and “dog” training, is the training working?
For “damage” and “healthy” training, is the training working?

[pddarrell]

@Morganh

For “cat” and “dog” training, is the training working?

No

For “damage” and “healthy” training, is the training working?
no

In both cases, and in all bisection stages, the PIL.UnidentifiedImageError is thrown on the penultimate frame (see earlier posts in this thread). As per my post #18, I have now verified all .jpgs (even the cat).