Error when executing tao classification, cannot identify image file

• Hardware AWS EC2 instance
• Network Type: Classification, efficientNetB1
• TAO Version 3.0-21.08

I’ve used the following command as always:
!tao classification evaluate -e $SPECS_DIR/classification_spec.cfg -k $KEY | tee “eval_output.txt”

And then these error appears (I put the full log):

Found 75995 images belonging to 133 classes.
2021-12-10 00:13:47,281 [INFO] main: Processing dataset (evaluation): /workspace/tao-experiments/data/split/TEST
Traceback (most recent call last):
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/makenet/scripts/evaluate.py”, line 228, in
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py”, line 494, in return_func
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py”, line 482, in return_func
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/makenet/scripts/evaluate.py”, line 224, in main
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/makenet/scripts/evaluate.py”, line 190, in run_evaluate
File “/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py”, line 91, in wrapper
return func(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training.py”, line 1472, in evaluate_generator
verbose=verbose)
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py”, line 330, in evaluate_generator
generator_output = next(output_generator)
File “/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py”, line 709, in get
six.reraise(*sys.exc_info())
File “/usr/local/lib/python3.6/dist-packages/six.py”, line 696, in reraise
raise value
File “/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py”, line 685, in get
inputs = self.queue.get(block=True).get()
File “/usr/lib/python3.6/multiprocessing/pool.py”, line 644, in get
raise self._value
File “/usr/lib/python3.6/multiprocessing/pool.py”, line 119, in worker
result = (True, func(*args, **kwds))
File “/usr/local/lib/python3.6/dist-packages/keras/utils/data_utils.py”, line 626, in next_sample
return six.next(_SHARED_SEQUENCES[uid])
File “/usr/local/lib/python3.6/dist-packages/keras_preprocessing/image/iterator.py”, line 104, in next
return self.next(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/keras_preprocessing/image/iterator.py”, line 116, in next
return self._get_batches_of_transformed_samples(index_array)
File “/usr/local/lib/python3.6/dist-packages/keras_preprocessing/image/iterator.py”, line 230, in _get_batches_of_transformed_samples
interpolation=self.interpolation)
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/makenet/utils/preprocess_crop.py”, line 68, in load_and_crop_img
File “/usr/local/lib/python3.6/dist-packages/keras_preprocessing/image/utils.py”, line 114, in load_img
img = pil_image.open(io.BytesIO(f.read()))
File “/usr/local/lib/python3.6/dist-packages/PIL/Image.py”, line 2818, in open
raise IOError(“cannot identify image file %r” % (filename if filename else fp))
OSError: cannot identify image file <_io.BytesIO object at 0x7f52337cc888>

I’ve never had this problem before and anything has changed before last time I used it and it worked fine.

Any idea of what could be the cause of this error?

Please check if the training images are available.

Yes, I have used the images for training the model previously and they’re still there. Also I’ve tried to train the pruned model and it works too. Now I’m trying to execute inference and it seems it’s ok, so the only thing I can’t use is evaluate

Please attach your classification_spec.cfg.

model_config {
arch: “efficientnet_b1”
n_layers: 18

Setting these parameters to true to match the template downloaded from NGC.

use_batch_norm: true
all_projections: true
freeze_blocks: 0
freeze_blocks: 1
freeze_blocks: 2
freeze_blocks: 3
freeze_blocks: 4
freeze_blocks: 5
freeze_blocks: 6
input_image_size: “3,240,240”
}
train_config {
train_dataset_path: “/workspace/tao-experiments/data/split/TRAIN”
val_dataset_path: “/workspace/tao-experiments/data/split/VAL”
pretrained_model_path: “/workspace/tao-experiments/classification/pretrained_efficientnet_b1/pretrained_classification_vefficientnet_b1_swish/efficientnet_b1_swish.hdf5”
optimizer {
sgd {
lr: 0.01
decay: 0.0
momentum: 0.9
nesterov: False
}
}
batch_size_per_gpu: 32
n_epochs: 12
n_workers: 16
preprocess_mode: “caffe”
enable_random_crop: True
enable_center_crop: True
label_smoothing: 0.0
mixup_alpha: 0.1

regularizer

reg_config {
type: “L2”
scope: “Conv2D,Dense”
weight_decay: 0.00005
}

learning_rate: 0.006

lr_config {
step {
learning_rate: 0.006
step_size: 10
gamma: 0.1
}
}
}
eval_config {
eval_dataset_path: “/workspace/tao-experiments/data/split/VAL”
model_path: “/workspace/tao-experiments/classification/output/weights/efficientnet_b1_009.tlt”
top_k: 3
batch_size: 256
n_workers: 8
enable_center_crop: True
}

I checked that the images are still in /workspace/tao-experiments/data/split/VAL

Please backup the VAL folder. And then narrow down via trying less images.

Thank you so much, finally worked.

Could you tell me if there’s a limit on the amount of images I can use for the evaluation process?

No limitation. Above way is just trying to check which file is not wrong.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.