AssertionError: Cannot find val tfrecords for fold 0for tfrecord: /home/tfrecords. Please check the validation fold number and retry

Accelerated Computing Intelligent Video Analytics TAO Toolkit
#1

Got the following error when train with detectnet_v2

root@ca8af7905bbc:/home# tlt-train detectnet_v2 -k ‘MnA0c3ZhYjQ5MTR1Y25mcmJvc2R0YzYwdHU6ZTE3MmNiNGItMzhmOC00MjczLWE1NWQtZmQyNmI4NmFlYzYz’ -r ‘/home/result/’ -e ‘/home/specs/train_spec.txt’
Using TensorFlow backend.

[[4173,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
Host: ca8af7905bbc

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.

2020-05-29 11:59:13,276 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at /home/specs/train_spec.txt.
2020-05-29 11:59:13,278 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /home//specs/train_spec.txt
Traceback (most recent call last):
File “/usr/local/bin/tlt-train-g1”, line 8, in
sys.exit(main())
File “./common/magnet_train.py”, line 47, in main
File “”, line 2, in main
File “./detectnet_v2/utilities/timer.py”, line 46, in wrapped_fn
File “./detectnet_v2/scripts/train.py”, line 667, in main
File “./detectnet_v2/scripts/train.py”, line 591, in run_experiment
File “./detectnet_v2/scripts/train.py”, line 457, in train_gridbox
File “./detectnet_v2/dataloader/build_dataloader.py”, line 109, in build_dataloader
File “./detectnet_v2/dataloader/default_dataloader.py”, line 84, in init
File “./detectnet_v2/dataloader/base_dataloader.py”, line 125, in get_data_sources
AssertionError: Cannot find val tfrecords for fold 0for tfrecord: /home/n/tfrecords. Please check the validation fold number and retry

#2

There should be something missing for val dataset.
Please see Training Object Detection with pretrained resnet 18 , following Detectnet_V2 example

#3

Hi,

While training i used -k ‘abc’. So where should i add this key ?

I got error

IOError: Invalid decryption. Unable to open file (File signature not found). The key used to load the model is incorrect.

#4

Can you elaborate you new issue? Command and log are appreciated.

#5

Issue solved by passing -k tlt_encode … Please add this in the PeopleNet documentation

#6

I read that TLT will not work with the dataset which contains different resolutioned images. My dataset consist of small and very high resolution images . Now training on the image without resizing , Got 16% accuracy too. So how its working now with my dataset?

#7

Suggest to resize your images to the same resolution.

#8

But half of the images are in 4k resolution , and the ROI’s are very small . If i resized into small sizes then the roi features will be lost

#9

For your case, you can consider the same sized cropping. To crop the 4k images with the same size.
This is also a way of data augmentation.

#10