Got the following error when train with detectnet_v2
root@ca8af7905bbc:/home# tlt-train detectnet_v2 -k ‘MnA0c3ZhYjQ5MTR1Y25mcmJvc2R0YzYwdHU6ZTE3MmNiNGItMzhmOC00MjczLWE1NWQtZmQyNmI4NmFlYzYz’ -r ‘/home/result/’ -e ‘/home/specs/train_spec.txt’
Using TensorFlow backend.[[4173,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:Module: OpenFabrics (openib)
Host: ca8af7905bbcAnother transport will be used instead, although this may result in
lower performance.NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.2020-05-29 11:59:13,276 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at /home/specs/train_spec.txt.
2020-05-29 11:59:13,278 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /home//specs/train_spec.txt
Traceback (most recent call last):
File “/usr/local/bin/tlt-train-g1”, line 8, in
sys.exit(main())
File “./common/magnet_train.py”, line 47, in main
File “”, line 2, in main
File “./detectnet_v2/utilities/timer.py”, line 46, in wrapped_fn
File “./detectnet_v2/scripts/train.py”, line 667, in main
File “./detectnet_v2/scripts/train.py”, line 591, in run_experiment
File “./detectnet_v2/scripts/train.py”, line 457, in train_gridbox
File “./detectnet_v2/dataloader/build_dataloader.py”, line 109, in build_dataloader
File “./detectnet_v2/dataloader/default_dataloader.py”, line 84, in init
File “./detectnet_v2/dataloader/base_dataloader.py”, line 125, in get_data_sources
AssertionError: Cannot find val tfrecords for fold 0for tfrecord: /home/n/tfrecords. Please check the validation fold number and retry