I have 1000 images in my Training folder, and 200 in Validation folder
I have noticed that while creating TFrecords for the Training Folder the images there are separated “again” into 2 groups - a small one and a large one, thus the 1000 images which were in the Trainig folder become (lets say) 800 and 200
Same thing happens in Validation folder where 200 mages become 160 and 40.
thus I get 4 sets of images.
It seems like it takes each set and divides it again into training and validation subsets. Am I right?
What it is done for? Does the same thing happens in YOLOv4 being run via darknet or it is something special to TAO?
When you create tfrecords file via “dataset_convert”, it will split the dataset via “val_split” parameter. See more info in DetectNet_v2 — TAO Toolkit 3.22.05 documentation .
You can control this parameter to convert the 100% KITTI dataset to the training set.
For example,
kitti_config {
root_directory_path: “/workspace/tao-experiments/data/”
image_dir_name: “training/image_2”
label_dir_name: “training/label_2”
image_extension: “.png”
partition_mode: “random”
num_partitions: 2
val_split: 0
num_shards: 10
}
And also, you can use the same method to convert all of the validation dataset to the validation tfrecords files.
Thank you. Guess I am starting to understand.
If so please approve that I understand right, especially the 2nd part of the two following :
- When I create TFrecords into training folder I should set the val_split =0
2)When I create TFrecords into validation folder I should set the val_split =0 again?
or maybe I shoud set it to be 100? (since this is validation folder thus 100% of the files there shoud be marked as validation?)
thanks
No, it is not a must. Please see more in DetectNet_v2 - NVIDIA Docs.
For example, if you set val_split =12, then 12% of all the dataset will be split into 0-index tfrecords files. The 88% are for 1-index . You can check the name and the file size of tfrecords files to double check.
In training spec file, there is a parameter "validation_fold: 0 ", it will take 0-index of the tfrecords as validation dataset(i.e., the 12% of dataset)
You can set val_split =0.
In a word, the “val_split” is just used to split. It does not mean whether it is a validation dataset or training dataset.
More, user can also set “validation_data_source” instead of "validation_fold: 0 ".
Ok, I have seen the link. Hope I got the point but to make sure, let me be more cpeciffic.
given that I want to have a validation set saved in one folder(I put there 1000 images) and a training set saved in other folder (say I put there another 200 images )
Thus, I should do as follows (see the attached please1):
And in the Jupyter I create TFrecords twice: (once for training folder and once for validation folder)
(see the attached please2)
Is it right?
And if yes am I also right that such a configuration will not make crossvalidation
Yes, you are right.
Your validation dataset(1000 images) is all generated as path_to_tfrecords/val*
Your training dataset(200 images) is all generated as path_to_tfrecords/train*
And also you set “validation_data_source” to use this standalone tfrecords/val*
More, actually, no matter how you set “val_split” value, all the files under path_to_tfrecords/val* are used for evaluation because you set a standalone “validation_data_source”. And the training “data_sources” are all from path_to_tfrecords/train* .
1 Like
Thank you!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.