Splitting dataset without tfrecords in TAO YOLOV4

[chijco]

I know that if I use tfrecords it is possible to set there val_split: X and it will use X percent of a dataset for valdation thus I do not have to split mydatasets manually to train and validation subsets

1)If I do not want to use tfrecords is there a similar and simple way to automatically split datasets?
if yes please help me

2)If no, I will probably diside to use the tfrecords
thus i have more questions:
2.1

1)how choud change my yolo_v4_train_resnet18.txt file?
Since I do not want to split data manually I guess there should be only the

   data_sources: {
       tfrecords_path: "/workspace/tao-experiments/data/training/tfrecords/<tfrecords pattern>"
       image_directory_path: "/workspace/tao-experiments/data/training"
   }

and no need in the

   validation_data_sources: {
       tfrecords_path: "/workspace/tao-experiments/data/val/tfrecords/<tfrecords pattern>"
       image_directory_path: "/workspace/tao-experiments/data/val"
   }

Right?

2.2
Do I understand right that there is no need in yolo_v4_tfrecords_kitti_val.txt
I actually need only the
yolo_v4_tfrecords_kitti_train.txt
right?

2.2
assuming that I have 3 datasets and not one
how should change my yolo_v4_tfrecords_kitti_train.txt?
*please give me an example

Thanks a lot in advance

[Morganh]

Please refer to YOLOv4

[chijco]

Hi, also here, it is not what I am asking. Please read my question it has several parts, and it describes the issue qute well.

[Morganh]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

@chijco

1)If I do not want to use tfrecords is there a similar and simple way to automatically split datasets?

Yes, please use the sequence format. More info about “sequence format” can be found in YOLOv4 - NVIDIA Docs

@chijco

Do I understand right that there is no need in yolo_v4_tfrecords_kitti_val.txt
I actually need only the
yolo_v4_tfrecords_kitti_train.txt
right?

Yes.

@chijco

assuming that I have 3 datasets and not one
how should change my yolo_v4_tfrecords_kitti_train.txt?

An example,

dataset_config {
  data_sources: {
    image_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/val/image"
    label_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/val/label"
  }
  data_sources: {
    image_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/training/image_2"
    label_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/training/label_2"
  }
  include_difficult_in_training: true
  image_extension: "png"
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "pedestrian"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "cyclist"
      value: "cyclist"
  }
  target_class_mapping {
      key: "van"
      value: "car"
  }
  target_class_mapping {
      key: "person_sitting"
      value: "pedestrian"
  }
  validation_data_sources: {
    image_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/val/image"
    label_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/val/label"
  }
  validation_data_sources: {
    image_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/training/image_2"
    label_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/training/label_2"
  }
}