Please provide the following information when requesting support.
• Hardware (T4/V100/Xavier/Nano/etc) RTX3090
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Segformer
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
docker run --rm -it --gpus all --shm-size=16g -v $PWD:/workspace -v $PWD/../data/segformer:/data nvcr.io/nvidia/tao/tao-toolkit:5.0.0-pyt segformer train -e /workspace/specs/train_isbi.yaml -r /workspace/isbi-experiment -g 4
Using the isbi example, we tested when the batch-size was 4 and 16, respectively. batch-size is just a change in the spec file.
As a result of my experiment, at 1000 iterations
When batch-size is 4, it takes 2 minutes and 20 seconds, and when batch-size is 16, it takes about 9 minutes.
This is not the right way to compare. Because the number of samples dealt are different. The samples per iter will vary based on num_gpus . So, it is important to compare the same no. of samples.
For example, you can compare something like:
32 bs per gpu - 1GPU - 4 iters time,
16 bs per gpu - 4 GPU - 2 iters time,
Above two cases will cover 128 samples and will be fair comparison.
if you set max_iters:1 and bs_per_gpu:2 , gpu:1 ==> then 2 images per iter. For each iter, the dataloader will randomly select 2 images.
if you set max_iters:1 and bs_per_gpu:2 , gpu:2 ==> then 4 images per iter. For each iter, the dataloader will randomly select 4 images.