DINO training gives error about insufficient shared memory (shm)

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware: RTX2080Ti * 2
• Network Type : DINO with tiny FAN backbone
• Tao 5.0.0
• Training spec file:
DINO_train.yaml (1.0 KB)

• How to reproduce the issue ? (

When launching DINO training with the following command:
tao model dino train -e /workspace/tao-runs/specs/DINO_train.yaml -r /workspace/tao-runs/artifacts/unpruned/ -k tlt_encode

I get the following error logs:
error.txt (22.2 KB)

It seems like Docker doesn’t have enough memory, I didn’t have this problem with smaller models like Detectnet_v2, so I guess it’s due to the number of parameters of DINO with tiny FAN. Do you have any idea of how I could make it run on two RTX 2080Ti without crashing?

2

Could you set workers: 1 and retry?
More, could you share $nvidia-smi when issue happens?

And also, you can try 1) smaller training size 2) AMP 3) resnet50 backbone .

3

Hi,

I tried all of your suggestions, and tao is still complaining about not having enough memory,

here is my nvidia-smi output:
nvidia-smi.txt (2.2 KB)

Note that setting workers: 1 didn’t change anything

I tried to use AMP but I also get the same error
error_amp.txt (47.0 KB)

The resnet50 backbone is also not working (which makes sense because it has even more parameters than tiny FAN)
error_restnet50.txt (35.2 KB)

4

Can you check why there is “RuntimeError: unable to write to file </torch_577_1506661897_1>: No space left on device (28)” ?
Are there any disk space in your machine?

5

Setting workers: 0 seems to resolve the issue,

However I guess that it’s not be the fastest option because now the dataset is loaded with 1 thread only. Is this gonna impact the time to train for 1 epoch? Or is it just gonna make the data loading step slower?

6

Please go through DINO - NVIDIA Docs to check if it works.

Optimize GPU Memory

There are various ways to optimize GPU memory usage. One obvious trick is to reduce dataset.batch_size. However, this can cause your training to take longer than usual. Hence, we recommend setting below configurations in order to optimize GPU consumption.

  • Set train.precision to fp16 to enable automatic mixed precision training. This can reduce your GPU memory usage by 50%.
  • Set train.activation_checkpoint to True to enable activation checkpointing. By recomputing the activations instead of caching them into memory, the memory usage can be improved.
  • Set train.distributed_strategy to ddp_sharded to enabled Sharded DDP training. This will share gradient calculation across different processes to help reduce GPU memory.
  • Try using more lightweight backbones like fan_tiny or freeze the backbone through setting model.train_backbone to False.
  • Try changing the augmentation resolution in dataset.augmentation depending on your dataset.

Optimize CPU Memory

To speed up data loading, it is a common practice to set high number of workers to spawn multiple processes. However, this can cause your CPU memory to become Out of Memory if the size of your annotation file is very large. Hence, we recommend setting below configurations in order to optimize CPU consumption.

  • Set dataset.dataset_type to serialized so that the COCO-based annotation data can be shared across different subprocesses.
  • Set dataset.augmentation.fixed_padding to True so that images are padded before the batch formulation. Due to random resize and random crop augmentation during training, the resulting image resolution after transform can vary across images. Such variable image resolutions can cause memory leak and the CPU memory to slowly stacks up until it becomes Out of Memory in the middle of training. This is the limitation of PyTorch so we advise setting fixed_padding to True to help stablize the CPU memory usage.
7

Thanks it seems like adding dataset_type: serialized fixed the memory issues,

However, I now get the following errors:
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [31,0,0] Assertion "index >= -sizes[i] && index < sizes[i] && index out of bounds" failed. CUDA error: device-side assert triggered
The error isn’t that precise, tao suggests to pass CUDA_LAUNCH_BLOCKING=1 to get a more precise error, but I’m not sure of how to pass such a variable to tao.

Here is my config and COCO dataset to reproduce the error:

I launch the training with the following command:
tao model dino train -e /workspace/tao-runs/specs/DINO_train.yaml -r /workspace/tao-runs/artifacts/unpruned/ -k tlt_encode

8

Please use docker run to login the docker.
$ docker run --runtime=nvidia -it --rm -v local_path:docker_path nvcr.io/nvidia/tao/tao-toolkit:5.0.0-pyt /bin/bash

Then run following commands.
$ export CUDA_LAUNCH_BLOCKING=1
$ dino train -e /workspace/tao-runs/specs/DINO_train.yaml -r /workspace/tao-runs/artifacts/unpruned/ -k tlt_encode

9

Thanks, however the error given with CUDA_LAUNCH_BLOCKING=1 isn’t more explicit at all, I still get the following error:
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [63,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. CUDA error: device-side assert triggered

error_assert.txt (20.3 KB)

10

Ok so I found the issue,
When using COCO datasets, the category ids must be zero based.

So the nvidia tao documentation is wrong at Data Annotation Format - NVIDIA Docs they state IMPORTANT The id in categories should start from 1. it should be “The id in categories should start from 0” instead

11

We set this NOTE due to Maskrcnn. Refer to The first class is always not detected in inference - #25 by Morganh

For Dino, may I know if you can reproduce any error when you run official notebook? See tao_tutorials/notebooks/tao_launcher_starter_kit/dino/dino.ipynb at main · NVIDIA/tao_tutorials · GitHub or the same in GPU-optimized AI, Machine Learning, & HPC Software | NVIDIA NGC | NVIDIA NGC ?

12

It seems like the note is wrong for DINO then.

I’m sorry but I will not be able to run the notebook because it requires me to download a 35Go COCO dataset, however I assume that your dataset uses category_id that start from 0 as well

13

May I know that you are running against above dataset, right? There are 1 training image and 1 validation image.

14

This was just a dummy dataset to narrow down the problem,

The problem was solved by setting category_id: 0 instead of category_id: 1, I also set the id field to 0 in categories.

15

Got it. Thanks for the info.

16

Hi,
Please use below way instead.

If users train for n classes, then please set num_classes to n+1.

Also still make sure the categories starting from 1.

In short,

  • the num_classes should be actual_class + 1
  • In categories dict, the id should start from 1
  • In annotations dict, the category_id should start from 1