I have to do training and evaluation parallel, When i tried to do this i got error ‘exhausted: OOM when allocating tensor with shape’. So is there any options to limit the gpu usage for tlt train?
Please share your training spec and training log. We can check how to solve the error.
For tlt-train, there is one option “–gpu” which can set multi-gpu training.