# Transfer Learning Toolkit Multi-GPUs

**URL:** <https://forums.developer.nvidia.com/t/transfer-learning-toolkit-multi-gpus/120863>\
**Category:** Deep Learning (Training & Inference)\
**Created:** [April 23, 2020, 11:55pm UTC](https://forums.developer.nvidia.com/t/transfer-learning-toolkit-multi-gpus/120863 "2020-04-23T23:55:46Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![brandon.russell](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@brandon.russell](https://forums.developer.nvidia.com/u/brandon.russell)\
**Post date:** [April 23, 2020, 11:55pm UTC](https://forums.developer.nvidia.com/t/transfer-learning-toolkit-multi-gpus/120863/1 "2020-04-23T23:55:46Z")

</div>

Hey everyone, I am trying to run the transfer learning toolkit on multiple GPUs. My company purchased a DGX system with 8 Tesla V100 GPUs. Running the detectnet\_v2 example from the TLT container on NGC seems to run fine on a single GPU, however when scaling up to 8 I receive the following output. It seems as though the model is still training but it just takes longer than expected. I believe this issue is related to the evaluation phase, since it looks like all epochs flow fine until it gets to every 10th epoch where it performs evaluation. Below is the evaluation\_config section of my experiment.prototxt file.

**Experiment Eval Section**  
evaluation\_config {  
validation\_period\_during\_training: 10  
first\_validation\_epoch: 1  
minimum\_detection\_ground\_truth\_overlap {  
key: “m1”  
value: 0.5  
}  
minimum\_detection\_ground\_truth\_overlap {  
key: “m2”  
value: 0.5  
}  
minimum\_detection\_ground\_truth\_overlap {  
key: “leopard”  
value: 0.5  
}  
evaluation\_box\_config {  
key: “m1”  
value {  
minimum\_height: 20  
maximum\_height: 9999  
minimum\_width: 10  
maximum\_width: 9999  
}  
}  
evaluation\_box\_config {  
key: “m2”  
value {  
minimum\_height: 20  
maximum\_height: 9999  
minimum\_width: 10  
maximum\_width: 9999  
}  
}  
evaluation\_box\_config {  
key: “leopard”  
value {  
minimum\_height: 20  
maximum\_height: 9999  
minimum\_width: 10  
maximum\_width: 9999  
}  
}  
average\_precision\_mode: INTEGRATE  
}

**Start Training**  
`tlt-train detectnet_v2 -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti_2gpu.txt \ -r $USER_EXPERIMENT_DIR/02/experiment_dir_unpruned \ -k $KEY \ -n resnet18_detector \ --gpus 8`

**Output**  
One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock.  
Stalled ops:DistributedAdamOptimizer\_Allreduce/HorovodAllreduce\_gradients\_AddN\_58\_0 [missing ranks: 0]  
[2020-04-23 23:49:04.302538: W horovod/common/operations.cc:588] DistributedAdamOptimizer\_Allreduce/HorovodAllreduce\_gradients\_AddN\_59\_0 [missing ranks: 0]

Does anyone have an idea as to what may be happening? Thanks in advance.