Custom Training TrafficCamNet

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hello,

I am trying to train DetectNet_v2 - ResNet18 on custom data using TrafficCamNet as pretrained model.

I download the weight file using below command:

!ngc registry model download-version nvidia/tlt_trafficcamnet:unpruned_v1.0 --dest .

Then I used the downloaded file resnet18_trafficcamnet.tlt
API_KEY: tlt_encode

The problem is, tlt_train command keeps on writing bunch of stuffs on the console but did not started the training.
I am using DGX1 with all 8 GPUs, and waiting from around 5 Hours but still no clue.

Here is my spec_file:
train.txt (10.7 KB)

Here is tlt_train console output (7MB):

Please help what went wrong.
Thanks

2

Could you set smaller epoch and try again to check if it can run?
For you case, set epochs 4000 to 10.

3

Sorry for the delayed update, I left the command running over night and finally it started training.
Though I’m not sure why it took so long.

4

Thanks for the info.