Issue Fine-tuning Faster-RCNN Model Using Unpruned TLT File

Accelerated Computing Intelligent Video Analytics TAO Toolkit
, , , ,
1

Hello NVIDIA Community,

I am seeking advice regarding a challenge I’m experiencing with the training process of a Faster-RCNN model that uses a ResNet 101 backbone.

Specifically, my issue arises during the fine-tuning stage of my model. I initially trained my model using a custom dataset, and this process was carried out over 20 epochs, resulting in a tlt weight file. Following this, I pruned the model and conducted a retraining session by substitively using the pruned weight file instead of the pre-trained weights.

As part of my iterative training process, I added more data and sought to fine-tune my model. The problem lies here: I aimed to use the unpruned weight files for this stage, but I am encountering difficulties. Despite numerous attempts, I am unable to successfully fine-tune my model using the original, unpruned tlt file.

I am unsure as to why this issue is occurring, given that I could successfully retrain with the pruned weight file. Has anyone encountered a similar issue in their work, or does anyone have suggestions for potential solutions or workarounds?

Any assistance or insight into this matter would be greatly appreciated. Thank you in advance for your time and consideration.

3

Is there any error log when you run this training?

4

image
image1591×443 117 KB

I want to understand how we can use our trained weight files which are in tlt format to train a new set of datasets. What is the general procedure for fine-tuning a model which is trained using pre-trained weights i.e., Resnet101 in my case with Fastercnn

5

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

The error log is the same as another topic you created. Error while re-training with custom dataset using tlt file- FasterRCNN - #3 by Morganh
Please update nvidia-driver to 525 to check if it works.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.