Hanging On: "Regularizers updated for the loaded model."

While I have been having difficulties getting the DetectNet_V2 training with an acceptable level of mAP above 0, I have found that I am able to get RCNN trained with some results that appear to be acceptable.

However when I have been retraining the pruned model I am getting a long hang on the “regularizers updated” statement. I have attached logs and spec.

train-pruned-resnet18.txt (3.6 KB)
pruned-train-rcnn.txt (7.8 KB)

Is it still a blocker? I am afraid it is no because I see your latest result in Tutorial Spec Error: Message type "RegularizerConfig" has no field named "reg_type"

@Morganh

This is still an issue because I have not been able to retrain the pruned model due to the error in the attached logs in this thread. In the other thread I am attempting to deploy the unpruned model and the untrained pruned model. If I can solve the issue in this thread then I will have a trained pruned model.

Have you ever run the default Faster-rcnn jupyter notebook inside the the tlt docker?
Can it run successfully with default KITTI dataset?

I have not yet run the faster rcnn notebook in the container. I have been following the nvidia documentation on tlt. https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/

The logs I have posted contain minor changes to reduce classes to 1 and train and validate on my dataset.

I can attempt to run the notebook if you think that will be helpful.

If possible, you can try to run. It can help you narrow down.
This is the first time I see this error.

For how to run jupyter notebook, please see
https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/text/requirements_and_installation.html#running-the-transfer-learning-toolkit

docker run --runtime=nvidia -it -v /home/username/tlt-experiments:/workspace/tlt-experiments -p 8888:8888 nvcr.io/nvidia/tlt-streamanalytics:version

Go to the examples folder: cd examples/ . Execute this command from inside Docker to start the jupyter notebook:

jupyter notebook --ip 0.0.0.0 --allow-root

I had an unexpected error in setting up notebook on a remote computer within the docker contained. Will take some time to get that resolved. I slightly modified the notebook’s spec for training.

I am getting the same error on hanging at the regularizer updated. See attached files.

pruned_retrain.txt (3.4 KB)

Log 1.18 (7.8 KB)

Can you run default setting in the notebook along with the default spec?

I had some difficulties running the notebook on a remote computer where I do my training. I am working on getting it running, I will post the output of that when I have completed it or encountered an error.

Do you have a local host PC?

Yes. It is not fit for training though.

OK, got it.