I’m running tlt-train using a python script,so i want to know when the training completes the entire epochs which is mentioned in the spec file without seeing manually. Is there any way to figure out this?
Actually you can estimate the approximate end time for entire epoches.
Take detectnet_v2 training as an example, if there is similar log as “epoch=60.123456789, loss=0.0007, xxx” , and you set epoch=120, that means the training may run half of the entire time.
For example , i have 1952 images and i set 26 epochs, how much time it will take to complete the 26 epoch based on the following log information for first epoch ?
Can i save the mAP value into a text file after tlt-evaluation (mAP.txt)?
594/1952 [========>.....................] - ETA: 11:18 - rpn_cls: 0.0127 - rpn_regr: 0.0066 - detector_cls: 0.0565 - detector_regr: 0.0479No positive ROIs.
1732/1952 [=========================>....] - ETA: 1:43 - rpn_cls: 0.0131 - rpn_regr: 0.0065 - detector_cls: 0.0550 - detector_regr: 0.0475No positive ROIs.
1837/1952 [===========================>..] - ETA: 54s - rpn_cls: 0.0131 - rpn_regr: 0.0064 - detector_cls: 0.0549 - detector_regr: 0.0474No positive ROIs.
1952/1952 [==============================] - 917s 470ms/step - rpn_cls: 0.0130 - rpn_regr: 0.0064 - detector_cls: 0.0548 - detector_regr: 0.0474
2019-12-07 05:07:29,443 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Mean number of bounding boxes from RPN overlapping ground truth boxes: 46.4528688525
2019-12-07 05:07:29,444 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Classifier accuracy for bounding boxes from RPN: 0.979456166752
2019-12-07 05:07:29,444 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss RPN classifier: 0.012899961031
2019-12-07 05:07:29,444 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss RPN regression: 0.0061052769286
2019-12-07 05:07:29,444 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss Detector classifier: 0.052835594191
2019-12-07 05:07:29,444 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss Detector regression: 0.0467913366585
2019-12-07 05:07:29,444 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Elapsed time: 916.701534033
2019-12-07 05:07:29,445 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Total loss changed from inf to 0.118632168809, saving weights
2019-12-07 05:07:37,429 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc:
1)You can estimate the total training time based on the timestamp.
2)For “Can i save the mAP value into a text file after tlt-evaluation (mAP.txt)?”,
==> yes, actually if you are running with Jupyter notebook, the ipynb file is saved automatically.
You can open it as a notebook or save it as html files or pdf file, etc.
If you run without Jupyter notebook, or maybe you are running with putty, it is accessbile for you to save all the putty log as a text file.
Then you can grep the mAP result from the text file.