Question: tao model yolo_v4_tiny train

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I am trying Running TAO Toolkit on Google Colab and Colab Enterprise

Here is the link:

When I execute the following code in Colab Enterprise, it gets stuck at this point and can no longer continue running.

image
image1896×244 31.4 KB

the log in Colab Enterprise is like

image
image2382×788 187 KB

But the code in Colab can run normally.

Please let me know how to resolve the problem.

3

From the log,

Epoch 1/10
153/153 [==============================] - 290s 2s/step - loss: 4810.6928
Epoch 2/10
153/153 [==============================] - 274s 2s/step - loss: 1598.6610
Epoch 3/10
153/153 [==============================] - 282s 2s/step - loss: 911.8972
Epoch 4/10
153/153 [==============================] - 290s 2s/step - loss: 599.1410
Epoch 5/10
153/153 [==============================] - 265s 2s/step - loss: 408.5377
Epoch 6/10
114/153 [=====================>........] - ETA: 1:11 - loss: 377.2500

The training is ongoing.
Can you continue to monitor?

4

The problem is when I run the code on Colab Enterprise.

Sorry, what do you mean “continue to the monitor”?

5

I mean the log shows that the training is still running.
You can check if it is running and if it can run to the end.

6

OK, But it has been running over 50 minutes and still not start training.

7

The training is ongoing. It is running on 6th epoch.

8

so the training will not output at the cell, right?
Is there any method can I check the progress of the training ?

9

Do you mean you cannot see the output?
Can you scroll down?
I can see below.

image
image3677×1861 419 KB

10

Here is my output be like

image
image2044×836 122 KB

I have scrolled down but it still don’t have any thing.

11

I can see the output from your shared colab file.

12

the link I gave is not the environment I run, but the code is the same.

The one I used is Colab Enterprise.
image

13

I am confused. I was thinking that is the link you are running.
TAO provides:

https://colab.research.google.com/github/NVIDIA-AI-IOT/nvidia-tao/blob/main/tensorflow/yolo_v4_tiny/yolo_v4_tiny.ipynb

Please run it.

14

No, I copy the file to the service in GCP called “Colab Enterprise”

15

TAO just provides
https://colab.research.google.com/github/NVIDIA-AI-IOT/nvidia-tao/blob/main/tensorflow/yolo_v4_tiny/yolo_v4_tiny.ipynb. So, please run on it.
We are not sure if other way works.

16

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.