My evaluation time increases non-linearly

I am training TrafficCamNet on a new computer.

 My config:
 OS: Ubuntu 20.04 focal
 Kernel: x86_64 Linux 5.13.0-35-generic
 Uptime: 1d 21h 4m
 Shell: bash 5.0.17
 Disk: 1,9T 
 CPU: AMD Ryzen Threadripper 2920X 12-Core @ 24x 3.5GHz
 RAM: 31969MiB

I set up the environment and everything seems to be working well. But I noticed that the validation time is very long. It literally changes non-linearly and I can’t figure out why?

 My log:

When I used to train TrafficCamNet on the same data, the validation was faster, and the s/step did not change

Please double check. And also check if the difference between previous environment and current new computer.

The previous computer definitely had more RAM:

 OS: Ubuntu 20.04 focal
 Kernel: x86_64 Linux 5.13.0-30-generic
 Shell: bash 5.0.17
 Disk: 1,7T
 CPU: Intel Core i7-6850K @ 12x 4GHz
 GPU: NVIDIA GeForce GTX 1080 Ti
 RAM: 128736MiB

But I don’t see any lack of RAM on my PC now.
Maybe it’s because of the difference in GPU memory?

I’m closing this topic due to there is no update from you for a period, assuming this issue was resolved.
If still need the support, please open a new topic. Thanks

Could you check “$nvidia-smi” when you run evaluation?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.