# Training Become very slow Yolov4 **URL:** \ **Category:** TAO Toolkit\ **Created:** [December 30, 2021, 7:57am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219 "2021-12-30T07:57:10Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [December 30, 2021, 7:57am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/1 "2021-12-30T07:57:10Z")
Training Spec file: [spec.txt](https://forums.developer.nvidia.com/uploads/short-url/aWYBKJm9ZrO0ZLGww4XeKQekbIE.txt) (2.2 KB) Training Snapshot : ![training_snapshot](https://global.discourse-cdn.com/nvidia/original/3X/d/d/dd64b33fcfcddefe9d83134094b313a08f3cf62a.png) TLT Version: docker\_tag: v3.21.08-py3 Network Type : Yolov\_4 Hi, I am trying to train yolov4 using custom dataset in which i have only one class. but training process running very slow and got killed in the 2nd epoch. i am unable to understand why this issue is happening even i am using command with gpu. I have also attached the screenshot and spec file for your reference. ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [December 31, 2021, 6:19am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/2 "2021-12-31T06:19:16Z")
Hi, please help me to resolve this issue. ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 1, 2022, 4:09am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/3 "2022-01-01T04:09:05Z")
Seems that it is killed due to OOM. You can try a lower batch-size during training. ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 2, 2022, 4:05pm UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/4 "2022-01-02T16:05:27Z")
Hi, I tried but what about the training speed because it is very low for each epoch it takes several minutes to complete. I tried training command with gpu but when the training part is running i checked that the gpu % was still 0%. as shown below Epoch 1/50 2/160 […] - ETA: 2:38:12 - loss: 13.5692/usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on\_batch\_end() is slow compared to the batch update (7.225892). Check your callbacks. % delta\_t\_median) 160/160 [==============================] - 1144s 7s/step - loss: 13.3983 8ba2ee0d2707:60:96 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1\<0\> [1]eth0:172.17.0.4\<0\> 8ba2ee0d2707:60:96 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation 8ba2ee0d2707:60:96 [0] NCCL INFO NET/IB : No device found. 8ba2ee0d2707:60:96 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1\<0\> [1]eth0:172.17.0.4\<0\> 8ba2ee0d2707:60:96 [0] NCCL INFO Using network Socket NCCL version 2.7.8+cuda11.1 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 00/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 01/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 02/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 03/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 04/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 05/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 06/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 07/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 08/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 09/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 10/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 11/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 12/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 13/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 14/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 15/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 16/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 17/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 18/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 19/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 20/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 21/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 22/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 23/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 24/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 25/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 26/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 27/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 28/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 29/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 30/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Channel 31/32 : 0 8ba2ee0d2707:60:96 [0] NCCL INFO Trees [0] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [1] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [2] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [3] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [4] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [5] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [6] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [7] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [8] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [9] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [10] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [11] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [12] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [13] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [14] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [15] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [16] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [17] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [18] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [19] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [20] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [21] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [22] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [23] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [24] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [25] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/-1/-1 [26] -1/-1/-1-\>0-\>-1|-1-\>0-\>-1/- 8ba2ee0d2707:60:96 [0] NCCL INFO Setting affinity for GPU 0 to ffff 8ba2ee0d2707:60:96 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer 8ba2ee0d2707:60:96 [0] NCCL INFO comm 0x7fb50af951a0 rank 0 nranks 1 cudaDev 0 busId 17000 - Init COMPLETE Epoch 2/50 142/160 [=========================\>…] - ETA: 2:08 - loss: 13.0120 ±----------------------------------------------------------------------------+ | NVIDIA-SMI 460.91.03 Driver Version: 460.91.03 CUDA Version: 11.2 | |-------------------------------±---------------------±---------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 5000 Off | 00000000:17:00.0 Off | Off | | 33% 38C P2 44W / 230W | 5118MiB / 16125MiB | 0% Default | | | | N/A | ±------------------------------±---------------------±---------------------+ | 1 Quadro RTX 5000 Off | 00000000:B3:00.0 Off | Off | | 33% 30C P8 3W / 230W | 20MiB / 16122MiB | 0% Default | | | | N/A | ±------------------------------±---------------------±---------------------+ As I am using custom dataset in which I have only one class “phone”. But when I tried same training command with another dataset in which I have also one class for that new dataset gpu is running and training speed is also increasing. Don’t know what’s the issue is can you please help me out from this training speed issue. ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 3, 2022, 7:16am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/5 "2022-01-03T07:16:32Z")
Hi, I am waiting for the response from your side. ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 3, 2022, 2:24pm UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/6 "2022-01-03T14:24:46Z")
> [@user86169](#): > > when the training part is running i checked that the gpu % was still 0%. From the nvidia-smi result, the memory(5118MiB / 16125MiB ) is consumed. You can run nvidia-smi with interval. > [@user86169](#): > > As I am using custom dataset in which I have only one class “phone”. But when I tried same training command with another dataset in which I have also one class for that new dataset gpu is running and training speed is also increasing. Any difference between the two datasets? ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 6, 2022, 9:39am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/7 "2022-01-06T09:39:22Z")
No, only 1 difference is the total no of images. in the dataset i am facing training speed issue having 1600 images with there annotations in kitti format and in the latest dataset i have images around 300. Both dataset having the single class ‘phone’. for my understanding that 1600 images dataset training is running on cpu instead of gpu. if I am correct then what exactly I need to do to resolve this training speed issue, also if I am wrong then what need to do? ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 10, 2022, 4:39am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/8 "2022-01-10T04:39:31Z")
Hi, I am waiting for the response from your side. ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 10, 2022, 6:47am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/9 "2022-01-10T06:47:35Z")
> [@user86169](#): > > for my understanding that 1600 images dataset training is running on cpu instead of gpu. May I know how did you trigger the tao docker? ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 10, 2022, 7:27am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/10 "2022-01-10T07:27:06Z")
Hi, I used the below the below command to trigger tao docker. tao yolo\_v4 train --gpus 1 -e /workspace/tao-experiments/specs/spec.txt -r /workspace/tao-experiments/results -k (Here I am using my key) ![image](https://global.discourse-cdn.com/nvidia/original/3X/b/b/bb70cc1a8e004711b38aea2036db8fd18f8d2777.png) ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 10, 2022, 7:30am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/11 "2022-01-10T07:30:58Z")
Can you check - “nvidia-smi” result before running above command - Run above command - “nvidia-smi” result after running above command ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 10, 2022, 9:18am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/12 "2022-01-10T09:18:40Z")
Before running training command :- 1. using Nvidia-smi Mon Jan 10 14:37:18 2022 ±----------------------------------------------------------------------------+ | NVIDIA-SMI 460.91.03 Driver Version: 460.91.03 CUDA Version: 11.2 | |-------------------------------±---------------------±---------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 5000 Off | 00000000:17:00.0 Off | Off | | 33% 37C P8 8W / 230W | 1MiB / 16125MiB | 0% Default | | | | N/A | ±------------------------------±---------------------±---------------------+ | 1 Quadro RTX 5000 Off | 00000000:B3:00.0 Off | Off | | 33% 35C P8 4W / 230W | 20MiB / 16122MiB | 0% Default | | | | N/A | ±------------------------------±---------------------±---------------------+ ±----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 1 N/A N/A 1335 G /usr/lib/xorg/Xorg 9MiB | | 1 N/A N/A 1414 G /usr/bin/gnome-shell 6MiB | ±----------------------------------------------------------------------------+ 1. After running training command :- now using Nvidia-smi Mon Jan 10 14:42:26 2022 ±----------------------------------------------------------------------------+ | NVIDIA-SMI 460.91.03 Driver Version: 460.91.03 CUDA Version: 11.2 | |-------------------------------±---------------------±---------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 5000 Off | 00000000:17:00.0 Off | Off | | 33% 38C P8 10W / 230W | 1366MiB / 16125MiB | 0% Default | | | | N/A | ±------------------------------±---------------------±---------------------+ | 1 Quadro RTX 5000 Off | 00000000:B3:00.0 Off | Off | | 33% 32C P8 3W / 230W | 20MiB / 16122MiB | 0% Default | | | | N/A | ±------------------------------±---------------------±---------------------+ ±----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 11699 C /usr/bin/python3.6 115MiB | | 0 N/A N/A 11753 C python3.6 1247MiB | | 1 N/A N/A 1335 G /usr/lib/xorg/Xorg 9MiB | | 1 N/A N/A 1414 G /usr/bin/gnome-shell 6MiB | ±----------------------------------------------------------------------------+ ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 10, 2022, 9:20am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/13 "2022-01-10T09:20:11Z")
So, from 1MiB / 16125MiB to 1366MiB / 16125MiB The GPU is up. ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 10, 2022, 9:23am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/14 "2022-01-10T09:23:15Z")
but as u see gpu running % is still 0. so what does it mean. What to do? ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 10, 2022, 9:42am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/15 "2022-01-10T09:42:17Z")
coz for this dataset training having 1600 images, gpu % is still 0 and training speed is also slow but for other datasets gpu % is increasing while training and training speed also good. ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 10, 2022, 10:08am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/16 "2022-01-10T10:08:11Z")
Can you double check via monitoring $ nvidia-smi dmon ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 10, 2022, 10:29am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/17 "2022-01-10T10:29:53Z")
Before running training command :- 1. using Nvidia-smi dmon # gpu pwr gtemp mtemp sm mem enc dec mclk pclk # Idx W C C % % % % MHz MHz ``` 0 8 28 - 0 0 0 0 405 300 1 3 29 - 0 0 0 0 405 300 0 8 28 - 0 0 0 0 405 300 1 3 29 - 0 0 0 0 405 300 0 8 28 - 0 0 0 0 405 300 1 3 29 - 0 0 0 0 405 300 0 8 28 - 0 0 0 0 405 300 1 3 29 - 0 0 0 0 405 300 0 8 28 - 0 0 0 0 405 300 1 3 29 - 0 0 0 0 405 300 0 8 28 - 0 0 0 0 405 300 1 3 29 - 0 0 0 0 405 300 ``` 1. After running training command :- now using Nvidia-smi dmon # gpu pwr gtemp mtemp sm mem enc dec mclk pclk # Idx W C C % % % % MHz MHz ``` 0 44 38 - 0 0 0 0 6500 1620 1 4 30 - 0 0 0 0 405 300 0 44 38 - 0 0 0 0 6500 1620 1 3 30 - 0 0 0 0 405 300 0 44 38 - 0 0 0 0 6500 1620 1 3 30 - 0 0 0 0 405 300 0 44 38 - 0 0 0 0 6500 1620 1 3 30 - 0 0 0 0 405 300 0 44 39 - 0 0 0 0 6500 1620 1 3 30 - 0 0 0 0 405 300 0 44 38 - 0 0 0 0 6500 1620 1 4 30 - 0 0 0 0 405 300 0 44 38 - 0 0 0 0 6500 1620 1 3 30 - 0 0 0 0 405 300 0 9 38 - 0 0 0 0 405 375 1 3 30 - 0 0 0 0 405 300 0 9 37 - 0 0 0 0 405 375 1 4 30 - 0 0 0 0 405 300 0 9 37 - 0 0 0 0 405 300 1 4 30 - 0 0 0 0 405 300 0 89 39 - 31 17 0 0 6500 1620 1 4 30 - 0 0 0 0 405 300 0 48 41 - 42 25 0 0 6500 1875 1 4 30 - 0 0 0 0 405 300 0 50 39 - 0 0 0 0 6500 1875 1 5 30 - 0 0 0 0 405 300 0 44 39 - 0 0 0 0 6500 1620 1 4 30 - 0 0 0 0 405 300 0 44 39 - 0 0 0 0 6500 1620 1 4 30 - 0 0 0 0 405 300 0 44 39 - 8 5 0 0 6500 1650 1 4 30 - 0 0 0 0 405 300 ``` ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 10, 2022, 10:33am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/18 "2022-01-10T10:33:39Z")
not able to understand what’s the issue? ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 10, 2022, 10:39am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/19 "2022-01-10T10:39:40Z")
Actually I am running a yolov4 training with COCO2017 dataset but cannot reproduce “gpu utils is always 0”. Suggest you to monitor the “sm” for a long time. More, please try to use KITTI dataset to train as well. ---
**Author:** ![user86169](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@user86169](https://forums.developer.nvidia.com/u/user86169)\ **Post date:** [January 10, 2022, 10:43am UTC](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219/20 "2022-01-10T10:43:19Z")
Hi, I am already using dataset in kitti format for training. as u can see it’s been around 15 min and still the training is in the 2nd epoch. Epoch 1/50 2/320 […] - ETA: 6:17:21 - loss: 13.3049 /usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on\_batch\_end() is slow compared to the batch update (10.254203). Check your callbacks. % delta\_t\_median) 153/320 [=============\>…] - ETA: 17:39 - loss: 13.6621 [Next page](https://forums.developer.nvidia.com/t/training-become-very-slow-yolov4/199219.md?page=2)