Error training detectnet_V2 with TAO

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc)
V100
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc)
Detectnet_v2
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
format_version: 2.0
toolkit_version: 3.22.05
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
Using TAO to train detectnet_V2 is stuck in running the train cell, after a long time there is no progress, and there is no other error message

image
image1290×520 13.1 KB

Any idea?Thanks.

2

How about below?

  • When the issue happens, can you check cpu memory and gpu memory?
  • Is it 100% reproduced?
  • Could you share $nvidia-smi
4

image
image643×675 29.7 KB

In fact tao is running, but very slowly
image
image770×266 6.72 KB

5

Suggest you to update nvidia-driver to 510.
https://docs.nvidia.com/tao/tao-toolkit/text/tao_toolkit_quick_start_guide.html#software-requirements

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.