# Error while training classification model with TAO

**URL:** <https://forums.developer.nvidia.com/t/error-while-training-classification-model-with-tao/227459>\
**Category:** TAO Toolkit\
**Created:** [September 9, 2022, 7:33am UTC](https://forums.developer.nvidia.com/t/error-while-training-classification-model-with-tao/227459 "2022-09-09T07:33:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Baot](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Baot](https://forums.developer.nvidia.com/u/Baot)\
**Post date:** [September 9, 2022, 7:33am UTC](https://forums.developer.nvidia.com/t/error-while-training-classification-model-with-tao/227459/1 "2022-09-09T07:33:04Z")

</div>

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc)  
V100  
• Network Type (Detectnet\_v2/Faster\_rcnn/Yolo\_v4/LPRnet/Mask\_rcnn/Classification/etc)  
Classification  
• TLT Version (Please run “tlt info --verbose” and share “docker\_tag” here)  
dockers: [‘nvidia/tao/tao-toolkit-tf’, ‘nvidia/tao/tao-toolkit-pyt’, ‘nvidia/tao/tao-toolkit-lm’]  
format\_version: 2.0  
toolkit\_version: 3.22.05  
published\_date: 05/25/2022  
• Training spec file(If have, please share here)  
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)  
When I use TAO to train a classification model, I get the following error:

```auto
Traceback (most recent call last):
  File "/usr/local/bin/classification", line 8, in <module>
    sys.exit(main())
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/makenet/entrypoint/makenet.py", line 12, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/entrypoint/entrypoint.py", line 263, in launch_job
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/entrypoint/entrypoint.py", line 47, in get_modules
  File "/usr/lib/python3.6/importlib/ __init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/makenet/scripts/evaluate.py", line 34, in <module>
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1585, in __init__
    super(Session, self). __init__ (target, graph, config=config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 699, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
2022-09-09 15:06:34,933 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

```

I am going to delete the old version of the driver

```auto
./NVIDIA-Linux-x86_64-515.65.01.run --uninstall

```

 ![image](https://global.discourse-cdn.com/nvidia/original/3X/9/0/907259f011b51015099f1b1d4e0061de2527fef2.png)  
So I am going to upgrade the driver, but the following error occurred：

```auto
./NVIDIA-Linux-x86_64-515.65.01.run

```

 ![image](https://global.discourse-cdn.com/nvidia/original/3X/6/6/6655c535dc35d155a426add42718e95000e60505.png)  
Still not working after restart.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [September 9, 2022, 8:24am UTC](https://forums.developer.nvidia.com/t/error-while-training-classification-model-with-tao/227459/3 "2022-09-09T08:24:34Z")

</div>

**There is no update from you for a period, assuming this is not an issue anymore.  
Hence we are closing this topic. If need further support, please open a new one.  
Thanks**

Usually you can update nvidia-driver via below way.  
$ apt-cache search nvidia | grep driver  
For example, for 510 driver,  
$ sudo apt install nvidia-driver-510

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [October 11, 2022, 3:13am UTC](https://forums.developer.nvidia.com/t/error-while-training-classification-model-with-tao/227459/5 "2022-10-11T03:13:45Z")

</div>

Closed on October 11, 2022, 3:13am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.