# WSL2 & TAO issues

**URL:** <https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476>\
**Category:** TAO Toolkit\
**Tags:** wsl, tao\
**Created:** [November 18, 2021, 3:02am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476 "2021-11-18T03:02:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![1442438890](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@1442438890](https://forums.developer.nvidia.com/u/1442438890)\
**Post date:** [November 18, 2021, 3:02am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/1 "2021-11-18T03:02:33Z")

</div>

Please provide the following information when requesting support.

• Hardware (RTX2060–\>wsl2–\>ubuntu20.04)  
• Network Type (Classification)  
• TLT Version (toolkit\_version: 3.21.08; docker(NCCL version 2.7.8+cuda11.1))

365/365 [==============================] - 134s 367ms/step - loss: 2.1175 - acc: 0.4348 - val\_loss: 1.6567 - val\_acc: 0.5323  
c3cd1da7de7d:96:132 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1\<0\> [1]eth0:172.17.0.2\<0\>  
c3cd1da7de7d:96:132 [0] NCCL INFO NET/Plugin : Plugin load returned 0 : libnccl-net.so: cannot open shared object file: No such file or directory.  
c3cd1da7de7d:96:132 [0] NCCL INFO NET/IB : No device found.  
c3cd1da7de7d:96:132 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1\<0\> [1]eth0:172.17.0.2\<0\>  
c3cd1da7de7d:96:132 [0] NCCL INFO Using network Socket  
NCCL version 2.7.8+cuda11.1

c3cd1da7de7d:96:132 [0] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci\_bus/0000:01/…/…/0000:01:00.0  
c3cd1da7de7d:96:132 [0] NCCL INFO graph/xml.cc:469 → 2  
c3cd1da7de7d:96:132 [0] NCCL INFO graph/xml.cc:660 → 2  
c3cd1da7de7d:96:132 [0] NCCL INFO graph/topo.cc:523 → 2  
c3cd1da7de7d:96:132 [0] NCCL INFO init.cc:581 → 2  
c3cd1da7de7d:96:132 [0] NCCL INFO init.cc:840 → 2  
c3cd1da7de7d:96:132 [0] NCCL INFO init.cc:876 → 2  
c3cd1da7de7d:96:132 [0] NCCL INFO init.cc:887 → 2  
Traceback (most recent call last):  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1365, in \_do\_call  
return fn(\*args)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1350, in \_run\_fn  
target\_list, run\_metadata)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1443, in \_call\_tf\_sessionrun  
run\_metadata)  
tensorflow.python.framework.errors\_impl.UnknownError: 2 root error(s) found.  
(0) Unknown: ncclCommInitRank failed: unhandled system error  
[[{{node MetricAverageCallback/HorovodAllreduce\_MetricAverageCallback\_acc\_0}}]]  
(1) Unknown: ncclCommInitRank failed: unhandled system error  
[[{{node MetricAverageCallback/HorovodAllreduce\_MetricAverageCallback\_acc\_0}}]]  
[[MetricAverageCallback/truediv/\_5113]]  
0 successful operations.  
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):  
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/makenet/scripts/train.py”, line 500, in   
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/common/utils.py”, line 494, in return\_func  
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/common/utils.py”, line 482, in return\_func  
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/makenet/scripts/train.py”, line 495, in main  
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/makenet/scripts/train.py”, line 468, in run\_experiment  
File “/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py”, line 91, in wrapper  
return func(\*args, \*\*kwargs)  
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training.py”, line 1418, in fit\_generator  
initial\_epoch=initial\_epoch)  
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training\_generator.py”, line 251, in fit\_generator  
callbacks.on\_epoch\_end(epoch, epoch\_logs)  
File “/usr/local/lib/python3.6/dist-packages/keras/callbacks.py”, line 79, in on\_epoch\_end  
callback.on\_epoch\_end(epoch, logs)  
File “/usr/local/lib/python3.6/dist-packages/horovod/\_keras/callbacks.py”, line 84, in on\_epoch\_end  
self.\_average\_metrics\_in\_place(logs)  
File “/usr/local/lib/python3.6/dist-packages/horovod/\_keras/callbacks.py”, line 77, in \_average\_metrics\_in\_place  
self.backend.get\_session().run(self.allreduce\_ops[metric])  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 956, in run  
run\_metadata\_ptr)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1180, in \_run  
feed\_dict\_tensor, options, run\_metadata)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1359, in \_do\_run  
run\_metadata)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1384, in \_do\_call  
raise type(e)(node\_def, op, message)  
tensorflow.python.framework.errors\_impl.UnknownError: 2 root error(s) found.  
(0) Unknown: ncclCommInitRank failed: unhandled system error  
[[node MetricAverageCallback/HorovodAllreduce\_MetricAverageCallback\_acc\_0 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py:1748)]]  
(1) Unknown: ncclCommInitRank failed: unhandled system error  
[[node MetricAverageCallback/HorovodAllreduce\_MetricAverageCallback\_acc\_0 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py:1748)]]  
[[MetricAverageCallback/truediv/\_5113]]  
0 successful operations.  
0 derived errors ignored.

Original stack trace for ‘MetricAverageCallback/HorovodAllreduce\_MetricAverageCallback\_acc\_0’:  
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/makenet/scripts/train.py”, line 500, in   
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/common/utils.py”, line 482, in return\_func  
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/makenet/scripts/train.py”, line 495, in main  
File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/makenet/scripts/train.py”, line 468, in run\_experiment  
File “/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py”, line 91, in wrapper  
return func(\*args, \*\*kwargs)  
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training.py”, line 1418, in fit\_generator  
initial\_epoch=initial\_epoch)  
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training\_generator.py”, line 251, in fit\_generator  
callbacks.on\_epoch\_end(epoch, epoch\_logs)  
File “/usr/local/lib/python3.6/dist-packages/keras/callbacks.py”, line 79, in on\_epoch\_end  
callback.on\_epoch\_end(epoch, logs)  
File “/usr/local/lib/python3.6/dist-packages/horovod/\_keras/callbacks.py”, line 84, in on\_epoch\_end  
self.\_average\_metrics\_in\_place(logs)  
File “/usr/local/lib/python3.6/dist-packages/horovod/\_keras/callbacks.py”, line 73, in \_average\_metrics\_in\_place  
self.\_make\_variable(metric, value)  
File “/usr/local/lib/python3.6/dist-packages/horovod/\_keras/callbacks.py”, line 58, in \_make\_variable  
allreduce\_op = hvd.allreduce(var, device\_dense=self.device)  
File “/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/ **init**.py”, line 80, in allreduce  
summed\_tensor\_compressed = \_allreduce(tensor\_compressed)  
File “/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/mpi\_ops.py”, line 86, in \_allreduce  
return MPI\_LIB.horovod\_allreduce(tensor, name=name)  
File “”, line 80, in horovod\_allreduce  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/op\_def\_library.py”, line 794, in \_apply\_op\_helper  
op\_def=op\_def)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/util/deprecation.py”, line 513, in new\_func  
return func(\*args, \*\*kwargs)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 3357, in create\_op  
attrs, op\_def, compute\_device)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 3426, in \_create\_op\_internal  
op\_def=op\_def)  
File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 1748, in **init**  
self.\_traceback = tf\_stack.extract\_stack()

2021-11-18 10:46:25,079 [INFO] tlt.components.docker\_handler.docker\_handler: Stopping container.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [November 18, 2021, 3:25am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/2 "2021-11-18T03:25:53Z")

</div>

For WSL, actually the TAO is not tested or verified internally.  
Based on the previous topics from forum end users, TLT2.0 can work well. But TLT 3.0 or TAO 3.0 does not.  
Could you pull the TLT 2.0 docker and try?

> **[NVIDIA TAO Documentation](https://docs.nvidia.com/tao/)**
>
> Reference the latest NVIDIA TLT Documentation.

---

<div class="post-metadata">

**Author:** ![1442438890](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@1442438890](https://forums.developer.nvidia.com/u/1442438890)\
**Post date:** [November 18, 2021, 3:39am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/3 "2021-11-18T03:39:59Z")

</div>

thanks so much

---

<div class="post-metadata">

**Author:** ![joshH](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@joshH](https://forums.developer.nvidia.com/u/joshH)\
**Post date:** [November 25, 2021, 12:09am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/4 "2021-11-25T00:09:18Z")

</div>

Why are libnccl2 and libnccl-dev not updated in the TAO images?

The newest versions do not exhibit this error and manually updating the image with the newest versions does make it work without any further changes.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [November 25, 2021, 12:53am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/5 "2021-11-25T00:53:40Z")

</div>

Thanks @joshH for the solution.  
I will sync with internal team for this.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 6, 2021, 4:12pm UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/6 "2021-12-06T16:12:35Z")

</div>

Thanks @joshH for the hint again.  
For latest TAO docker, it can also run in WSL.

Update the nccl via [https://developer.nvidia.com/nccl/nccl-download](https://developer.nvidia.com/nccl/nccl-download)  
For example, update to 2.11.4 version.  
sudo apt install libnccl2=2.11.4-1+cuda11.0 libnccl-dev=2.11.4-1+cuda11.0

---

<div class="post-metadata">

**Author:** ![1442438890](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@1442438890](https://forums.developer.nvidia.com/u/1442438890)\
**Post date:** [December 7, 2021, 10:47am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/7 "2021-12-07T10:47:34Z")

</div>

thanks a lot

---

<div class="post-metadata">

**Author:** ![1442438890](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@1442438890](https://forums.developer.nvidia.com/u/1442438890)\
**Post date:** [December 7, 2021, 11:50am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/8 "2021-12-07T11:50:27Z")

</div>

But how can i upgrade nccl in tao-docker

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 7, 2021, 1:12pm UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/9 "2021-12-07T13:12:22Z")

</div>

@1442438890  
Inside the docker, run below command or similar ones.  
`#` sudo apt install libnccl2=2.11.4-1+cuda11.0 libnccl-dev=2.11.4-1+cuda11.0

---

<div class="post-metadata">

**Author:** ![1442438890](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@1442438890](https://forums.developer.nvidia.com/u/1442438890)\
**Post date:** [December 9, 2021, 7:01am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/10 "2021-12-09T07:01:32Z")

</div>

The version of NCCL is bound to the tao-docker image. tao command automatically runs images as a temporary container every time. Can I find the Dokcerfile that is called by tao command and modify it so that it can run NCCL upgrade commands by default every time the container is built?

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 9, 2021, 7:54am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/11 "2021-12-09T07:54:02Z")

</div>

> [@1442438890](#):
>
> Can I find the Dokcerfile that is called by tao command

Currently, the docker will be downloaded when you run tao command for the first time.  
You can find the tao docker image via “docker images”.  
It should match one of the images when you run “tao info --verbose”

Upgrading NCCL inside the docker is just a workaround as of now. If you want to load a final docker image every time, you can build a new docker based on the tao docker.

In future release of tao, we’ll update the NCCL.

---

<div class="post-metadata">

**Author:** ![1442438890](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@1442438890](https://forums.developer.nvidia.com/u/1442438890)\
**Post date:** [December 9, 2021, 10:50am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/12 "2021-12-09T10:50:59Z")

</div>

thanks so lot

---

<div class="post-metadata">

**Author:** ![chongyeh91](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@chongyeh91](https://forums.developer.nvidia.com/u/chongyeh91)\
**Post date:** [December 12, 2021, 5:47am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/13 "2021-12-12T05:47:25Z")

</div>

Have this been fixed in the new release? I still have the same problem with NCCL

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 12, 2021, 2:27pm UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/14 "2021-12-12T14:27:25Z")

</div>

@chongyeh91  
No, the latest 3.21.11 is not including the update.  
Please update the NCCL inside the docker.

---

<div class="post-metadata">

**Author:** ![chongyeh91](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@chongyeh91](https://forums.developer.nvidia.com/u/chongyeh91)\
**Post date:** [December 12, 2021, 4:11pm UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/15 "2021-12-12T16:11:34Z")

</div>

I am not sure how it can be done…

Do I run “docker images” or “docker container list” to view the docker ID? Then run “docker exec [DOCKER ID] apt install libnccl2=2.11.4-1+cuda11.0 libnccl-dev=2.11.4-1+cuda11.0”?

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 13, 2021, 6:51am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/16 "2021-12-13T06:51:49Z")

</div>

When you trigger the tao docker, for example  
$ tao ssd  
then you can find the docker ID via  
$ docker ps

Then,  
$ docker exec -it `<`DOCKER ID`>` /bin/bash  
`#` apt install libnccl2=2.11.4-1+cuda11.0 libnccl-dev=2.11.4-1+cuda11.0

Then, the NCCL will be installed in this docker( assume this docker is still running)

---

<div class="post-metadata">

**Author:** ![joshH](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@joshH](https://forums.developer.nvidia.com/u/joshH)\
**Post date:** [December 17, 2021, 5:30pm UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/17 "2021-12-17T17:30:15Z")

</div>

@Morganh No Problem!  
NGC shows a updated TAO image (updated 17.12), does this fix it? The version seems unchanged.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 18, 2021, 3:00am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/18 "2021-12-18T03:00:14Z")

</div>

> [@joshH](#):
>
> shows a updated TAO image (updated 17.12)

Could you share the exact link?

---

<div class="post-metadata">

**Author:** ![joshH](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@joshH](https://forums.developer.nvidia.com/u/joshH)\
**Post date:** [December 18, 2021, 11:42am UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/19 "2021-12-18T11:42:35Z")

</div>

Of cause: [NGC link](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/containers/tao-toolkit-tf)

I guess the ngc entry was edited and not the image necessarily?

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 18, 2021, 1:16pm UTC](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476/20 "2021-12-18T13:16:33Z")

</div>

Thanks for the info. The dockers are still the ones which are released last month.

> **[TAO Toolkit for Computer Vision | NVIDIA NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/containers/tao-toolkit-tf/tags)**
>
> Docker container with workflows implemented in TensorFlow as part of the Train Adapt Optimize (TAO) Toolkit.

[Next page](https://forums.developer.nvidia.com/t/wsl2-tao-issues/195476.md?page=2)
