Inconsistent conda environment in nvidia pytorch container

The conda environments in NVIDIA’s pytorch containers are inconsistent.

The simplest steps to reproduce are to run any conda install command in the newest container, like:

docker run --rm  -it --entrypoint /bin/bash
$ conda install conda

Then look for “The environment is inconsistent” near the top of the output (it takes a lot of time and you have to scroll up to see it in the very long output).

The last container in which this doesn’t happen is 22.01.

However, even the 22.01 container still has a possibly related issue.
Inside it, conda and pip don’t see opencv installed (running conda list | grep cv and pip list | grep cv in the container gives empty results).
But there are files in the environment as if it were installed, namely in /opt/conda/lib/python3.8/site-packages/cv2 with an old version 3.4.11 of opencv.
Because of this, after running the following:

docker run --rm  -it --entrypoint /bin/bash
$ conda list | grep cv
$ conda install opencv -y
$ conda list | grep cv
$ python3 -c "import cv2; print(cv2.__version__)" 

even though opencv 4.6.0 gets installed by conda, the python command still shows 3.4.11. Which causes all sorts of things to fail. A workaround is to rm --rf the old cv2 files from /opt/conda/lib/python3.8/site-packages/cv2.
This happens also in the newest, 22.07 container, and even if I first activate the conda base environment with

$ conda init
$ . ~/.bashrc
$ conda activate


the issue is not reproducible and conda doesn’t seem to have issues in resolving conflicts:

== PyTorch ==

NVIDIA Release 22.07 (build 40241807)
PyTorch Version 1.13.0a0+08820cb

Container image Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copyright (c) 2014-2022 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:

NOTE: CUDA Forward Compatibility mode ENABLED.
  Using CUDA 11.7 driver version 515.48.08 with kernel driver version 450.119.04.
  See for details.

root@99ec0d4e5b4f:/workspace# conda update conda
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda

  added / updated specs:
    - conda

The following packages will be downloaded:

    package                    |            build
    chardet-5.0.0              |   py38h578d9bd_0         225 KB  conda-forge
    charset-normalizer-2.1.0   |     pyhd8ed1ab_0          35 KB  conda-forge
    colorama-0.4.5             |     pyhd8ed1ab_0          18 KB  conda-forge
    cryptography-37.0.4        |   py38h2b5fc30_0         1.5 MB  conda-forge
    executing-0.9.1            |     pyhd8ed1ab_0          19 KB  conda-forge
    libuv-1.44.2               |       h166bdaf_0         1.0 MB  conda-forge
    libxml2-2.9.14             |       h22db469_3         781 KB  conda-forge
    libzlib-1.2.12             |       h166bdaf_2          63 KB  conda-forge
    openssl-1.1.1q             |       h166bdaf_0         2.1 MB  conda-forge
    patchelf-0.15.0            |       h58526e2_0          95 KB  conda-forge
    readline-8.1.2             |       h0f457ee_0         291 KB  conda-forge
    requests-2.28.1            |     pyhd8ed1ab_0          53 KB  conda-forge
    shellingham-1.5.0          |     pyhd8ed1ab_0          12 KB  conda-forge
    soupsieve-2.3.2.post1      |     pyhd8ed1ab_0          34 KB  conda-forge
    sqlite-3.39.2              |       h4ff8645_0         1.5 MB  conda-forge
    srsly-2.4.4                |   py38hfa26641_0         540 KB  conda-forge
    typer-0.4.2                |     pyhd8ed1ab_0          45 KB  conda-forge
    urllib3-1.26.11            |     pyhd8ed1ab_0         102 KB  conda-forge
    wasabi-0.10.0              |     pyhd8ed1ab_0          26 KB  conda-forge
    zlib-1.2.12                |       h166bdaf_2          91 KB  conda-forge
                                           Total:         8.6 MB

The following packages will be REMOVED:


The following packages will be UPDATED:

  chardet                              4.0.0-py38h578d9bd_3 --> 5.0.0-py38h578d9bd_0
  charset-normalizer                    2.0.12-pyhd8ed1ab_0 --> 2.1.0-pyhd8ed1ab_0
  colorama                               0.4.4-pyh9f0ad1d_0 --> 0.4.5-pyhd8ed1ab_0
  cryptography                        37.0.2-py38h2b5fc30_0 --> 37.0.4-py38h2b5fc30_0
  executing                              0.8.3-pyhd8ed1ab_0 --> 0.9.1-pyhd8ed1ab_0
  libuv                                   1.43.0-h7f98852_0 --> 1.44.2-h166bdaf_0
  libxml2                                 2.9.14-h22db469_2 --> 2.9.14-h22db469_3
  libzlib                                 1.2.12-h166bdaf_0 --> 1.2.12-h166bdaf_2
  openssl                                 1.1.1p-h166bdaf_0 --> 1.1.1q-h166bdaf_0
  patchelf                                0.14.5-h58526e2_0 --> 0.15.0-h58526e2_0
  readline                                   8.1-h46c0cb4_0 --> 8.1.2-h0f457ee_0
  requests                              2.27.1-pyhd8ed1ab_0 --> 2.28.1-pyhd8ed1ab_0
  shellingham                            1.4.0-pyh44b312d_0 --> 1.5.0-pyhd8ed1ab_0
  soupsieve                              2.3.1-pyhd8ed1ab_0 --> 2.3.2.post1-pyhd8ed1ab_0
  sqlite                                  3.38.5-h4ff8645_0 --> 3.39.2-h4ff8645_0
  srsly                                2.4.3-py38hfa26641_1 --> 2.4.4-py38hfa26641_0
  typer                                  0.4.1-pyhd8ed1ab_0 --> 0.4.2-pyhd8ed1ab_0
  urllib3                               1.26.9-pyhd8ed1ab_0 --> 1.26.11-pyhd8ed1ab_0
  wasabi                                 0.9.1-pyhd8ed1ab_0 --> 0.10.0-pyhd8ed1ab_0
  zlib                                    1.2.12-h166bdaf_0 --> 1.2.12-h166bdaf_2

Proceed ([y]/n)? 

Thanks for pointing out the OpenCV2 issue. It’s good to see you have a workaround and we will look into the installation.