The conda environments in NVIDIA’s pytorch containers are inconsistent.
The simplest steps to reproduce are to run any conda install command in the newest container, like:
docker run --rm -it --entrypoint /bin/bash nvcr.io/nvidia/pytorch:22.07-py3
$ conda install conda
Then look for “The environment is inconsistent” near the top of the output (it takes a lot of time and you have to scroll up to see it in the very long output).
The last container in which this doesn’t happen is 22.01.
However, even the 22.01 container still has a possibly related issue.
Inside it, conda and pip don’t see opencv installed (running conda list | grep cv and pip list | grep cv in the container gives empty results).
But there are files in the environment as if it were installed, namely in /opt/conda/lib/python3.8/site-packages/cv2 with an old version 3.4.11 of opencv.
Because of this, after running the following:
docker run --rm -it --entrypoint /bin/bash nvcr.io/nvidia/pytorch:22.01-py3
$ conda list | grep cv
$ conda install opencv -y
$ conda list | grep cv
$ python3 -c "import cv2; print(cv2.__version__)"
even though opencv 4.6.0 gets installed by conda, the python command still shows 3.4.11. Which causes all sorts of things to fail. A workaround is to rm --rf the old cv2 files from /opt/conda/lib/python3.8/site-packages/cv2.
This happens also in the newest, 22.07 container, and even if I first activate the conda base environment with
$ conda init
$ . ~/.bashrc
$ conda activate
Hi,
the issue is not reproducible and conda doesn’t seem to have issues in resolving conflicts:
=============
== PyTorch ==
=============
NVIDIA Release 22.07 (build 40241807)
PyTorch Version 1.13.0a0+08820cb
Container image Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2022 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
NOTE: CUDA Forward Compatibility mode ENABLED.
Using CUDA 11.7 driver version 515.48.08 with kernel driver version 450.119.04.
See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.
root@99ec0d4e5b4f:/workspace# conda update conda
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /opt/conda
added / updated specs:
- conda
The following packages will be downloaded:
package | build
---------------------------|-----------------
chardet-5.0.0 | py38h578d9bd_0 225 KB conda-forge
charset-normalizer-2.1.0 | pyhd8ed1ab_0 35 KB conda-forge
colorama-0.4.5 | pyhd8ed1ab_0 18 KB conda-forge
cryptography-37.0.4 | py38h2b5fc30_0 1.5 MB conda-forge
executing-0.9.1 | pyhd8ed1ab_0 19 KB conda-forge
libuv-1.44.2 | h166bdaf_0 1.0 MB conda-forge
libxml2-2.9.14 | h22db469_3 781 KB conda-forge
libzlib-1.2.12 | h166bdaf_2 63 KB conda-forge
openssl-1.1.1q | h166bdaf_0 2.1 MB conda-forge
patchelf-0.15.0 | h58526e2_0 95 KB conda-forge
readline-8.1.2 | h0f457ee_0 291 KB conda-forge
requests-2.28.1 | pyhd8ed1ab_0 53 KB conda-forge
shellingham-1.5.0 | pyhd8ed1ab_0 12 KB conda-forge
soupsieve-2.3.2.post1 | pyhd8ed1ab_0 34 KB conda-forge
sqlite-3.39.2 | h4ff8645_0 1.5 MB conda-forge
srsly-2.4.4 | py38hfa26641_0 540 KB conda-forge
typer-0.4.2 | pyhd8ed1ab_0 45 KB conda-forge
urllib3-1.26.11 | pyhd8ed1ab_0 102 KB conda-forge
wasabi-0.10.0 | pyhd8ed1ab_0 26 KB conda-forge
zlib-1.2.12 | h166bdaf_2 91 KB conda-forge
------------------------------------------------------------
Total: 8.6 MB
The following packages will be REMOVED:
libgomp-12.1.0-h8d9b700_16
tzdata-2022a-h191b570_0
The following packages will be UPDATED:
chardet 4.0.0-py38h578d9bd_3 --> 5.0.0-py38h578d9bd_0
charset-normalizer 2.0.12-pyhd8ed1ab_0 --> 2.1.0-pyhd8ed1ab_0
colorama 0.4.4-pyh9f0ad1d_0 --> 0.4.5-pyhd8ed1ab_0
cryptography 37.0.2-py38h2b5fc30_0 --> 37.0.4-py38h2b5fc30_0
executing 0.8.3-pyhd8ed1ab_0 --> 0.9.1-pyhd8ed1ab_0
libuv 1.43.0-h7f98852_0 --> 1.44.2-h166bdaf_0
libxml2 2.9.14-h22db469_2 --> 2.9.14-h22db469_3
libzlib 1.2.12-h166bdaf_0 --> 1.2.12-h166bdaf_2
openssl 1.1.1p-h166bdaf_0 --> 1.1.1q-h166bdaf_0
patchelf 0.14.5-h58526e2_0 --> 0.15.0-h58526e2_0
readline 8.1-h46c0cb4_0 --> 8.1.2-h0f457ee_0
requests 2.27.1-pyhd8ed1ab_0 --> 2.28.1-pyhd8ed1ab_0
shellingham 1.4.0-pyh44b312d_0 --> 1.5.0-pyhd8ed1ab_0
soupsieve 2.3.1-pyhd8ed1ab_0 --> 2.3.2.post1-pyhd8ed1ab_0
sqlite 3.38.5-h4ff8645_0 --> 3.39.2-h4ff8645_0
srsly 2.4.3-py38hfa26641_1 --> 2.4.4-py38hfa26641_0
typer 0.4.1-pyhd8ed1ab_0 --> 0.4.2-pyhd8ed1ab_0
urllib3 1.26.9-pyhd8ed1ab_0 --> 1.26.11-pyhd8ed1ab_0
wasabi 0.9.1-pyhd8ed1ab_0 --> 0.10.0-pyhd8ed1ab_0
zlib 1.2.12-h166bdaf_0 --> 1.2.12-h166bdaf_2
Proceed ([y]/n)?
Thanks for pointing out the OpenCV2 issue. It’s good to see you have a workaround and we will look into the installation.