How would you create and use a python virtual environment on a Jetson TX2

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Dear Community,

I am currently working on strengthening a product with 3 jetsons TX2, i.e. making it production ready.

For this second purpose, I am trying to use python virtual environments on the Jetson.

I would like to be able to choose what version of python my application is running with, with the appropriate versions of openCV and tensorflow.

There are some articles that point how to achieve this with OpenCV, e.g.:

But I have the feeling that I am missing the big picture.

I am therefore looking for help: “how would you have multiple isolated python environments on your jetson, each of them being able to make use of the appropriate version of openCV and tensorflow (and without breaking each other) ?”

Thanks,

Emmanuel

  • Cortexia.ch
2

Hi,

Maybe you can try this:

1. Install package in some customized folder rather than /usr/lib/.

2. Add the path via sys module:

import sys
sys.path.insert(0,'/path/to/the_directory')

Thanks.

3

Hi,

Yes, you are right, it should go in a custom location, and pyenv does the job within ~/.pyenv/shims/python

Following the installation process from pyenv guides works well (https://github.com/pyenv/pyenv#installation):

# system version before installing pyenv
$ python --version
Python 2.7.12

# installing pyenv
$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
$ echo 'export PATH="~/.pyenv/bin:$PATH" 
	eval "$(pyenv init -)" 
	eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc

# installing and setting python 3.6.7 as default version to use
$ pyenv install 3.6.7
$ pyenv global 3.6.7

I am able to get the python version I want

$ pyenv versions
  system
* 3.6.7 (set by /home/nvidia/.pyenv/version)
$ python --version
Python 3.6.7

But that’s not sufficient and I still miss the access to the interesting libs :

$ python -c 'import cv2; print(cv2.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cv2'

$ python -c 'import tensorflow; print(tensorflow.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'tensorflow'

At this point, I have mixed the steps from the two following articles, which explain how to compile OpenCV with CUDA support, and within the freshly installed pyenv

The result is satisfactory so far:

$ python -c 'import cv2; print(cv2.__version__)'
3.4.0

But I am hitting a wall with tensorflow, following the official guidelines from https://docs.nvidia.com/deeplearning/dgx/install-tf-jetsontx2/index.html

$ pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp33 tensorflow-gpu
Looking in indexes: https://pypi.org/simple, https://developer.download.nvidia.com/compute/redist/jp33
Collecting tensorflow-gpu
  Could not find a version that satisfies the requirement tensorflow-gpu (from versions: )
No matching distribution found for tensorflow-gpu

And moreover, I am wondering if I will hit other walls for other libs, and if the approach is accurate…

Feedback and help welcome !
Emmanuel

4

Feedback still welcome on my approach and better solutions…

I was thinking on docker to isolate things, but it looks from other threads that the support for docker-nvidia2 is still lacking and does not allow the containers to take advantage of the GPU.

Any remarks / ideas appreciated

Emmanuel
– Cortexia.ch

5

Hi,

Docker may be good for your use case.

You don’t need to use nvidia-docker.
Just use official docker with some hardware setting for Jetson:
[url]https://github.com/Technica-Corporation/Tegra-Docker/blob/master/bin/tx2-docker[/url]

Thanks.

6

This looks promising! Thanks for the link.

We will not be able to test it during the next weeks (holidays) but we will surely investigate starting next year.

I will therefore come back with my feedback in January :)

7

Sure.

Happy Holdidays!

8

Hi guys,

I have not (yet) explored yet the container way because I wanted something with less doubts for now.

Hence a more conventionnal strategy: adapting our development environment to the one of the Jetson, with the python3 version that comes with jetpack.

It still requires a bit of customization to get everything working, so here are the main lines:

A/ On the Jetson side

First, set up manually everything needed for the application (a Makefile helps a lot here)

  1. Flashing the OS with JetPack 3.3 (nothing special here)
  2. install cmake and all libs required to compile OpenCV for python3
  3. install tensorflow for python3
  4. install our application specific libs and python requirements

Secondly, create an clone image of the newly installed jetson

And finally, use this image to flash any new jetson.

The following links provide a lot of detailed information:

  • https://github.com/Naurislv/NVidia-Jetson-TX2-Install-Guide-Lines
  • https://askubuntu.com/questions/672808/sudo-apt-get-install-python-pip-is-failing
  • https://stackoverflow.com/questions/36394101/pip-install-locale-error-unsupported-locale-setting
  • https://docs.nvidia.com/deeplearning/dgx/install-tf-jetsontx2/index.html
  • https://github.com/NVIDIA-AI-IOT/tf_trt_models
  • https://elinux.org/Jetson/TX2_Cloning

B/ On the development side

First, I have used pyenv to be able to install as many versions of python as I need, without interfering. At the time of writing, I use python 3.5.2 since that’s the version that comes with the Jetpack 3.3

Secondly, I use pipenv on project-basis, which goes along well with pyenv and allows me to create a virtual environment both isolated, and identical to the ones on the jetson.

C/ You asked about the updates?

I will follow the Jetpack rythm, which means

  1. Creating a new image when the needs rise (i.e when a new version of python3 will come with the jetpack).
  2. Cloning the new image to the jetsons. That step still requires a wire, but no human interaction at all, and is done in 20minutes or so.
  3. Installing the new version of python with pyenv on the development side, and pipenv install again

I know it is not what I was asking for when opening the thread, quite the other way around :)

… but it serves the use case of aligning the development environments with the jetson environment, with control. Which is what really matters.

9

Hi,

Good to hear that you already have a way to deal with the problem.
Also, thanks for sharing this information with us.