Setting python development environment Isaac Sim

I have set everything and run the examples and I even manage to load my robot (UR3e) from URDF and fix everything to make it work with the gripper Robotiq Hand-e.

However it has been really hard to work with the python environment because I have not figure out a way to make VScode to read the omni python libraries. For now, I’m running Nucleus on my machine Ubuntu 18.04 and I use the Container for the Isaac Sim with the isaac-sim folder share with the host, so I have persistent files on my local machine.

Also I can’t make the Ubuntu terminal or the UI console to print the python “print” statements.

If anyone can help me set those things, that would be awesome. It would be a lot easier to code-test-debug.

I would like to share my writes notes about how I work with Isaac Sim in a container in the hope that it will helps in one way or other.



First, I follow the next steps to excecute Isaac Sim in a container manually in order to vizualize the info/debug/warning/error messages and also, my python “print”… Also, to allow the VSCode remote connection to my code throught ssh

1) Login into NGC before downloading the Isaac Sim container

Get the NGC API Key by clicking Generate API Key at https://ngc.nvidia.com/setup/api-key

docker login nvcr.io

2) Download the Remote Workstation container and run it detached

docker pull nvcr.io/nvidia/isaac-sim:2020.1_ea
docker run --gpus all -e "ACCEPT_EULA=Y" --name isaac-sim-template-base --interactive --entrypoint /bin/bash --detach nvcr.io/nvidia/isaac-sim:2020.1_ea

3) Run the container

docker exec -it isaac-sim-template-base bash

4) Allow ssh access

Set a memorable password for root (like ‘root’)

passwd root

Install shh service

apt install ssh
service ssh restart

Edit the ssh configuration file to allow root login. Uncomment PermitRootLogin and replace by PermitRootLogin yes

nano /etc/ssh/sshd_config

After allow root login, restart the ssh service

service ssh restart

5) Commit the modified container to a docker image

Detach from the docker container using the key sequence: Ctrl + P + Ctrl + Q and commit the changes

docker commit isaac-sim-template-base isaac-sim-template

To see the committed image (isaac-sim-template) use the command docker images.
Now, you can remove the container (isaac-sim-template-base)

docker stop isaac-sim-template-base
docker container rm isaac-sim-template-base

6) Run the development container and detach form it

  • Export ssh port to 12222: -p 12222:22

  • And, export some general-purpose ports for future usage: -p 55000-55003:55000-55003 like tensorboard, for example

docker run -it --gpus all -e "ACCEPT_EULA=Y" --name isaac-sim-sample --entrypoint /bin/bash -p 47995-48012:47995-48012/udp -p 47995-48012:47995-48012/tcp -p 49000-49007:49000-49007/tcp -p 49000-49007:49000-49007/udp -p 12222:22 -p 55000-55003:55000-55003 isaac-sim-template

Detach from the docker container using the key sequence: Ctrl + P + Ctrl + Q

7) Execute the container inside a tmux/screen session to keep Isaac Sim alive

docker exec -it isaac-sim-sample bash

8) Run Isaac Sim

/bin/sh -c /isaac-sim/runheadless.sh


Then, I use VSCode with the Remote - SSH extension to open and edit my code using the previous exported port for the ssh conection

As I said before, this way allows me to see all messages and print in the remote workstation terminal while I deploy the code on my laptop

1 Like

Hi,

Thanks for sharing.

I just tried it. But I got stuck in the last step =(
It just does not allow me to connect through ssh. It keeps saying connection refused no matter what I do.

I found that VSCode also have Remote Containers and since I just want to connect to the container in my local machine it worked flawlessly.

However, that still does not solve the problem of configuring “intellisense” for the Nvidia python libraries. I thought that because VSCode is connected to the container it could some how read the PYTHONPATH or something, but for now it just doesn’t.

Hi, I found how to enable IntelliSense for custom package locations (at least for the omniverse kit) in the container

It is necessary to install Remote - SSH and Python extension

Then, you need to edit the .vscode/settings.json and write the next lines where:

  • /isaac-sim/_build/target-deps/kit_sdk_release/_build/target-deps/python/bin/python3 is the python used by Isaac Sim
  • /isaac-sim/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/plugins/bindings-python contains the python bindings for omniverse kit
{
    "python.pythonPath": "/isaac-sim/_build/target-deps/kit_sdk_release/_build/target-deps/python/bin/python3",
    "python.autoComplete.extraPaths": [
        "/isaac-sim/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/plugins/bindings-python"
    ]
}

The next image shows an example of IntelliSense working

Also, to guarantee the ssh connection with the container the next tips can be helpful:

  • Check the ssh service status in the container: service ssh status. If the service is not working, the restart it: service ssh restart

  • According to the steps that I use to connect to my container, I export the ssh port to 12222 (-p 12222:22). So, I specify the port in the ssh configuration file:

    Host Isaac
        HostName IP_OF_THE_REMOTE_WORKSTATION
        User root
        Port 12222
    
1 Like

Hi, thanks. That works well for me. Though the c++ libraries with python bindings are still hidden, but anyway, do you know where I can find API documentation of the Nvidia libraries?

I manage to do the VSCode setup with the Remote containers that I mentioned above.

I tried again setting up the SSH server but it just does not allow me to connect nor does it give me any useful error message.

When I try to connect it just show me this:
I connect to the Container and check that the SSH server is running:

docker exec -it trufus-isaac /bin/bash
root@0c0daa30cb5a:/isaac-sim# service ssh status
* sshd is not running
root@0c0daa30cb5a:/isaac-sim# service ssh start
* Starting OpenBSD Secure Shell server sshd [ OK ]
root@0c0daa30cb5a:/isaac-sim# grep PermitRootLogin /etc/ssh/sshd_config
#PermitRootLogin prohibit-password
PermitRootLogin yes

Then from a local terminal, I can see that the Container port 22 is exposed to 12222 and I can ping it

cambel@cambel:~ docker port trufus-isaac 22 0.0.0.0:12222 cambel@cambel:~ ping 172.17.0.2
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.076 ms
64 bytes from 172.17.0.2: icmp_seq=3 ttl=64 time=0.038 ms
— 172.17.0.2 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2044ms
rtt min/avg/max/mdev = 0.038/0.053/0.076/0.016 ms

But when I try to SSH it, it just refuse the connection =/

cambel@cambel:~ ssh root@172.17.0.2 -p 12222 ssh: connect to host 172.17.0.2 port 12222: Connection refused cambel@cambel:~ ssh root@172.17.0.2 -p 12222 -vvvv
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /home/cambel/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving “172.17.0.2” port 12222
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 172.17.0.2 [172.17.0.2] port 12222.
debug1: connect to address 172.17.0.2 port 12222: Connection refused
ssh: connect to host 172.17.0.2 port 12222: Connection refused

If you happen to know something about this problem, please let me know.

Hi,

The links of the documentation are the next:


Respect to the ssh connection:

If you have the ssh service installed and enabled, the root login access allowed and the port mapped, then the ssh connection should work…

You can try some new things based on this docker documentation to see if it work like:

  • install openssh-server instead of ssh (this last one works for me)
  • execute some of the commands to fix the SSH login
  • use localhost as IP address if you are on the Docker daemon host
1 Like

Hi,

Thanks for the documentation’s links!

I’ll check the ssh later, for now I can work with the Remote Containers XD

Cheers,

@toni.sm, So I could enable the Intellisense inside VSCode but still not yet hook up on the libraries, as in this image:

Thanks.

Hi @Tadinu,

There are two different concepts here:

  1. IntelliSense is a code editing feature that allows code completion, content assist, and code hinting in any VS Code editor window and not in the terminal

  2. The Python environment, PYTHONPATH, and the terminal:

    From my point of view Isaac Sim has a lot of packages, modules, and dependencies distributed over its file system structure. So, In order to access to those files it has defined his own PYTHONPATH

    Then, if you want to get access to those files from a VS Code terminal you must invoke using bash the script located in /isaac-sim/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/python.sh that internally setups the python environment as in this code:

    root@db64aa754fab:/isaac-sim/_build/linux-x86_64/release/exts# bash /isaac-sim/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/python.sh 
    Python 3.6.8 (default, Jan  8 2020, 19:27:45) 
    [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import omni
    >>> from pxr import Usd
    >>> print(Usd)
    <module 'pxr.Usd._usd' from '/isaac-sim/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/plugins/bindings-python/pxr/Usd/_usd.so'>
    >>> 
    
1 Like

Yeah, I meant that PYTHONPATH actually.
I could see CARB_APP_PATH definition and the series of paths be also defined in isaac-sim/_build/linux-x86_64/release/setup_python_env.sh
It is super clear now in terms of setting up a working python environment.

Thanks for your pointer!