# Deep Reinforcement Learning in Robotics

**URL:** <https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908>\
**Category:** Jetson Nano\
**Tags:** jetson-inference\
**Created:** [June 21, 2020, 2:10am UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908 "2020-06-21T02:10:23Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![dkcog123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dkcog123](https://forums.developer.nvidia.com/u/dkcog123)\
**Post date:** [June 21, 2020, 2:10am UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/1 "2020-06-21T02:10:23Z")

</div>

I am working on two days to demo tutorial. i am at the deep reinforcement learning in robotics:

> **[Five Steps to Reinforcement Learning - Two Days to a Demo](https://developer.nvidia.com/embedded/twodaystoademo#collapseFour)**
>
> Two Days to a Demo is our introductory series of deep learning tutorials for deploying AI and computer vision to the field with NVIDIA Jetson AGX Xavier, Jetson TX2, Jetson TX1 and Jetson Nano. This tutorial takes roughly two days to complete from...

I am at “Building from Source”

> **[GitHub - dusty-nv/jetson-reinforcement: Deep reinforcement learning GPU libraries for...](https://github.com/dusty-nv/jetson-reinforcement#building-from-source)**
>
> Deep reinforcement learning GPU libraries for NVIDIA Jetson TX1/TX2 with PyTorch, OpenAI Gym, and Gazebo robotics simulator.

Here is the step:  
$ sudo apt-get install cmake  
$ git clone [GitHub - dusty-nv/jetson-reinforcement: Deep reinforcement learning GPU libraries for NVIDIA Jetson TX1/TX2 with PyTorch, OpenAI Gym, and Gazebo robotics simulator. · GitHub](http://github.com/dusty-nv/jetson-reinforcement)  
$ cd jetson-reinforcement  
$ git submodule update --init  
$ mkdir build  
$ cd build  
$ cmake ../  
$ make

An error occurs at “cmake ../” step.  
As a result, no PyTorch is not available.

Please see attached two log files: Cmake Error and Cmake Output log files.

I have tried twice, but both failed at the same place. Here is what I did:

1. Flash SD card for Nano
2. Initial Ubuntu setup
3. Directly jump to the above step.

Please help me! I want to try out this tutorial.

Thanks!

[CMakeError.log](https://forums.developer.nvidia.com/uploads/short-url/4for8Hb7Ny33I0WxaSfRGdK6cmw.log) (2.8 KB) [CMakeOutput.log](https://forums.developer.nvidia.com/uploads/short-url/uNhXMXdu3AF3tsXssPfjX0mIf2B.log) (45.6 KB)

---

<div class="post-metadata">

**Author:** ![therealleonmusk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/therealleonmusk/32/16457_2.png) [@therealleonmusk](https://forums.developer.nvidia.com/u/therealleonmusk)\
**Post date:** [June 21, 2020, 2:55am UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/2 "2020-06-21T02:55:28Z")

</div>

Hi I’ve had a few issues too, Usually found out its some dependency has been updated to a newer version. I think they just reposted an updated dependancy list to show which particular versions you need as some of them had bits left out as obsolete to new roll out. Hope this helps feel free to share your resolve as i may run into it myself at some stage and am always curious about others experience with the system! I’m only new to platform and haven’t programmed in nearly 15 years so might be way off though so good luck and best wishes :)

---

<div class="post-metadata">

**Author:** ![dkcog123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dkcog123](https://forums.developer.nvidia.com/u/dkcog123)\
**Post date:** [June 21, 2020, 6:15pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/3 "2020-06-21T18:15:10Z")

</div>

Thanks for your quick response! Yes, I agree with you. Jetpack 4.4 might uses a slight different dependency tree, which causes the error, which stops building PyTorch, which is a part of Jetson-reinforcement.

I am actually able to build PyTorch with the normal jetson-inference.

I am thinking about making PyTorch available form jetson-inference. I don’t know how though at this point.

Still it is the best if the jetson-reinforcement branch works by itself. Hopefully, somebody can help.

---

<div class="post-metadata">

**Author:** ![dusty\_nv](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/dusty_nv/32/14043_2.png) [@dusty\_nv](https://forums.developer.nvidia.com/u/dusty_nv)\
**Post date:** [June 22, 2020, 4:08pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/5 "2020-06-22T16:08:06Z")

</div>

Hi @dkcog123, the DQN code from jetson-reinforcement is based on older version of PyTorch (0.3) and newer versions of PyTorch have breaking non-backwards-compatible changes. I believe the version of Gazebo has also updated with changes. I have added a disclaimer and some more recent resources about RL to the top of the readme, sorry about that -

[https://github.com/dusty-nv/jetson-reinforcement/blob/master/README.md](https://github.com/dusty-nv/jetson-reinforcement/blob/master/README.md)

---

<div class="post-metadata">

**Author:** ![therealleonmusk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/therealleonmusk/32/16457_2.png) [@therealleonmusk](https://forums.developer.nvidia.com/u/therealleonmusk)\
**Post date:** [June 24, 2020, 1:30am UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/6 "2020-06-24T01:30:42Z")

</div>

Ps in case I haven’t said it in a while “thanks dusty your an inspiration!” Between you and the team at Nvidia my business partner and I have started down the path of creating invercargill first maker space. We’ve collected approximately 80% of the tooling to build lab space for 3 trainers and 8 labs spaces. Setting it up with metal deposition stations, compressed air and vacuum feeds with 3.3, 5,12,and 20 volt supplies at each station an almost 1 cubic metre 3d printer that has a built in vacuum forming bed and a print farm of smaller bed higher definition 3d printers using recycled plastic from a friend’s e waste plant also doing training seminars and fitting each station with a jetson nano my business partner planning the details of that side we play to our strengths 😁! Sorry I digress exciting times but to put a bit of perspective on why I wanted you to know how much the world needs more people like you you have literally bought technology to the “A\*#'hole of the world!” Quote by Mick Jeager (apparently) just something to remind you guys your helping to make a better world and we thank you ☺️

---

<div class="post-metadata">

**Author:** ![dusty\_nv](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/dusty_nv/32/14043_2.png) [@dusty\_nv](https://forums.developer.nvidia.com/u/dusty_nv)\
**Post date:** [June 24, 2020, 4:32pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/7 "2020-06-24T16:32:41Z")

</div>

@therealleonmusk, thank you, it’s my pleasure. Great to hear about your maker space, that is very exciting! Great that you are bringing all that technology into the hands of others.

---

<div class="post-metadata">

**Author:** ![dkcog123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dkcog123](https://forums.developer.nvidia.com/u/dkcog123)\
**Post date:** [July 3, 2020, 1:48pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/9 "2020-07-03T13:48:36Z")

</div>

I am not 100% understand what you mean in your blog as follows:

**note** : this repo supports PyTorch v0.3 and JetPack 3.2. For newer examples, check out:  
- [`openai_ros` package](http://wiki.ros.org/openai_ros)  
- [`gym_gazebo2` repo](https://github.com/AcutronicRobotics/gym-gazebo2)  
- [`Isaac SDK` samples](https://docs.nvidia.com/isaac/isaac/packages/rl/doc/rl_dolly_docking.html)

I’ve briefly checked out each of the above. I cannot understand fully. They seem independent from “Deep Reinforcement Learning in Robotics.”

Is there any way to execute “Deep Reinforcement Learning in Robotics” on my new Xavier NX?

If not, how to create as a similar capability with Xavier NX?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![dusty\_nv](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/dusty_nv/32/14043_2.png) [@dusty\_nv](https://forums.developer.nvidia.com/u/dusty_nv)\
**Post date:** [July 3, 2020, 4:43pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/10 "2020-07-03T16:43:22Z")

</div>

> [@dkcog123](#):
>
> I’ve briefly checked out each of the above. I cannot understand fully. They seem independent from “Deep Reinforcement Learning in Robotics.”

Hi @dkcog123, each of those resources is using reinforcement learning in ROS (Robot Operating System) , the Gazebo robotics simulator, or Isaac robotics SDK.

> [@dkcog123](#):
>
> Is there any way to execute “Deep Reinforcement Learning in Robotics” on my new Xavier NX?

Not with the DQN code in that project as-is, because it was written for PyTorch 0.3. And with PyTorch 0.4 and newer, there were some breaking API changes in PyTorch such that it doesn’t run. Unfortunately PyTorch 0.3 can’t be installed on Xavier NX because it is older and uses older version of cuDNN than Xavier. The Gazebo simulator version has also updated (it is now Gazebo9).

You can try using this upstream PyTorch DQN learner that I believe should work with newer versions of PyTorch that can be installed on Xavier NX: [Reinforcement Learning (DQN) Tutorial — PyTorch Tutorials 2.11.0+cu130 documentation](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html)

---

<div class="post-metadata">

**Author:** ![dkcog123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dkcog123](https://forums.developer.nvidia.com/u/dkcog123)\
**Post date:** [July 7, 2020, 1:36am UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/11 "2020-07-07T01:36:58Z")

</div>

[https://pytorch.org/tutorials/intermediate/reinforcement\_q\_learning.h](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html)

I got another road block. Please help me. Thanks in advance!

**Here comes an error message:**

(This is run on **l4t-ml:r32.4.2-py3** container)

* * *

NoSuchDisplayException Traceback (most recent call last)  
 in   
37 env.reset()  
38 plt.figure()  
—\> 39 plt.imshow(get\_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),  
40 interpolation=‘none’)  
41 plt.title(‘Example extracted screen’)

in get\_screen()  
12 # Returned screen requested by gym is 400x600x3, but is sometimes larger  
13 # such as 800x1200x3. Transpose it into torch order (CHW).  
—\> 14 screen = env.render(mode=‘rgb\_array’).transpose((2, 0, 1))  
15 # Cart is in the lower half, so strip off the top and bottom of the screen  
16 \_, screen\_height, screen\_width = screen.shape

/usr/local/lib/python3.6/dist-packages/gym/envs/classic\_control/cartpole.py in render(self, mode)  
172  
173 if self.viewer is None:  
 → 174 from gym.envs.classic\_control import rendering  
175 self.viewer = rendering.Viewer(screen\_width, screen\_height)  
176 l, r, t, b = -cartwidth / 2, cartwidth / 2, cartheight / 2, -cartheight / 2

/usr/local/lib/python3.6/dist-packages/gym/envs/classic\_control/rendering.py in   
23  
24 try:  
—\> 25 from pyglet.gl import \*  
26 except ImportError as e:  
27 raise ImportError(‘’’

/usr/local/lib/python3.6/dist-packages/pyglet/gl/ **init**.py in   
242 # trickery is for circular import  
243 \_pyglet.gl = \_sys.modules[**name**]  
 → 244 import pyglet.window

/usr/local/lib/python3.6/dist-packages/pyglet/window/ **init**.py in   
1878 if not \_is\_pyglet\_doc\_run:  
1879 pyglet.window = sys.modules[**name**]  
 → 1880 gl.\_create\_shadow\_window()

/usr/local/lib/python3.6/dist-packages/pyglet/gl/ **init**.py in \_create\_shadow\_window()  
218  
219 from pyglet.window import Window  
 → 220 \_shadow\_window = Window(width=1, height=1, visible=False)  
221 \_shadow\_window.switch\_to()  
222

/usr/local/lib/python3.6/dist-packages/pyglet/window/xlib/ **init**.py in **init** (self, \*args, \*\*kwargs)  
163 self.\_event\_handlers[message] = func  
164  
 → 165 super(XlibWindow, self). **init** (\*args, \*\*kwargs)  
166  
167 global \_can\_detect\_autorepeat

/usr/local/lib/python3.6/dist-packages/pyglet/window/ **init**.py in **init** (self, width, height, caption, resizable, style, fullscreen, visible, vsync, display, screen, config, context, mode)  
568  
569 if not display:  
 → 570 display = pyglet.canvas.get\_display()  
571  
572 if not screen:

/usr/local/lib/python3.6/dist-packages/pyglet/canvas/ **init**.py in get\_display()  
92  
93 # Otherwise, create a new display and return it.  
—\> 94 return Display()  
95  
96

/usr/local/lib/python3.6/dist-packages/pyglet/canvas/xlib.py in **init** (self, name, x\_screen)  
121 self.\_display = xlib.XOpenDisplay(name)  
122 if not self.\_display:  
 → 123 raise NoSuchDisplayException(‘Cannot connect to “%s”’ % name)  
124  
125 screen\_count = xlib.XScreenCount(self.\_display)

NoSuchDisplayException: Cannot connect to “None”

---

<div class="post-metadata">

**Author:** ![dusty\_nv](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/dusty_nv/32/14043_2.png) [@dusty\_nv](https://forums.developer.nvidia.com/u/dusty_nv)\
**Post date:** [July 7, 2020, 5:33pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/12 "2020-07-07T17:33:57Z")

</div>

It seems like it is not able to find the display inside the container.

Try launching the container like this to enable the display inside the container:

```bash
$ sudo xhost +si:localuser:root
$ sudo docker run --runtime nvidia -it --rm --network host -e DISPLAY=$DISPLAY \
    -v /tmp/.X11-unix/:/tmp/.X11-unix \
    nvcr.io/nvidia/l4t-pytorch:r32.4.2-pth1.5-py3

```

If that still doesn’t work, you might want to try running this outside of container first.

---

<div class="post-metadata">

**Author:** ![dkcog123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dkcog123](https://forums.developer.nvidia.com/u/dkcog123)\
**Post date:** [July 8, 2020, 1:58am UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/13 "2020-07-08T01:58:47Z")

</div>

Fantastic! Thanks, Dusty!  
It worked!!!.

---

<div class="post-metadata">

**Author:** ![dkcog123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dkcog123](https://forums.developer.nvidia.com/u/dkcog123)\
**Post date:** [July 8, 2020, 7:09pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/14 "2020-07-08T19:09:38Z")

</div>

Now, I am trying to run outside of the L4t-ml container. I have installed  
– gym (this requires to install gfortran. Then it worked.)

Then I am getting another roadblock.  
So far, I have reinstalled libgl using the following command:  
– sudo apt-get install --reinstalllibgl1-mesa-dri

I don’t see any change. Both cases give the same error message below.

Thanks in advance!

> python3 reinforcement\_q\_learning.py  
> MESA-LOADER: failed to open swrast (search paths /usr/lib/aarch64-linux-gnu/dri:$${ORIGIN}/dri:/usr/lib/dri)  
> libGL error: failed to load driver: swrast  
> Traceback (most recent call last):  
> File “reinforcement\_q\_learning.py”, line 285, in   
> plt.imshow(get\_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),  
> File “reinforcement\_q\_learning.py”, line 260, in get\_screen  
> screen = env.render(mode=‘rgb\_array’).transpose((2, 0, 1))  
> File “/home/dave/.local/lib/python3.6/site-packages/gym/envs/classic\_control/cartpole.py”, line 174, in render  
> from gym.envs.classic\_control import rendering  
> File “/home/dave/.local/lib/python3.6/site-packages/gym/envs/classic\_control/rendering.py”, line 25, in   
> from pyglet.gl import \*  
> File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/ **init**.py”, line 244, in   
> import pyglet.window  
> File “/home/dave/.local/lib/python3.6/site-packages/pyglet/window/ **init**.py”, line 1880, in   
> gl.\_create\_shadow\_window()  
> File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/ **init**.py”, line 220, in \_create\_shadow\_window  
> \_shadow\_window = Window(width=1, height=1, visible=False)  
> File “/home/dave/.local/lib/python3.6/site-packages/pyglet/window/xlib/ **init**.py”, line 165, in **init**  
> super(XlibWindow, self). **init** (\*args, \*\*kwargs)  
> File “/home/dave/.local/lib/python3.6/site-packages/pyglet/window/ **init**.py”, line 591, in **init**  
> context = config.create\_context(gl.current\_context)  
> File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 204, in create\_context  
> return XlibContextARB(self, share)  
> File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 314, in **init**  
> super(XlibContext13, self). **init** (config, share)  
> File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 218, in **init**  
> raise gl.ContextException(‘Could not create GL context’)  
> pyglet.gl.ContextException: Could not create GL context

---

<div class="post-metadata">

**Author:** ![dusty\_nv](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/dusty_nv/32/14043_2.png) [@dusty\_nv](https://forums.developer.nvidia.com/u/dusty_nv)\
**Post date:** [July 8, 2020, 7:35pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/15 "2020-07-08T19:35:35Z")

</div>

Hmm if you reboot and then `sudo apt-get install mesa-utils`, are you able to run glxgears and glxinfo?

---

<div class="post-metadata">

**Author:** ![dkcog123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dkcog123](https://forums.developer.nvidia.com/u/dkcog123)\
**Post date:** [July 8, 2020, 8:46pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/16 "2020-07-08T20:46:01Z")

</div>

Yes, I reboot and installed mesa-utils.  
Yes, glxgears program runs.

However, I am still getting an error.  
Thus, i have installed pyglet by  
pip3 install pyglet.

Still I am getting the same error below.  
I’ve checked again the same DQL python code in the L4t-ml container with gym, python3-pk installed. Yes,it still works.

Please help. Thanks in advance!

$ python3 reinforcement\_q\_learning.py  
MESA-LOADER: failed to open swrast (search paths /usr/lib/aarch64-linux-gnu/dri:$${ORIGIN}/dri:/usr/lib/dri)  
libGL error: failed to load driver: swrast  
Traceback (most recent call last):  
File “reinforcement\_q\_learning.py”, line 285, in   
plt.imshow(get\_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),  
File “reinforcement\_q\_learning.py”, line 260, in get\_screen  
screen = env.render(mode=‘rgb\_array’).transpose((2, 0, 1))  
File “/home/dave/.local/lib/python3.6/site-packages/gym/envs/classic\_control/cartpole.py”, line 174, in render  
from gym.envs.classic\_control import rendering  
File “/home/dave/.local/lib/python3.6/site-packages/gym/envs/classic\_control/rendering.py”, line 25, in   
from pyglet.gl import \*  
File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/ **init**.py”, line 243, in   
import pyglet.window  
File “/home/dave/.local/lib/python3.6/site-packages/pyglet/window/ **init**.py”, line 1897, in   
gl.\_create\_shadow\_window()  
File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/ **init**.py”, line 220, in \_create\_shadow\_window  
\_shadow\_window = Window(width=1, height=1, visible=False)  
File “/home/dave/.local/lib/python3.6/site-packages/pyglet/window/xlib/ **init**.py”, line 173, in **init**  
super(XlibWindow, self). **init** (\*args, \*\*kwargs)  
File “/home/dave/.local/lib/python3.6/site-packages/pyglet/window/ **init**.py”, line 606, in **init**  
context = config.create\_context(gl.current\_context)  
File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 204, in create\_context  
return XlibContextARB(self, share)  
File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 314, in **init**  
super(XlibContext13, self). **init** (config, share)  
File “/home/dave/.local/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 218, in **init**  
raise gl.ContextException(‘Could not create GL context’)  
pyglet.gl.ContextException: Could not create GL context

---

<div class="post-metadata">

**Author:** ![dkcog123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@dkcog123](https://forums.developer.nvidia.com/u/dkcog123)\
**Post date:** [July 15, 2020, 11:11pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/17 "2020-07-15T23:11:30Z")

</div>

I have newly flashed SD Card, and I tested DQL Python code again in the same L4t-ml container. It worked!

I am still curious about why it did not work, but I am good now.  
I also watched the NX intro video, while building software today. Well done! I really enjoyed.

I plan to try out Isaac SDK 2020.1.

Thanks again, Dusty!

---

<div class="post-metadata">

**Author:** ![nadeemm](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nadeemm/32/367020_2.png) [@nadeemm](https://forums.developer.nvidia.com/u/nadeemm)\
**Post date:** [October 18, 2021, 5:53pm UTC](https://forums.developer.nvidia.com/t/deep-reinforcement-learning-in-robotics/128908/18 "2021-10-18T17:53:06Z")

</div>

Closed on October 18, 2021, 5:53pm UTC