Multiprocessing error

system · February 17, 2019, 9:03pm

Hi, I couldn’t work with gym and baselines for a while now but couldn’t seem to find the solution to the problem.
I think the problem is with the multi-threading issue and I hope someone with expertise could help. I provided the details to replicate the issue.

To install stable-baselines, just clone the github repository and remove the requirements for tensorflow and opencv in setup.py

Here is the sample code for the cart pole problem:

import gym
import numpy as np

from stable_baselines.common.policies import MlpPolicy
from stable_baselines.common.vec_env import SubprocVecEnv
from stable_baselines.common import set_global_seeds
from stable_baselines import ACKTR

def make_env(env_id, rank, seed=0):
    """
    Utility function for multiprocessed env.

    :param env_id: (str) the environment ID
    :param num_env: (int) the number of environments you wish to have in subprocesses
    :param seed: (int) the inital seed for RNG
    :param rank: (int) index of the subprocess
    """
    def _init():
        env = gym.make(env_id)
        env.seed(seed + rank)
        return env
    set_global_seeds(seed)
    return _init

env_id = "CartPole-v1"
num_cpu = 4  # Number of processes to use
# Create the vectorized environment
env = SubprocVecEnv([make_env(env_id, i) for i in range(num_cpu)])

model = ACKTR(MlpPolicy, env, verbose=1)
model.learn(total_timesteps=25000)

obs = env.reset()
for _ in range(1000):
    action, _states = model.predict(obs)
    obs, rewards, dones, info = env.step(action)
    env.render()

Here is the error that I got:

[jetson-0423318029132:15691] *** Process received signal ***
[jetson-0423318029132:15691] Signal: Segmentation fault (11)
[jetson-0423318029132:15691] Signal code: Address not mapped (1)
[jetson-0423318029132:15691] Failing at address: 0x7f4e1cafc0
[jetson-0423318029132:15691] *** End of error message ***
[jetson-0423318029132:15692] *** Process received signal ***
[jetson-0423318029132:15692] Signal: Segmentation fault (11)
[jetson-0423318029132:15692] Signal code: Address not mapped (1)
[jetson-0423318029132:15692] Failing at address: 0x7f4e1cafc0
[jetson-0423318029132:15692] *** End of error message ***
[jetson-0423318029132:15690] *** Process received signal ***
[jetson-0423318029132:15690] Signal: Segmentation fault (11)
[jetson-0423318029132:15690] Signal code: Address not mapped (1)
[jetson-0423318029132:15690] Failing at address: 0x7f4e1cafc0
[jetson-0423318029132:15690] *** End of error message ***
[jetson-0423318029132:15693] *** Process received signal ***
[jetson-0423318029132:15693] Signal: Segmentation fault (11)
[jetson-0423318029132:15693] Signal code: Address not mapped (1)
[jetson-0423318029132:15693] Failing at address: 0x7f4e1cafc0
[jetson-0423318029132:15693] *** End of error message ***
Traceback (most recent call last):
  File "cartpole.py", line 36, in <module>
    env.render()
  File "/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/stable_baselines-2.4.1-py3.6.egg/stable_baselines/common/vec_env/subproc_vec_env.py", line 100, in render
    imgs = [pipe.recv() for pipe in self.remotes]
  File "/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/stable_baselines-2.4.1-py3.6.egg/stable_baselines/common/vec_env/subproc_vec_env.py", line 100, in <listcomp>
    imgs = [pipe.recv() for pipe in self.remotes]
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

AastaLLL · February 18, 2019, 1:59am

Hi,

Could you check if the app works without env.render()?
If yes, you may meet the same issue of this topic:
[url]https://devtalk.nvidia.com/default/topic/1045888/jetson-tx2/segmentation-fault-core-dumped-/post/5308381/#5308381[/url]

Please check the above comment for the solution.
Thanks.

system · February 18, 2019, 12:54pm

Yes, the app works without env.render()
I did try ./tegrastats and sha1sum -c /etc/nv_tegra_release without any problems

here are the errors occured when debugging with gdb

[jetson-0423318029132:13291] *** Process received signal ***
[jetson-0423318029132:13290] *** Process received signal ***
[jetson-0423318029132:13291] Signal: Segmentation fault (11)
[jetson-0423318029132:13291] Signal code: Address not mapped (1)
[jetson-0423318029132:13291] Failing at address: 0x7f51a72fc0
[jetson-0423318029132:13290] Signal: Segmentation fault (11)
[jetson-0423318029132:13290] Signal code: Address not mapped (1)
[jetson-0423318029132:13290] Failing at address: 0x7f51a72fc0
[jetson-0423318029132:13289] *** Process received signal ***
[jetson-0423318029132:13289] Signal: Segmentation fault (11)
[jetson-0423318029132:13289] Signal code: Address not mapped (1)
[jetson-0423318029132:13289] Failing at address: 0x7f51a72fc0
[jetson-0423318029132:13292] *** Process received signal ***
[jetson-0423318029132:13292] Signal: Segmentation fault (11)
[jetson-0423318029132:13292] Signal code: Address not mapped (1)
[jetson-0423318029132:13292] Failing at address: 0x7f51a72fc0
[jetson-0423318029132:13291] *** End of error message ***
[jetson-0423318029132:13289] *** End of error message ***
[jetson-0423318029132:13290] *** End of error message ***
[jetson-0423318029132:13292] *** End of error message ***
Traceback (most recent call last):
  File "cartpole.py", line 36, in <module>
    env.render()
  File "/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/stable_baselines-2.4.1-py3.6.egg/stable_baselines/common/vec_env/subproc_vec_env.py", line 100, in render
    imgs = [pipe.recv() for pipe in self.remotes]
  File "/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/stable_baselines-2.4.1-py3.6.egg/stable_baselines/common/vec_env/subproc_vec_env.py", line 100, in <listcomp>
    imgs = [pipe.recv() for pipe in self.remotes]
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
[Thread 0x7faf3af1f0 (LWP 13287) exited]
[Thread 0x7faebae1f0 (LWP 13285) exited]
[Thread 0x7f36ffd1f0 (LWP 13307) exited]
[Thread 0x7f377fe1f0 (LWP 13306) exited]
[Thread 0x7f37fff1f0 (LWP 13305) exited]
[Thread 0x7f367fc1f0 (LWP 13308) exited]
[Thread 0x7f3cffd1f0 (LWP 13304) exited]
[Thread 0x7f35ffb1f0 (LWP 13309) exited]
[Thread 0x7f3d7fe1f0 (LWP 13303) exited]
[Thread 0x7fb7ff6010 (LWP 13251) exited]
[Thread 0x7f17fff1f0 (LWP 13345) exited]
[Thread 0x7f4affd1f0 (LWP 13299) exited]
[Thread 0x7fb0bb01f0 (LWP 13288) exited]
[Thread 0x7f514111f0 (LWP 13295) exited]
[Thread 0x7f4bfff1f0 (LWP 13297) exited]
[Thread 0x7fb23b11f0 (LWP 13293) exited]
[Thread 0x7f488531f0 (LWP 13301) exited]
[Thread 0x7f3dfff1f0 (LWP 13302) exited]
[Thread 0x7f50c101f0 (LWP 13296) exited]
[Thread 0x7f490541f0 (LWP 13300) exited]
[Thread 0x7f51c121f0 (LWP 13294) exited]
[Inferior 1 (process 13251) exited with code 01]

system · February 18, 2019, 12:55pm

sudo pip3 install pyglet==1.3.1
sudo sed -i ‘s/_have_getprocaddress = True/_have_getprocaddress = False/’ /usr/local/lib/python3.6/dist-packages/pyglet/gl/lib_glx.py

also does not work

system · February 19, 2019, 1:57am

So, I still couldn’t figure out the problem so I built openmpi from source and tried to run the other guy’s code,

[code]import gym
import faulthandler

faulthandler.enable()

env = gym.make(‘CartPole-v0’)
for i_episode in range(20):
observation = env.reset()
for t in range(100):
env.render()
print(observation)
action = env.action_space.sample()
observation, reward, done, info = env.step(action)
if done:
print(“Episode finished after {} timesteps”.format(t+1))
break
[/code]

here is the error

[code]mpirun python cartpole1.py
Fatal Python error: Segmentation fault

Current thread 0x0000007f885ba010 (most recent call first):
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/lib_glx.py”, line 74 in link_GL
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/glx.py”, line 440 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 16 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/init.py”, line 221 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py”, line 23 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py”, line 150 in render
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/core.py”, line 237 in render
File “cartpole1.py”, line 10 in
Fatal Python error: Segmentation fault

Current thread 0x0000007f8c252010 (most recent call first):
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/lib_glx.py”, line 74 in link_GL
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/glx.py”, line 440 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 16 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/init.py”, line 221 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py”, line 23 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py”, line 150 in render
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/core.py”, line 237 in render
File “cartpole1.py”, line 10 in
Fatal Python error: Segmentation fault

Current thread 0x0000007f8658a010 (most recent call first):
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/lib_glx.py”, line 74 in link_GL
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/glx.py”, line 440 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 16 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/init.py”, line 221 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py”, line 23 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py”, line 150 in render
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/core.py”, line 237 in render
File “cartpole1.py”, line 10 in
Fatal Python error: Segmentation fault

Current thread 0x0000007f94e4a010 (most recent call first):
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/lib_glx.py”, line 74 in link_GL
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/glx.py”, line 440 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 16 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/init.py”, line 221 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py”, line 23 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py”, line 150 in render
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/core.py”, line 237 in render
File “cartpole1.py”, line 10 in
Fatal Python error: Segmentation fault

Current thread 0x0000007fa99b6010 (most recent call first):
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/lib_glx.py”, line 74 in link_GL
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/glx.py”, line 440 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 16 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/init.py”, line 221 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py”, line 23 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py”, line 150 in render
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/core.py”, line 237 in render
File “cartpole1.py”, line 10 in
Fatal Python error: Segmentation fault

Current thread 0x0000007fa9def010 (most recent call first):
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/lib_glx.py”, line 74 in link_GL
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/glx.py”, line 440 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 16 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/init.py”, line 221 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py”, line 23 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py”, line 150 in render
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/core.py”, line 237 in render
File “cartpole1.py”, line 10 in
Fatal Python error: Segmentation fault

Current thread 0x0000007faf4f9010 (most recent call first):
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/lib_glx.py”, line 74 in link_GL
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/glx.py”, line 440 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 16 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/init.py”, line 221 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py”, line 23 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py”, line 150 in render
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/core.py”, line 237 in render
File “cartpole1.py”, line 10 in
Fatal Python error: Segmentation fault

Current thread 0x0000007f85dca010 (most recent call first):
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/lib_glx.py”, line 74 in link_GL
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/glx.py”, line 440 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/xlib.py”, line 16 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/pyglet/gl/init.py”, line 221 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/rendering.py”, line 23 in
File “”, line 219 in _call_with_frames_removed
File “”, line 678 in exec_module
File “”, line 665 in _load_unlocked
File “”, line 955 in _find_and_load_unlocked
File “”, line 971 in _find_and_load
File “”, line 219 in _call_with_frames_removed
File “”, line 1023 in _handle_fromlist
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/envs/classic_control/cartpole.py”, line 150 in render
File “/home/nvidia/.virtualenvs/py3/lib/python3.6/site-packages/gym/core.py”, line 237 in render
File “cartpole1.py”, line 10 in

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

mpirun noticed that process rank 5 with PID 0 on node jetson-0423318029132 exited on signal 11 (Segmentation fault).
[/code]

system · February 19, 2019, 1:59am

Here is the error in gdb debugging

gdb python3
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...(no debugging symbols found)...done.
(gdb) run cartpole1.py
Starting program: /home/nvidia/.virtualenvs/py3/bin/python3 cartpole1.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fb6bb41f0 (LWP 8435)]
[New Thread 0x7fb53b31f0 (LWP 8436)]
[New Thread 0x7fb3bb21f0 (LWP 8437)]
[New Thread 0x7fb23b11f0 (LWP 8438)]
[New Thread 0x7fb0bb01f0 (LWP 8439)]
[New Thread 0x7faf3af1f0 (LWP 8440)]
[New Thread 0x7fadbae1f0 (LWP 8441)]
[Thread 0x7faf3af1f0 (LWP 8440) exited]
[Thread 0x7fadbae1f0 (LWP 8441) exited]
[Thread 0x7fb6bb41f0 (LWP 8435) exited]
[Thread 0x7fb0bb01f0 (LWP 8439) exited]
[Thread 0x7fb23b11f0 (LWP 8438) exited]
[Thread 0x7fb3bb21f0 (LWP 8437) exited]
[Thread 0x7fb53b31f0 (LWP 8436) exited]

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x0000007fb6a5b670 in ?? () from /usr/lib/aarch64-linux-gnu/libGLX.so.0

system · February 19, 2019, 2:03am

sha1sum -c /etc/nv_tegra_release
/usr/lib/aarch64-linux-gnu/tegra/libnvosd.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_camera.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvomx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_force.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnveventlib.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_utils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libglx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvexif.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtx_helper.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketserver.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvscf.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnetstorehdfx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_parser.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_contentpipe.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_utils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_gpucompute.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_tbc.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvos.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtnr.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_graphics.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvimp.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvphs.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvavp.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcapture.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnetstoredefog.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_il.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtracebuf.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvidia-egl-wayland.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdc.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_boot.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_generic.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvapputil.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcameratools.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcam_imageencoder.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnveglstream_camconsumer.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_utils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvomxilclient.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvwinsys.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvargus.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_spincircle.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvphsd.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnveglstreamproducer.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_runtime.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvll.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcolorutil.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtestresults.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcamv4l2.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcamerautils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcamlog.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvparser.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_core.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_os.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_ui.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so: OK
/usr/lib/aarch64-linux-gnu/libv4l/plugins/libv4l2_nvvidconv.so: OK
/usr/lib/aarch64-linux-gnu/libv4l/plugins/libv4l2_nvvideocodec.so: OK
/usr/lib/xorg/modules/drivers/nvidia_drv.so: OK
/usr/lib/xorg/modules/extensions/libglx.so: OK

system · February 19, 2019, 2:06am

using cart pole simplest code base

import gym
env = gym.make('CartPole-v0')
env.reset()
for _ in range(1000):
    env.render()
    env.step(env.action_space.sample()) # take a random action

here is the error

Starting program: /home/nvidia/.virtualenvs/py3/bin/python3 cartpolesim.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fb6bb41f0 (LWP 8497)]
[New Thread 0x7fb63b31f0 (LWP 8498)]
[New Thread 0x7fb4bb21f0 (LWP 8499)]
[New Thread 0x7fb23b11f0 (LWP 8500)]
[New Thread 0x7fb1bb01f0 (LWP 8502)]
[New Thread 0x7fb03af1f0 (LWP 8503)]</li>

[New Thread 0x7fadbae1f0 (LWP 8504)]
[Thread 0x7fb63b31f0 (LWP 8498) exited]
[Thread 0x7fb03af1f0 (LWP 8503) exited]
[Thread 0x7fadbae1f0 (LWP 8504) exited]
[Thread 0x7fb1bb01f0 (LWP 8502) exited]
[Thread 0x7fb23b11f0 (LWP 8500) exited]
[Thread 0x7fb6bb41f0 (LWP 8497) exited]
[Thread 0x7fb4bb21f0 (LWP 8499) exited]

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x0000007fb6a5b670 in ?? () from /usr/lib/aarch64-linux-gnu/libGLX.so.0

system · February 19, 2019, 2:25am

installed gym using steps in https://github.com/openai/gym

Due to the required dependency of a newer gym in the stuff I am working on, is there any way to fix this issue without downgrading gym to a lower version that jetson-reinforcement github uses?

AastaLLL · February 21, 2019, 9:04am

Hi,

Thanks for your update.

Just want to confirm first: do you test it with virtualenv.
We have found some different behavior inside the virtual environment.

Thanks.

system · February 21, 2019, 12:29pm

Well, I found a way around the issue.

There are some still some occasional output of errors (pyglet &
Thread 1 “python3” received signal SIGSEGV, Segmentation fault.
0x0000007f54080670 in ?? () from /usr/lib/aarch64-linux-gnu/libGLX.so.0 )

but either way it works.

so here is what I did:

add library path manually into the ~/.bashrc file

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra-egl:/usr/lib/aarch64-linux-gnu:/usr/local/lib/aarch64-linux-gnu:/usr/local/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export PATH=/usr/local/cuda/bin:$PATH

downgrade pyglet to version 1.2.4
I tested on 1.3.0 too and it worked, but just to be safe use 1.2.4

for gpu threading refer to
[url]
https://devtalk.nvidia.com/default/topic/1044858/tensorflow-integrate-with-baselines-package-in-openai-gym/?offset=9[/url]
but with numpy==1.16

from there I stole their code and modify it for my purpose. To disable logging just delete logs in the code. I still have a vague understanding of what this does but here:

[code]import os
import platform
import numpy as np
import tensorflow as tf

import gym
import multiprocessing

from baselines import logger
from mpi4py import MPI
from tensorflow.python.client import device_lib

def guess_available_gpus(n_gpus=None):

local_device_protos = device_lib.list_local_devices()
return [x.name for x in local_device_protos if x.device_type == 'GPU']

def setup_mpi_gpus():
“”"
Set CUDA_VISIBLE_DEVICES using MPI.
“”"
available_gpus = guess_available_gpus()

node_id = platform.node()
nodes_ordered_by_rank = MPI.COMM_WORLD.allgather(node_id)
processes_outranked_on_this_node = [n for n in nodes_ordered_by_rank[:MPI.COMM_WORLD.Get_rank()] if n == node_id]
local_rank = len(processes_outranked_on_this_node)
os.environ['CUDA_VISIBLE_DEVICES'] = str(available_gpus[local_rank])

def guess_available_cpus():
return int(multiprocessing.cpu_count())

def setup_tensorflow_session():
num_cpu = guess_available_cpus()

tf_config = tf.ConfigProto(
    inter_op_parallelism_threads=num_cpu,
    intra_op_parallelism_threads=num_cpu
)
return tf.Session(config=tf_config)

def get_experiment_environment():

setup_mpi_gpus()
logger_context = logger.scoped_configure(dir=None,
                                         format_strs=['stdout', 'log',
                                                      'csv'] if MPI.COMM_WORLD.Get_rank() == 0 else ['log'])
tf_context = setup_tensorflow_session()
return logger_context, tf_context

log, tf_sess = get_experiment_environment()
env = gym.make(‘CartPole-v0’)
env.reset()
with log, tf_sess:
for _ in range(1000):
env.render()
env.step(env.action_space.sample()) # take a random action
[/code]

AastaLLL · February 25, 2019, 4:59am

Good to know it works!

Topic		Replies	Views
Multiprocessing error Jetson AGX Xavier	2	435	February 20, 2019
segmentation fault (core dumped) Jetson TX2	13	3211	October 18, 2021
Off load task from CPU to GPU Jetson Nano camera , ubuntu , gstreamer , gpu , sdk	77	3870	December 22, 2021
Issues Installing pytorch through jetson inference install script Jetson Nano pytorch	7	1296	October 16, 2023
Nvarguscamerasrc jetpack-4.3, No cameras available, nvbuf_utils: Could not get EGL display connection, requirements Jetson TX2 camera	39	3624	October 18, 2021
Dual NVIDIA P600 BaseMosaic freezes system if booting from UEFI with 390.42 during X restart Linux	30	3363	July 4, 2018
Jetson Nano Development Kits Stopped Booting Jetson Nano boot	13	738	June 21, 2023
Failed to initialize the NVIDIA graphics device! Jetson Xavier NX	45	3968	November 9, 2022
Jetson TX2 kernel crash on tegra_xudc_gadget_pullup Jetson TX2 kernel , usb , nvbugs	13	1042	October 18, 2021
Add third party vc mipi nvidia module, how to add porting? Jetson Orin Nano camera	14	105	August 29, 2024

Multiprocessing error

Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted.

Related topics

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.