Error on running 1080_balls_of_solitude and other simulations examples when using GPU pipeline

Autonomous Machines Robotics - Isaac Isaac Gym
1

I am new to Isaac Gym and tried to use it to run a simple simulation of a ball falling down due to gravity. However, when I set sim_params.use_gpu_pipeline = True, then I get the error pasted below. The same problem persists when I set the sim_params.use_gpu_pipeline = True in example file isaacgym/python/examples1080_balls_of_solitude.py. But the simulations run fine when the said flag is set to False.
Why is this happening?

On the other hand, the cartpole and ant examples run fine with or without GPU.
I’m using a GeForce RTX™ 2080.

I’ve attached my code (with the said flag set to True) and the corresponding command-line output below.

cl_op.txt (2.1 KB)
test_docs.py (2.8 KB)

[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3362
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3417
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3428
/buildAgent/work/45f70df4210b2e3e/source/physx/src/NpScene.cpp (4280) : invalid operation : PxScene::copyActorData, data has to be valid pointer.

[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 5620
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 926
[Error] [carb.gym.plugin] Failed to fill rigid body state tensor
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3442
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3492
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3336
/buildAgent/work/45f70df4210b2e3e/source/physx/src/NpScene.cpp (2869) : internal error : PhysX Internal CUDA error. Simulation can not continue!

/buildAgent/work/45f70df4210b2e3e/source/physx/src/NpScene.cpp (3175) : invalid operation : PxScene::fetchResults: fetchResults() called illegally! It must be called after advance() or simulate()
2

Hi

CUDA ver.?
Pytorch ver.?

3

Hi, here are the details:

CUDA version 11.4
Driver version 470.86
torch==1.9.1
torch-tb-profiler==0.3.1
torchaudio==0.9.1
torchvision==0.10.1

4

This is a bad guy.
How about changing to torch :1.8.2

My environment is below.
Driver Version: 495.44 CUDA Version: 11.5

I’m using RTX A6000.

5

The text in the screen recording you shared is hazy. Can you please confirm if you enabled the GPU pipeline?

Also, I tried changing to torch 1.8.2. Still the same errors. Any other ideas?

1 Like
6

I was also running on the cpu pipeline.
“sim_params.use_gpu_pipeline = False” doesn’t seem to change.

What you can do with the GPU pipeline is:

  • app_forces_at_pos.py
  • app_forces.py
  • franka_cube_ik_osc.py
  • franka_osc.py

Set “- -pipeline gpu” as an argument.

Screenshot from 2021-12-03 15-35-50
Screenshot from 2021-12-03 15-35-501920Ă—1080 335 KB

7

Some of the non-RL examples (including 1080_balls_of_solitude) are CPU only, so enabling the GPU pipeline on those will throw errors. You’ll see in the code in some cases that we’ll force the use_gpu_pipeline flag to be False.

1 Like
9

Why are some of the non-RL examples CPU only ? Are there some features that must be matching CPU tensors?

10

Only the tensor APIs can be run with GPU pipeline. Some of the non-RL examples were introduced before we added the tensor APIs, so thy won’t work on GPU if they have not been implemented with the tensor APIs.

1 Like