Hello! I’m currently working on a use case that requires using the Camera API of IsaacGym. However, the use case also requires the code to be run in headless mode. However, when I try to create a camera sensor, I get the following error:
[Error] [carb.gym.plugin] SetCameraLocation: Error: could not find camera with handle -1 in environment 0
For reference, I’m using IsaacGym Preview 2 on Ubuntu 18.04.5 LTS. Here is the code snippet that I’m talking about - any advice is preferred!
cam_props = gymapi.CameraProperties()
cam_props.width = self.cam_width
cam_props.height = self.cam_height
cam_props.enable_tensors = True
cam_handle = self.gym.create_camera_sensor(env, cam_props) # This is the line that has the error
print(f"Cam Handle: {cam_handle}") # Prints -1
self.gym.set_camera_location(cam_handle, env, gymapi.Vec3(1.2, 0, 2.5), gymapi.Vec3(0, 0, 1.2))
self.cams.append(cam_handle)
# obtain camera tensor
cam_tensor = self.gym.get_camera_image_gpu_tensor(self.sim, env, cam_handle, gymapi.IMAGE_COLOR)
# wrap camera tensor in a pytorch tensor
torch_cam_tensor = gymtorch.wrap_tensor(cam_tensor)
self.cam_tensors_np.append(torch_cam_tensor)
print("Got camera tensor with shape", cam_tensor.shape)
print(" Torch camera tensor device:", torch_cam_tensor.device)
print(" Torch camera tensor shape:", torch_cam_tensor.shape)
Hi, your code looks very similar to what we have in our interop_torch.py example, which should work in headless mode. Can you confirm whether you are able to run that example in headless mode? You can use python interop_torch.py --headless.
Thanks for the reply! I tried running that command and unfortunately got the below error:
Importing module 'gym_37' (<path-to-directory>/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to <path-to-directory>/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.8.1+cu102
Device count 1
<path-to-directory>isaacgym/python/isaacgym/_bindings/src/gymtorch
Using<path-to-home>/varunl/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file <path-to-home>/varunl/.cache/torch_extensions/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
Segmentation fault
Is there anything that I’m doing immediately wrong?
For the record, the crash appears to be happening when sim = gym.create_sim(args.compute_device_id, args.graphics_device_id, args.physics_engine, sim_params) is called. Here, both the compute and graphics device IDs are 0, the physics engine is PhysX, and the sim params are as default. The code is running on a Tesla P100-PCIE-12GB which has been verified to work with cuda by using torch.cuda.is_available()
i have the same issue with one of our machines, which has two NVIDIA Quadro P5000. The interop_torch.py example causes segmentation faults, as soon as create_sim() is called. Curiously enough, the example works fine with a much smaller machine (laptop with a NVIDIA T500). I hope you can shed some light into the problem.
Having more than one GPU in the machine could the cause of the issue. Does it work for you if you force it to just have 1 GPU available with set CUDA_VISIBLE_DEVICES?
Hello, I am running into this problem right now. When I run create_sim with graphics_device_id=-1 then I the error: SetCameraLocation: Error: could not find camera with handle -1 in environment 0 , if I run it with any other ID set then I get a segmentation fault.
I also realized that if I set cam_props.enable_tensors=False then I don’t get a segmentation fault anymore.
Oh also changing CUDA_VISIBLE_DEVICES didn’t really make any difference.
Is there any solution to this problem, any help is very much appreciated!