External Extensions: OpenXR compact binding for creating extended reality applications

Omniverse Isaac Sim
,
1

omni.add_on.openxr: OpenXR compact binding for creating extended reality applications on NVIDIA Omniverse

This extension provides a compact python binding (on top of the open standard OpenXR for augmented reality (AR) and virtual reality (VR)) to create extended reality applications taking advantage of NVIDIA Omniverse rendering capabilities. In addition to updating views (e.g., head-mounted display), it enables subscription to any input event (e.g., controller buttons and triggers) and execution of output actions (e.g., haptic vibration) through a simple and efficient API for accessing conformant devices such as HTC Vive, Oculus and others…

GitHub links:


2

Amazing. I unzipped the folder in C:\Users\Garrett\AppData\Local\ov\pkg\create-2021.3.7\exts and restarted CREATE.
It loaded the extension in the window>extensions however there is a red triangle warning sign.

image
image1138×777 158 KB

I hover over it and don’t know what that means.
Loaded improperly?
Restarted Create > looks like it works.
Selected AutoLoad > restarted create.
It did autoload.
The Add On (top menu bar) is not loading to select anything as per github instructions. Maybe I have to load add-ons elsewhere?
image
image789×528 105 KB

3

Hi @garretthoyos

It looks like you are using Windows …
Currently this extension has been compiled only for Linux systems.

4

😭 😭 😭 😭 😭 😭 😭 #virtual reality

5

Is there a way to allow me to use Omniverse Create in VR (preferably on Windows) using Oculus link or Airlink?

6

Mmmm,

The only I can say (a common Omniverse user) is that this extension allows you to use any OpenXR-conformant devices (as shown in OpenXR Overview - The Khronos Group Inc)…

7

With respect to Windows, maybe you can try to compile it

8

You don’t work for Nvidia?

9

Nop

10

come work for us. :)

11

👀

12

This will be handy, but I am getting the following error when I run the accompanying code in the script editor. Any help would be appreciated.

IsaacSim pkg: 2021.2.1
OS: Ubuntu 18.04
SteamVR: 1.20.4

Error:

2021-12-29 23:37:48 [Error] [carb.physx.python] AttributeError: ‘NoneType’ object has no attribute ‘GetPropertyNames’

At:
omni/add_on/openxr/openxr.py(823): _openxr.OpenXR.teleport_prim
executing: Python 0…(48): simStep

testSteamVRInterface.py (2.0 KB)

13

Hi @j.tigue

This error is related to a bad rendering configuration.
Unfortunately I don’t have access to the VR equipment until I return from vacation for the end of the year. Until then it is impossible for me to test the code :(

However, there are some things you can try and show here in order to solve this issue…

  • Test the return of the main functions

    The main functions that take care of configuring the system return a boolean value.
    Then it is posible to know what configuration step is wrong…

    print("xr.init:", xr.init())
print("xr.create_instance:", xr.create_instance())
print("xr.get_system:", xr.get_system())
...
print("xr.create_session:", xr.create_session())

  • Inspect output messages from the extension on the terminal

    Messages related to the configuration and use of this extension should appear in the terminal where Isaac Sim is running. To get more detailed output, set the following environment variables in the terminal prior to running Isaac Sim

    export XR_LOADER_DEBUG=all
export VR_LOG_DEBUG=1
14

Thanks for the assist. One other thing I should note is that I am running with a “null” headset driver trying to just use the trackers/controllers:

##----------------------------Below are the outputs:
print(“Init:”,xr.init())
print(“Create Instance:”,xr.create_instance())
print(“Get System:”,xr.get_system())
print(“Create Session:”,xr.create_session())

##--------------------------------Output:
Init: True
Create Instance: True
Get System: True
Create Session: True

##----------------------------Extension Loading:
[8.160s] [ext: omni.add_on.openxr-0.0.1] startup
[8.190s] app started
[15.128s] Isaac Sim App is loaded.
[15.185s] Checking for Isaac Sim assets on Nucleus
[15.240s] Nucleus detected successfully: omniverse://localhost
[INFO] OpenXR initialized using pybind11 interface
Init: True
OpenXR API layers (0)
OpenXR extensions (13)
|-- XR_KHR_vulkan_enable
|-- XR_KHR_vulkan_enable2
|-- XR_KHR_opengl_enable
| (requested)
|-- XR_KHR_binding_modification
|-- XR_VALVE_analog_threshold
|-- XR_EXT_hand_tracking
|-- XR_EXT_hand_joints_motion_range
|-- XR_EXT_hp_mixed_reality_controller
|-- XR_HTC_vive_cosmos_controller_interaction
|-- XR_KHR_visibility_mask
|-- XR_UNITY_hand_model_pose
|-- XR_KHR_composition_layer_depth
|-- XR_EXT_debug_utils
Create Instance: True
Runtime
|-- name: SteamVR/OpenXR
|-- version: 0.1.0
System
|-- system id: 1153031043452764510
|-- system name: SteamVR/OpenXR : null
|-- vendor id: 10462
|-- max layers: 16
|-- max swapchain height: 2056
|-- max swapchain width: 1852
|-- orientation tracking: 1
|-- position tracking: 1
View configurations (1)
|-- type 2 (The OpenXR Specification)
| (requested)
View configuration properties
|-- configuration type: 2 (The OpenXR Specification)
|-- fov mutable (bool): 1
View configuration views (2)
|-- view 0
| |-- recommended resolution: 1852 x 2056
| |-- max resolution: 1852 x 2056
| |-- recommended swapchain samples: 1
| |-- max swapchain samples: 1
|-- view 1
| |-- recommended resolution: 1852 x 2056
| |-- max resolution: 1852 x 2056
| |-- recommended swapchain samples: 1
| |-- max swapchain samples: 1
Environment blend modes (1)
|-- mode: 1 (The OpenXR Specification)
| (requested)
Get System: True
OpenGL requirements
|-- min API version: 4.3.0
|-- max API version: 4.6.0
Graphics binding: OpenGL
|-- xDisplay: 0x1dfe0ac0
|-- visualid: 0
|-- glxFBConfig: 0
|-- glxDrawable: 121634827
|-- glxContext: 0x1dfc0208
Reference spaces (3)
|-- type: 1 (The OpenXR Specification)
| |-- reference space bounds
| | |-- width: 0
| | |-- height: 0
|-- type: 2 (The OpenXR Specification)
| |-- reference space bounds
| | |-- width: 0
| | |-- height: 0
|-- type: 3 (The OpenXR Specification)
| |-- reference space bounds
| | |-- width: 1
| | |-- height: 1
Suggested interaction bindings by profiles
|-- /interaction_profiles/khr/simple_controller (1)
|-- /interaction_profiles/google/daydream_controller (1)
|-- /interaction_profiles/htc/vive_controller (2)
|-- /interaction_profiles/htc/vive_pro (0)
|-- /interaction_profiles/microsoft/motion_controller (2)
|-- /interaction_profiles/microsoft/xbox_controller (0)
|-- /interaction_profiles/oculus/go_controller (1)
|-- /interaction_profiles/oculus/touch_controller (2)
|-- /interaction_profiles/valve/index_controller (2)
Swapchain formats (8)
|-- format: 32859
|-- format: 34842
| (selected)
|-- format: 34843
|-- format: 35905
|-- format: 35907
|-- format: 33189
|-- format: 33190
|-- format: 33191
Created swapchain (2)
|-- swapchain: 0
| |-- width: 1852
| |-- height: 2056
| |-- sample count: 1
| |-- swapchain images: 3
|-- swapchain: 1
| |-- width: 1852
| |-- height: 2056
| |-- sample count: 1
| |-- swapchain images: 3
Create Session: True

15

Hi @j.tigue

According to the logs, the configuration looks good.
Regarding the use of the HTC Vive Trackers without Headset, this blog refers to OpenVR, however, the omni.add_on.openxr extension is built using the OpenXR standard.

According to the OpenXR standard, it is necessary to call the render loop in order to update the actions (controller pose, for example)*.

Then, you can create an empty render function to change the default frame loop behaviour. By doing this, the graphics handler will not be called and no graphics will be rendered.

def render_callback(num_views, views, configuration_views):
     pass

xr.subscribe_render_event(render_callback)

I will have access to VR equipment next week.
By the way, are you able to run the default example (code shown in GitHub) using the full VR equipment (with the headset)?

*Note: I remember I read and tested it but I can not find the right reference in the specification right now. The closest reference I can found is the OpenXR session-lifecycle

16

Also, your configuration refers to a stereo configuration (XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO):

View configurations (1)
|-- type 2 (https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#XrViewConfigurationType)

However, in the python code you provided you are calling the setup_mono_view instead of the setup_stereo_view method

17

@toni.sm

That did it. After I added the render_callback (and subscription) and changed to setup_stereo_view it worked.

As far as the HTC Vive Trackers without Headset I was able to operate without the headset just using the normal HTC vive controllers. OpenXR using the blog-post documentation.

Here is code that was working:
testSteamVRInterface.py (2.5 KB)

Thanks for all of your help!

18

Hi @j.tigue

Glad to hear it worked 😁

Regarding the next quote, I could not fully understand 🙈. Could you please provide more details about it? Did you get the trackers pose? What code did you use for it? Using OpenXR and not OpenVR?

19

@toni.sm
I was able to get either the HTC Vive Trackers or the HTC Vive Controllers connected to SteamVR without needing a headset (goggles/HMD) which operates with OpenXR. As the blog pose suggests you need to make a few changes to some setting files:

  1. in “SteamDirectory\steamapps\common\SteamVR\drivers\null\resources\settings\default.vrsettings”
    “enable”: true,

  2. “SteamDirectory\config\steamvr.vrsettings”
    “forcedDriver”: “null”,
    “activateMultipleDrivers”: “true”,

  3. “SteamDirectory\steamapps\common\SteamVR\resources\settings\default.vrsettings”
    “requireHmd”: false,
    “forcedDriver”: “null”,
    “activateMultipleDrivers”: true,

This allows the tracking of controllers or trackers without the use of a headset. A virtual headset is still displayed on the monitor but it can be neglected. I am still dealing with calibration of the workspace issues without the headset but it is currently tracking based on one of the base stations as the origin.

The openXR binding extension (that you created/provided) connects the the SteamVR/OpenXR information to Isaac Sim. So far I am only connecting the controller pose/trigger/haptics and not the tracker pose.

2 Likes
20

Thank you for clarifying 😁