How to use omni.kit.xr.profile.vr extension with IsaacSim 4.5?

[mattmazzola]

Isaac Sim Version

4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

• Model: RTX 4090
• Driver Version: 560.94

Topic Description

Detailed Description

I have an Isaac Sim extension using VR working with version 4.2.
I would like to use the latest 4.5; however, the extension crashes on load to breaking changes in the extensions: Renaming Extensions in Isaac Sim — Isaac Sim Documentation

Steps to Reproduce

1. Enable VR extension
2. Open user extension
3. See error in console

Error Messages

2025-02-03 22:34:08  [Error] [omni.kit.app._impl]   File "d:/repos/project-echelon/omniverse_assets/DemonstrationCollector/demo_collector/vr_controllers/xr/control.py", line 17, in <module>
2025-02-03 22:34:08  [Error] [omni.kit.app._impl]     from omni.kit.xr.core import XRCore, XRDevice, XRGestureEventType
2025-02-03 22:34:08  [Error] [omni.kit.app._impl] ImportError: cannot import name 'XRDevice' from 'omni.kit.xr.core' (c:/users/mattm/appdata/local/ov/pkg/isaac-sim-4.5.0/extscache/omni.kit.xr.core-106.5.13+106.5.0.wx64.r.cp310/omni/kit/xr/core/__init__.py)
2025-02-03 22:39:17  [Error] [gpu.foundation.plugin] Couldn't process https://dw290v42wisod.cloudfront.net/exts/kit/community/3/./resources/8e544b8998e3c2dd910546e070c43ca9.png, it might not have written completely. Reason: STB Failed to load image info: Image not of any known type, or corrupt
2025-02-03 22:39:17  [Error] [omni.ui] Failed to upload UI Image 'https://dw290v42wisod.cloudfront.net/exts/kit/community/3/./resources/8e544b8998e3c2dd910546e070c43ca9.png'

Screenshots or Videos

N/A

Additional Information

What I’ve Tried

N/A

Related Issues

N/A

Additional Context

N/A

[ltorabi]

Please check out the release note for ant related changes

[mattmazzola]

check out the release note for ant related changes

It would be helpful if you could provide a link to the release note you want me to check out.
There are many release notes, and it is not clear what you mean.

Is “ant” a typo or is it intentional and a protocol used in the VR extension?
I’ve only known ANT as a wireless device protocol; however, we are using the VR device with USB3.0 connection so this does not seem like a limitation.

I have reviewed the Isaac Sim 4.5 Release Notes and it only says that the extension was “changed”. If it was unsupported I think it would be in a different category or more explicitly highlighted.

[Simplychenable]

@mattmazzola i am just another user, and i think it’s a typo (“any” instead of “ant”) but i’ve been wrong before.

regarding the VR question, personally don’t use VR so feel free to take my input as a grain of salt. that said, i did come across a thread about one of the kit params that was changed in 106.5:

VR Stopped Working with Kit 106.5

I think I found the issue (not really an issue). The default VR system changed from SteamVR to CloudXR. [image] So! When I ran it, nothing happened because it wasn’t trying to go through SteamVR anymore. Running it after v106.4 probably cached SteamVR as the last used system. Is there a way to keep track of these changes? With the kit 106.4 update, a function in XR Core was renamed, and I didn’t see any mention of it in the changelog. Finding that I had to rename this function took a while to…

i wonder if it could be related in your case?

[mattmazzola]

i think it’s a typo (“any” instead of “ant”)

Ah, yea, probably correct. Although that makes the moderator’s message more generic about any release note which is even more meaningless.

Given the other silly errors I’ve found in I saw the error

Couldn’t process https://dw-removed-a9.png, it might not have written completely. Reason: STB Failed to load image info: Image not of any known type, or corrupt

And assumed it was some corrupt package. However, after your mention of “kit params” caused more investigation.

I see the change from Isaac Sim 4.2 to 4.5 implicitly causes update of omni.kit.xr.profile.vr extension 106.1.24 → 106.5.13

I tried to find the release notes for new version of the VR extension but could not. Closest was Kit 106.5.0 Release Notes — Omniverse Developer Guide which doesn’t mention.

From digging through some of the code I see

...\isaac-sim-4.5.0\extscache\omni.kit.xr.core-106.5.13+106.5.0.wx64.r.cp310\omni\kit\xr\core\scripts\__init__.py

exports XRInputDevice from

...\isaac-sim-4.5.0\extscache\omni.kit.xr.core-106.5.13+106.5.0.wx64.r.cp310\omni\kit\xr\core\scripts\xr_class_wrappers\xrinputdevice_class_wrapper.py

Perhaps XRDevice was renamed XRInputDevice?
However, this would be a breaking change and cause for major version increase such as 107.0.0.
I have to finish up some other things, but I will test this later.

[dvelazco]

There have been quite a few changes from version to version that have broken working configurations. Update 106.3 → 106.4 had function renames and, from my other thread, 106.4 → 106.5 had defaults changed. There was probably more, but those were the ones that affected my setup in some shape or form. With that being said, you can probably do a dif between the 106.1 version and the 106.5 version. It is likely, like you said, that XRDevice was renamed to XRInputDevice.

[mattmazzola]

I wanted to give an update on progress.

I was able to find more of the breaking changes made to XR extension and update my extension to use the correct types. I can now at least open and partially run operations in my extension; however, Isaac Sim 4.5 crashes shortly after so I believe there is still work to be done.

Changes to omni.kit.xr.core 106.1.24 → 106.5.13

Given I couldn’t find release notes declaring the changes, I resorted to @dvelazco 's suggestion to diff the extensions versions.

Folder Compare

...\ov\pkg\isaac-sim-4.2.0\extscache\omni.kit.xr.core-106.1.24+106.1.0.wx64.r.cp310\omni\kit\xr\core
...\ov\pkg\isaac-sim-4.5.0\extscache\omni.kit.xr.core-106.5.13+106.5.0.wx64.r.cp310\omni\kit\xr\core

XRDevice → XRInputDevice

XRCore start_singleton becomes private, use get_singleton

Entire xr_gesture_recognizer_manager removed

This means XRGestureEventType can’t be used.

Questions

1. What is the new equivalent to subscribe to gesture events?

I was previously using XRGestureEventType.begin to subscribe to gesture events with
event_stream.create_subscription_to_push_by_type(XRGestureEventType.begin, ...

Is there some combination XCoreEvent and XToken that gives the same stream?

[mbickley]

Hey there @mattmazzola, there are indeed some small changes to XRCore → XRProfile that did not make it to a changelog, apologies for that! Let me get some input from the team and come back with some guidance.

[mattmazzola]

@mbickley

Let me get some input from the team and come back with some guidance.

Can you provide any new information based on your feedback from the team?

Perhaps there are work arounds that have the same effect?

Given how helpful VR is in controlling robots in 3D environments I imagine there are many others that have applications relying on these APIs.

[mattmazzola]

@mbickley I realize that my post above used a quote from your post but didn’t explicitly @ mention you which may not notify. I am posting again with explicit mention.

[hfriedman]

Hi @mattmazzola,

Thanks for your patience. I’ll be handling this while Max is away and I am catching up on the details.

I tested the latest public release of Isaac Sims 4.5 with the VR profile (OpenXR) using the Meta Quest 2 headset, and it worked as expected.

1. Could you let me know if you’re using the batch file isaac-sim.xr.vr.bat or did you manually add "omni.kit.xr.profile.vr" = {} to the .kit file dependencies list?

2. If you began updating your files by renaming, can you try to dowload the latest Isaac Sims 4.5 and verify if you are able to execute VR in that version. If yes, then we can narrow down the problem to the renaming workflow.

3. Could you also try using the recommended driver 537.58 ? I noticed that you are currently using 560.94 . This will help us determine if the driver is causing the issue and allow us to eliminate it as a potential problem.

[mattmazzola]

@hfriedman

I tested … it worked as expected

Can you provide more details about the “test” you performed and what you mean by “it worked”?

Given I analyzed the source code and the API we depend on was removed, I am skeptical that you can say “it worked” without providing evidence of a new or alternate API. This implies what you tested was not direct replacement of functionality.

I think a good test would be:

1. setup a GestureEventType subscription
2. set a break point in the event callback
3. provide screenshot of locals which proves it was called with the expected values we can use for our application.

if you’re using the batch file

No

did you manually add "omni.kit.xr.profile.vr" = {} to the .kit file dependencies list?

It’s not a .kit file, it is the extension.toml, but yes, it is declared.

dowload the latest Isaac Sims 4.5

Are you implying there are changes made to Isaac Sim 4.5 download that are different from the download 3 months ago? If so, I think this implies a version change such as 4.5.1

I see in my 4.5.0 VERSION file there is 4.5.0-rc.36+release.19112.f59b3005.gl
What version are you using?

verify if you are able to execute VR in that version

There is related issue Help analyzing GPU crash when using Isaac Sim 4.5? which would prevent me from executing the portion of our extension which tests VR.

This post was about getting the extension to load with the VR code included since it is missing APIs
In other words, I will not be able to verify execution, but I could verify the APIs exist since I can see the source code and import it.

try using the recommended driver 537.58 ?

Maybe when I get to a point where upgrading to 4.5 is primary task I could, but I don’t want to change my graphics card drivers at this time. The risk of breaking other things I am working on is too high.
I think using the latest drivers is usually recommended since they include fixes for other issues.
I may wait for Isaac Sim to support them instead.

[hfriedman]

Thank you for your patience @mattmazzola, based on internal documentation that will be made public in the next release, I have answers to some your questions regarding the XR changes in Isaac Sim 4.5 and how to resolve some of the issues you’re facing.

Question 1: Was XRDevice intentionally renamed without documentation?
Answer:
Yes, XRDevice was replaced by XRInputDevice. The internal documentation confirms that the new class XRInputDevice is used to manage input devices such as controllers and head-mounted displays. Developers can retrieve a list of input devices using the get_input_devices method from the XRCore class:

`input_devices: list[XRInputDevice] = xr_core.get_input_devices()`

This change aligns with the broader updates in XRCore to standardize input handling and improve functionality. While the updates were intentional, some details may not have been reflected in the public release notes, which could have led to confusion for developers.

Question 2: What replaces XRGestureEventType.begin subscriptions?
Answer:
The functionality previously provided by XRGestureEventType.begin has been replaced by an event generator mechanism. Developers can bind an event generator to an input device to handle specific gestures or button states. For example:

event_generator: XREventGenerator = input_device.bind_event_generator("menu", "xr_menu", ("press", "release"))

This creates events such as "xr_menu.press" and "xr_menu.release" for the “menu” button. Developers can subscribe to these events via the message bus:

import carb 

def on_xr_menu(event: carb.events.IEvent): 
    print("Menu button pressed") 

message_bus:carb.events.IEventStream = xr_core.get_message_bus() 
message_type: int = carb.events.type_from_string("xr_menu.press") 
message_bus.create_subscription_to_pop_by_type(message_type, on_xr_menu)

Additionally, poses (position and orientation data) are no longer included directly in events but can be queried separately from the input device when needed. If you want the pose in the stage, use input_device.get_virtual_world_pose(). If you want the pose in physical world, use input_device.get_pose().

pose: Gf.Matrix4d = input_device.get_pose()
position: Gf.Vec3d = pose.ExtractTranslation()
orientation: Gf.Quatd = pose.ExtractRotation()

This approach provides greater flexibility and customization for handling input events compared to the previous method.

Question 3: Is extension.toml sufficient for XR dependencies in 4.5?
Answer:
Yes, extension.toml is sufficient for declaring XR dependencies in Isaac Sim 4.5. The documentation specifies that profiles like VR (omni.kit.xr.profile.vr) must be included as extensions in the project. To enable a profile programmatically, developers can use:

xr_core.request_enable_profile("vr")

The profile configuration is defined in its corresponding extension.toml file, which includes render settings and other dependencies needed for XR experiences.