XR Core events, action maps and eventgenerators for modifying controls

[AwieArmin]

Hello guys,
I am new to Omniverse and try setting up a extension with xr.core extension. Right now i am trying to recieve input events from my htc vive controllers but i am not recieving anything from the IEventStream Message bus nor can i bind anything with a eventgenerator like described in the readme of the extension. I already checked the selected action map but i dont have a clue anymore, why i am not recieving a trigger for the print. My ultimate goal is to listen to the grabbing of the user and then decide if the object is interactable or not.

Thanks for your tips and inputs in advance, i really appreciate it!

Here is my code for the message bus:

def somefunc(self) → None:
self.message_type = carb.events.type_from_string(“xr_right_grab.click”)

                        print(f"Message Type: {self.message_type}")

                        self.\_event_bus.create_subscription_to_pop_by_type(self.message_type, self.on_xr_right_grab)

def on_xr_right_grab(self, event: carb.events.IEvent):

    print("Right grab event detected")

[Richard3D]

Let me see if I can find someone to help you with this. This is very advanced. Why do you need to modify the vive controllers?

[AwieArmin]

Hey Richard, my first goal was to modify the grab mechanic so i can for example allow the user to move objects along the XYZ axis (and therefore being able to lock certain axis for e.g. only X-direction movement) and my second goal was to setup object-grab-moving with e.g. surface snapping instead of free grab movement.

[Richard3D]

Ok thanks. I have put a message in with the XR engineers. I will circle back to you.

[Richard3D]

Ok this is direct from the engineers:

Make sure the user is using XR message bus. Try
from omni.kit.xr.core import XRCore XRCore.get_singleton().get_message_bus().create_subscription_to_pop_by_type(…)

Also, I didn’t see xr_right_grab.click in XR grab tool. Should it be xr_right_grab.press ?

there’s also release and suspend:
self.register_message_bus_event_handler(“xr_right_grab.press”, self.grab_press),
self.register_message_bus_event_handler(“xr_right_grab.release”, self.grab_release),
self.register_message_bus_event_handler(“xr_right_grab.suspend”, self.grab_suspend),

[Richard3D]

Here are some general instructions.

XRCore Input Events

The input devices can generate events when buttons are pressed or released or can send out events every frame with the state of the buttons. To start generating events from an input device you need to bind an event generator to the input device. An input event generator is a class that can be used to generate events when certain events happen in the XR system. They are configured to send a specific event when a certain gesture is done, like a button being pressed or a thumbstick being moved.

To bind an event generator to an input device you can use the bind_event_generator method:

event_generator: XREventGenerator = input_device.bind_event_generator(input_name, event_name, event_list, tooltips)

For example to bind an event generator to the input device that sends out an event when the “menu” button is pressed and released you can use the following code:

event_generator: XREventGenerator = input_device.bind_event_generator("menu", "xr_menu", ("press", "release"))

The event generator will now send out an event with the name “xr_menu.press” when the “menu” button is pressed and an event with the name “xr_menu.release” when the “menu” button is released. The extra step of adding an event generator to an input device is to allow to customize the event name and to only send the events that are needed. The event generator sends out the events to the message bus and you can subscribe to them with the following code:

import carb

def on_xr_menu(event: carb.events.IEvent):
    print(f"XR menu button pressed")

message_bus: carb.events.IEventStream = xr_core.get_message_bus()
message_type: int = carb.events.type_from_string("xr_menu.press")
message_bus.create_subscription_to_pop_by_type(message_type, on_xr_menu)

The event generator monitors a given input button and can generate the following events:

• press: button is pressed
• release: button is released
• update: button is pressed and the state of the button is packaged in the event. This event is sent each frame
• touch: button is touched
• lift: button is no longer touched
• suspend: some other event generator is grabbing focus from the button
• resume: the other event generator relinquished control
• state: send each frame with values of all the gesture state for the button

The name of the event that is sent out is a combination of the input name and the gesture name. In the example above the event name is “xr_menu” and the input name is “menu” and the gesture name is “press”. This will result in an event with the name “xr_menu.press” being sent out.

The state of the button is also packaged in the event and is a dictionary with the following keys:

• touch: the button is touched
• click: the button is clicked
• value: for buttons like trigger or squeeze indicating how much the button is pressed
• x: x value of the thumbstick/trackpad
• y: y value of the thumbstick/trackpad

To make it easier to extract the state from the event, we added a class “XRInputDeviceGeneratorEvent” that can be used to wrap the event and extract the state from it.

def on_xr_left_trackpad(event: carb.events.IEvent):
    input_event: XRInputDeviceGeneratorEvent = XRInputDeviceGeneratorEvent(event)

    # you can get the state of the button from the event
    trackpad_x: float = input_event.x
    trackpad_y: float = input_event.y
    click: bool = input_event.click
    touch: bool = input_event.touch
    value: float = input_event.value
    dt: float = input_event.dt # time passed since last event

    # You can also access who sent the event
    input: str = input_event.input # the input name
    input_device: str = input_event.input_device # the input device name
    input_device_type: str = input_event.input_device_type # the input device type

input_device: XRInputDevice = xr_core.get_input_device("/user/hand/left")
event_generator: XREventGenerator = input_device.bind_event_generator("trackpad", "xr_left_trackpad", ("state"))
message_bus: carb.events.IEventStream = xr_core.get_message_bus()
message_type: int = carb.events.type_from_string("xr_left_trackpad.state")
message_bus.create_subscription_to_pop_by_type(message_type, on_xr_left_trackpad)

The other advantage of using input event generators is that they can be used to temporarily take over the input of an input device. To do this you can define an input generator for an input that is already bound. When you release the input generator, the input device will go back to the previous input generator. This can be used to implement things like move an object when the user is pointing at an object and bind movement of the object to the trackpad. When the user has moved the object, the input generator is released and the input device goes back to allowing the user to fly through the scene.

input_device: XRInputDevice = xr_core.get_input_device("/user/hand/left")
event_generator: XREventGenerator = input_device.bind_event_generator("trackpad", "fly_through_scene", ("state"))

# bound to fly through scene

second_event_generator: XREventGenerator = input_device.bind_event_generator("trackpad", "move_object", ("state"))

# Do logic here

second_event_generator = None # release input generator

# Back to fly through scene

By default an event generator is setup to stop generating events when the event generator that is returned is released. If the function “set_auto_unbind” is set to False, the event generator will not stop generating events and you need to manually stop it by calling “unbind_event_generator” on the input device.

event_generator: XREventGenerator = input_device.bind_event_generator("trackpad", "fly_through_scene", ("state"))
event_generator.set_auto_unbind(False)

# Do logic here

input_device.unbind_event_generator("fly_through_scene")

[AwieArmin]

Hey Richard, so I am I right in the assumption that i use the Eventgenerator to unbind the original grab tool from e.g. my squeeze button and then create a own grab tool with XRToolComponentBase and the XRUsdLayer with its add_link method?

That’s how i understand the readme doc so far.

[AwieArmin]

Update: I was able to get my own Grab mechanic up and runnig, but i encountered an issue where the commit_link_transform does not apply the transform to the Cube object that i am testing with. Here is my code so far:

def _init_input_system(self):

        stage = omni.usd.get_context().get_stage()

        action_map = self._xr_core.get_action_map()

        name = action_map.get_name() if action_map else "Unknown"

        tools = action_map.get_tool_list() if action_map else []

        print(f"Input system initialized with: AM Name:{name}, Tools: {[tool for tool in tools]}")

        self.usd = self._xr_core.create_xr_usd_layer("/_xr/gui/controllers")

        self.base_path_right: str = self.usd.ensure_device_prim_path("/user/hand/right")

        self.beam_prim_path_right: str = self.return_beam_path(self.base_path_right)

        self.beam_mat = "/_xr/gui/controllers/_coord/assets/_generic_materials__selection_emissive_material_usd/_coord/_ref/Looks/selection_emissive_material"

        self.link_prim_path_right: str = self.return_link_path(self.base_path_right)

        if stage.GetPrimAtPath(self.beam_prim_path_right):

            self.usd.remove(self.beam_prim_path_right)

        self.usd.add_beam(self.beam_prim_path_right, group= "my_grab_interaction", material_reference=self.beam_mat, max_length=20, tube_radius=10, visible=False)

        self.message_bus = self._xr_core.get_message_bus()

        #msg_type_grab_press = carb.events.type_from_string("xr_right_grab.press")

        #msg_type_grab_release = carb.events.type_from_string("xr_right_grab.release")

        #self.sub_grab_press=self.message_bus.create_subscription_to_pop_by_type(msg_type_grab_press, self.on_xr_right_grab_press)

        #self.sub_grab_release=self.message_bus.create_subscription_to_pop_by_type(msg_type_grab_release, self.on_xr_right_grab_release)

        print("Trying to bind right controls, left is untouched")

        self.right_vive = self._xr_core.get_input_device("/user/hand/right")

        self.mybinding = self.right_vive.bind_event_generator("squeeze","my_grab", ("press", "release"))

        #if self.mybinding is not None:

        #    input_device_name: str = str(self.mybinding.get_input_device_name())

         #   input_name: str = str(self.mybinding.get_input_name())

           # print(f"Binding right controls: Device Name: {input_device_name}, Input Name: {input_name}")

        msg_type_grab_press = carb.events.type_from_string("my_grab.press")

        msg_type_grab_release = carb.events.type_from_string("my_grab.release")

        self.sub_my_grab_press=self.message_bus.create_subscription_to_pop_by_type(msg_type_grab_press, self.on_my_grab_press)

        self.sub_my_grab_release=self.message_bus.create_subscription_to_pop_by_type(msg_type_grab_release, self.on_my_grab_release)

        self.grab_state= {"grabbed_usd_path": None, "has_link": False}

    def return_beam_path(self, base_path : str):

        return f"{base_path}/my_beam"

    def return_link_path(self, base_path : str):

        return f"{base_path}/my_link"

    def on_my_grab_press(self, msg):

            print("my right grab pressed")

            self.usd.show(self.beam_prim_path_right)

            target_info = self.usd.get_target_info(self.beam_prim_path_right)

            self.grab_state["grabbed_usd_path"] = target_info.get_target_enclosing_model_usd_path()

            print(f"Target Path: {self.grab_state['grabbed_usd_path']}")

            if self.grab_state["grabbed_usd_path"] is not None:

                self.grab_state["link"] = self.usd.add_link(self.link_prim_path_right, group="my_grab_link", link_path=self.grab_state["grabbed_usd_path"], transform_type=XRTransformType.stage)

                self.grab_state["has_link"] = True

    def on_my_grab_release(self, msg):

            print("my right grab released")

            self.usd.hide(self.beam_prim_path_right)

            if self.grab_state["has_link"]:

                #link_world = self.usd.get_transform(self.link_prim_path_right,

                #                                    transform_type=XRTransformType.world,

                 #                                   use_usd=False

                  #                                  )

                #layer = self._xr_core.suggest_edit_layer_for_prim(self.grab_state["grabbed_usd_path"])

                #self._xr_core.set_world_transform_matrix(self.grab_state["grabbed_usd_path"], link_world, layer_identifier=layer)

                self.usd.commit_link_transform(self.grab_state["link"])

                self.usd.remove(self.link_prim_path_right)

                self.grab_state["has_link"] = False

                self.grab_state["grabbed_usd_path"] = None

[Richard3D]

Ok understood. Let me ask the engineers.

[adrian.chmieletzki.de]

Hi, I’m facing the same issue. I cannot get the API to work as described in the documentation for omni.kit.xr.core.

I am currently developing an OV VR app, with the Meta Quest 3 as a target device. I would like to retain the current tools and actions, and simply rebind the buttons in a context-sensitive manner (e.g. if a prim is selected and I press the “menu” button, the default VR menu should not open, instead call a custom function and do XYZ)

I have tried Kit 106.5, 107.3 and 108.1. But the same issues persist.

According to the documentation, something like this should work:

import omni.kit.xr.core
import carb

xr_core = omni.kit.xr.core.XRCore.get_singleton()
message_bus = xr_core.get_message_bus()

def handle_input(msg):
	print("never gets called...")

device = xr_core.get_input_device("/user/hand/right")
generator = device.bind_event_generator("menu", "my_custom_event", ('press', 'release'))
message_type = carb.events.type_from_string("my_custom_event.release")
sub = message_bus.create_subscription_to_pop_by_type(message_type, handle_input)

This isn’t in an extension, simply in the script editor of the app. (the extension code executes something similar, after the VR profile has been enabled. These general events do get sent out and I can handle them just fine. But the input bindings simply don’t want to work)

Even if I replace “menu” for an explicit button like “b”, or a mapped action like “xr_menu”, the function never gets called. Replacing ‘press’ and ‘release’ for ‘click’ or ‘touch’ makes no difference either.

The same applies to the, lets call it, “non-input-device” way of creating generators

generator = xr_core.bind_input_event_generator("xr_menu", ('press', 'release'))

Can you please provide an example script how to hook this up properly? Because the documentation is lacking in that regard.
My goal: As described in the documentation, I want to be able to put custom event handlers on already bound buttons (and subsequently remove them, restoring the default functionality), so I can create a context-sensitive workflow.

[Richard3D]

Your Omniverse XR script is not working because the event subscription never fires, and this typically comes down to how the event generator and message bus are coordinated, and especially how custom event types are created and dispatched within Kit’s event system.

Core Issues and Solution

1. Your Callback Function

Python requires consistent indentation. Your handle_input function should be indented properly:

python

def handle_input(msg):
    print("never gets called...")

2. Event Type Naming and Registration

• The event type string passed to carb.events.type_from_string must match the event type actually published by the input device/event generator. If you are creating a completely new event type ("my_custom_event.release"), this type must be both:
  • Registered (made known to the message bus)
  • Actually published by some part of the code, or by a system extension handling XR input.

3. Scope of Subscription Object

• The subscription returned by create_subscription_to_pop_by_type must be held in a variable that stays alive; if it goes out of scope, the callback will not receive messages.​

4. Event Generator/Device Binding

Most Omniverse XR extension samples use predefined event types like "xr_right_menu.click" or similar, and input generators are set up with those.

• Custom events require both a sender to publish and a listener to subscribe; your code only subscribes and binds, but likely nothing is pushing an event of "my_custom_event.release".

5. Recommended Pattern

Referencing working examples:​

python

import omni.kit.xr.core
import carb

xr_core = omni.kit.xr.core.XRCore.get_singleton()
message_bus = xr_core.get_message_bus()

def handle_input(event):  # event argument is a carb.events.IEvent
    print("Event received!", event)
    print("Payload:", event.payload)

# Use a standard event type or ensure the device will emit your custom one
event_type = carb.events.type_from_string("xr_right_menu.click")
# Alternatively: event_type = carb.events.type_from_string("xr_right_menu.release")

sub = message_bus.create_subscription_to_pop_by_type(event_type, handle_input)

# Ensure 'sub' is not collected by holding a reference in a class or at module level.

• Here, "xr_right_menu.click" should match an event actually emitted by your XR device/controller.
• For custom events, ensure both the sender (e.g., the device’s event generator) and receiver (your handler) use exactly the same string and registration method.

6. Debugging Tips

• Test your handler with a known event type and confirm it gets called.
• Add print statements immediately after the subscription to confirm registration.

Summary Table

Common Pitfall	Solution
Handler never called (custom event)	Use a standard event first, or ensure a sender
Subscription variable lost	Hold sub in a global/class variable
No events published of given type	Check device profile, bindings, documentation

---

In summary:

• Start with a standard event type like "xr_right_menu.click" to confirm your handler and subscription work.
• Retain the subscription object.
• For fully custom events, ensure something is actively publishing them to the bus, and that both sender and listener agree on the event type.

If you need to bind a custom button or gesture to a new event, ensure you have registered the generator properly and that your code is the one pushing/publishing the event. Let me know if you want a minimal, working sample for registering and publishing a custom event in XR!