NVIDIA Isaac Sight Virtual Gamepad Window not functional properly

Discovered a major issue with the Virtual Gamepad/Keypad (VGP/VKP). If there is a connectivity issue and the front-end loses connection with the backend while the VGP/VKP is in use, the last pressed key value is retained and keeps executing this last command in a loop even when disconnected from backend. I connected the request from the virtual gamepad bridge to our joystick codelet. And printed out the Json messages being received from the VGP/VKP in Isaac Sight.

fig 1: Channel initialization to receive data from Nvidia Sight VGP/VKG

fig 2: Receiving data from VGP/VKG into codelet

fig 3: source and target channels of the data being received from sight

fig 4: Messages before connecting to VGP/VKP

fig 5 : Messages after connecting to Sight. On disconnecting from sight, the messages remain the same.

According to the Sight JavaScript code (fig 6), the front-end needs to send an “alive” command if the connection between the front end and backend is still active. However, I noticed that before the connection is established no messages are sent from VGP/VKG to backend. But, once the connection is established and I disconnect from the backend, the “alive” command is still sent from the front-end to the backend (fig 5). The code section shown below verifies this. This is a reference to the JavaScript code in sdk → packages → sight → webroot → js → virtual_gamepad_window.js

fig 6: JavaScript code for VGP/VKP communication of front end with backend

In the JavaScript code in sdk → packages → sight → webroot → js → virtual_keypad_joystick.js (fig 4), I found that on disconnecting from the backend the VGP/VKP needs to be initialized to no activity else the last pressed key value is retained.

fig 7: No activity initialization of VKP on connection being established with backend for the first time.

fig 8: VKP Failsafe is present only if the connection is established. If the connection is lost the return causes the last keypress to be retained.

This poses a huge safety hazard for our robots. While testing our robot, we were moving it forward by pressing ‘w’ on the VKP during which we lost connection with the backend. The robot kept moving forward instead of stopping when the connection was lost. We cannot solve this issue without the assistance of NVIDIA especially since we don’t have edit permissions to these scripts unless we fork them and even then, we cannot get the Sight front-end built. Kindly assist us in solving this issue.

For your information, we have created our entire navigation stack using the NVIDIA Isaac SDK. This problem in the Virtual Gamepad/Keypad (VGP/VKP) feature is blocked for us on our project. Any assistance that we could get will be greatly appreciated. To virtually control our robot using VKP, we followed this page in the NVIDIA Isaac(2020.2) documentation.
Please advise. Thanks!

@cfr @hemals @Swapnesh @sdesai @sdebnath ^^^

Thank you for such a detailed analysis! If I followed correctly, there was an unexpected network interruption that disconnected the websocket yet the backend kept repeating the last key press. There seems to be a couple of faults here behind the failure. Let me make sure I understand what’s happening here.

  1. The VirtualGamepageWindow does not seem to get notified of websocket disconnects like other windows, such as ReplayWindow, in packages/sign/webroot/index.html, and so will never reset connectedToBackend_, continuing to send “alive” messages, but these “alive” messages should not be getting received by the backend because the websocket is disconnected.

  2. The backend VirtualGamepadBridge is not aware that the websocket connection with the widget has been disconnected, so continues to process repeating replies until the sight widget connection timeout is reached at which point it should stop.

With the above faults, I can see how the last key press can keep going since the timeout is the only mechanism to stop the key repeat. It should be stopping after the connection timeout is reached though which is set by the sight_widget_connection_timeout parameter on VirtualGamepadBridge to 2.5s by default.

Is the failure you’re seeing that the repeating key continues after the 2.5s? This could be resolved otherwise by lowering the sight_widget_connection_timeout for now to something more responsive.

@hemals , Thanks for the quick response!
A couple of things to point out:

  1. I don’t think we have access to the VirtualGamepadBridge backend package/scripts (is included within libnavigation_module.so → refer fig a). As a result, we are not aware of what it contains. We can only use it in the app-graph to receive data from the virtual gamepad (Fig 3).

    fig a

  2. The deadman switch does not work as intended. The widget is supposed to stop sending messages to the backend when a key press event has not occurred as seen in Fig 7. But in our case the messages are still being sent even though none of the keypad keys have been pressed. We also noticed that when the connection reoccurs and we click connect to backend button, the widget resets the axes to [0.0,0.0,0.0,0.0] (initialize to inactive state - Fig 7)

  3. Is the sight_widget_connection_timeout set to timeout if there are no messages received within that time (2.5s)?

Is this something that you are recommending that you would change along with fixing the virtual keypad bridge’s deadman switch and update the Isaac SDK 2020.2?

Optional request: Also, I observed the deadman-button functionality and failsafe button functionalities are very different from joystick, mousepad, and keypad. Along with this change is it possible to add a physical failsafe button even for keypad rather than just counting on the keypress and keyup?

VirtualGamepadBridge is the backend class receiving the VKC commands from the frontend. You are correct it is not available for modification. Yes, the timeout happens when there are no messages received from the frontend in 2.5s including heartbeart (no connection is inferred). Reducing that timeout duration to get a more responsive disconnect behavior is a workaround. We’ll of course prioritize revisiting the virtual gamepad bridge for the next version of Isaac SDK.

@hemals Thanks for providing the workaround and as far the next version of Isaac SDK, what’s the timeline for that since we need to use VGP for our products?