Omniverse web dev websocket

[chrisw6]

Hello NVIDIA Omniverse Support Team,

I am currently integrating external AI agents into our multi-agent pipeline extension within Omniverse Kit SDK 107.2. Specifically, we attempted integrating ChatGPT-4.5 via Bing and Google AI services using Selenium WebDriver automation (Edge and Chrome browsers respectively). While we’ve had significant progress outside Omniverse, we’re encountering persistent issues within Omniverse when connecting through WebSockets.

Environment Setup & Context:
Omniverse Kit Version: 107.2 RC (Black Box environment updated)

System Configuration: Windows 11, Python 3.11 (via Anaconda), Selenium WebDriver latest stable release.

Browsers & Drivers:

Microsoft Edge with EdgeDriver (msedgedriver.exe)

Google Chrome with ChromeDriver

Handler Structure:

AI agents initiated externally via Selenium automation scripts.

Communication established via WebSocket on local network (127.0.0.1 or localhost).

Problem Summary:
We successfully automated and established stable WebSocket connections externally (outside Omniverse environment) using both Bing (Edge) and Google (Chrome) agents. After comprehensive external tests confirmed proper WebSocket connectivity, message sending, and receiving, we proceeded to integrate these agents into our Omniverse extension handlers.

However, upon integration into Omniverse Kit SDK 107.2, we encountered persistent WebSocket connectivity failures, preventing the agents from communicating effectively with the Omniverse extension. Despite numerous attempts, adjustments, and tests, the issue remained unresolved within Omniverse.

Steps Taken for Troubleshooting:
WebSocket External Validation:

Successfully tested connections externally via Python Selenium scripts.

Confirmed active and stable WebSocket connections to both Bing and Google AI endpoints.

Validated message payload sending and response receipt.

Firewall & Network Configurations:

Explicitly allowed WebSocket ports through Windows Firewall.

Tested connectivity via PowerShell/WebSocket test tools (outside Omniverse successfully).

Omniverse Environment Adjustments:

Used Omniverse’s built-in Python (pipapi) to install Selenium correctly.

Ensured Selenium WebDriver binary (msedgedriver.exe and ChromeDriver) paths explicitly defined and verified.

Confirmed no conflicting Omniverse extensions or scripts.

Debugging & Error Analysis within Omniverse:

Omniverse logs repeatedly indicated timeouts and connection refusals specifically when WebSocket connections were attempted.

Verified no resource contention or port conflicts within Omniverse’s runtime.

Persistent Error Description (Typical):
plaintext
Copy code
[Error] [omni.kit.websocket] Failed to establish WebSocket connection to ws://localhost:[port]: Connection refused.
This error consistently occurs only within the Omniverse environment. No errors or issues occur during external tests with identical parameters.

Specific Questions & Assistance Needed:
Are there known limitations or special requirements when establishing local WebSocket connections (via Selenium WebDriver automation scripts) within Omniverse Kit SDK 107.2?

Could there be any known conflicts or network restrictions specifically applied by Omniverse’s internal environment or extensions that we should explicitly configure or bypass?

Is there a recommended approach for debugging such WebSocket connectivity issues in Omniverse SDK environments, especially when external validations confirm successful connectivity?

Additional Notes:
Extensive logs and configurations can be provided upon request for deeper debugging.

We have explicitly followed NVIDIA’s recommended best practices for handler and script development within Omniverse Kit SDK 107.2.

Thank you very much for your support and guidance. We are eager to resolve this integration issue, as successful connectivity is critical for our project’s progress and stability.

Warm regards,
Chris Wirth
Creative Lab / Multi-Agent Pipeline Development

[Richard3D]

Hi Chris,
Ok so silly question, but where are you following a tutorial or documentation on Omniverse and Websockets? The reason I ask is that I have not seen anything on using websockets with omniverse, except for specific use with Streaming. But that all has been programmed into kit. I am not aware that we support them.

[chrisw6]

Hello,

Thank you for your response.

To confirm, NVIDIA Omniverse Kit SDK 107.2 officially supports WebSocket-based streaming through the following extensions:

WebSocket Streaming Client (omni.services.streamclient.websocket)
Official documentation: WebSocket Browser Client

General Livestream Client Extensions overview:
Livestream Extensions

Both extensions are supported and documented for use with Kit SDK 107.2.

Please let us know if you require any additional information.

[Richard3D]

Yes I am agree, but that is just for streaming. But you are talking about websockets to connect other things. AI through extensions etc. Can you show me your code? As I understand it, you are not talking about Streaming.

[chrisw6]

WebSocketHandler.py

NVIDIA Omniverse Kit SDK 107.x Example WebSocket Handler (NOT for streaming)

CHANGELOG:

- v1.0.0 (2025-06-03): Initial creation for NVIDIA code review. Supports basic message send/receive for agent/AI communication.

import asyncio
import omni.ext
import carb

Ensure ‘websockets’ is installed via pipapi before using

try:
import websockets
except ImportError:
carb.log_error(“websockets not installed. Run in Script Editor:\nimport omni.kit.pipapi; omni.kit.pipapi.install(‘websockets’)”)
raise

class WebSocketHandler:
def init(self, host=‘localhost’, port=8765):
self.host = host
self.port = port
self.server = None
self.clients = set()
self.loop = asyncio.get_event_loop()
carb.log_info(f"[WebSocketHandler] Initialized on {self.host}:{self.port}")

async def handler(self, websocket, path):
    carb.log_info(f"[WebSocketHandler] Client connected: {websocket.remote_address}")
    self.clients.add(websocket)
    try:
        async for message in websocket:
            carb.log_info(f"[WebSocketHandler] Received: {message}")
            await websocket.send(f"Echo: {message}")
    except Exception as e:
        carb.log_error(f"[WebSocketHandler] Error: {e}")
    finally:
        self.clients.remove(websocket)
        carb.log_info("[WebSocketHandler] Client disconnected.")

def start_server(self):
    carb.log_info(f"[WebSocketHandler] Starting server at ws://{self.host}:{self.port}")
    self.server = websockets.serve(self.handler, self.host, self.port)
    self.loop.run_until_complete(self.server)
    carb.log_info("[WebSocketHandler] Server started.")

def stop_server(self):
    if self.server:
        self.server.ws_server.close()
        carb.log_info("[WebSocketHandler] Server stopped.")

async def send_to_all(self, message):
    carb.log_info(f"[WebSocketHandler] Sending to all: {message}")
    await asyncio.gather(*[client.send(message) for client in self.clients])

Minimal usage example for Omniverse extension:

ws_handler = WebSocketHandler()

ws_handler.start_server()

await ws_handler.send_to_all(“Hello agents!”)

• This code is explicitly for real-time agent/messaging use, NOT video/RTX streaming.
• It is fully decoupled from the Nucleus/RTX Streaming API.
• Usage: AI pipelines, remote agent messaging, tool orchestration—not for media or frame transport.
• Compliant with Omniverse Kit 107.x extension development best practices.

[chrisw6]

I wanted to share that we’ve successfully implemented a robust workflow for integrating web-based AI models within an Omniverse extension—without any changes or modifications to the locked Kit Python environment. This approach enables direct AI interaction, live retrieval, and contextual processing from within Omniverse, while fully maintaining the integrity and security of the Kit runtime.

If this is an area of ongoing interest for your roadmap, happy to discuss approaches or future-proofing considerations. Thank you for building such a powerful and flexible platform!

[Richard3D]

Great I am glad to hear it. Please feel free to post any relevant help, tips or information for others to use.