Description
I have deployed the one‑click Tokkio pipeline on my PC (baremetal) and successfully customized the scene using manual resource uploads. However, the current solution streams a passive video (via RTP over UDP) to the web browser. I would like to modify this setup to enable interactive navigation (e.g., “walking” around within the scene) directly from the browser. I have attempted to access the documented ports (e.g., those for the Omniverse Renderer microservice), but it appears that the renderer only outputs a fixed video stream without exposing underlying scene data or control endpoints.
I’m looking for guidance or potential solutions to:
- Either expose the raw scene or USD data to a browser‑based renderer (using, e.g., Three.js) that would allow interactive control,
or - Extend the existing microservices (such as by controlling the camera via the Animation Graph HTTP API) to update the view interactively.
Any pointers to documentation, reference projects, or suggested architectures would be greatly appreciated.
Environment
TensorRT Version: N/A
GPU Type: NVIDIA RTX 3080
Nvidia Driver Version: 535.xx
CUDA Version: 11.8
CUDNN Version: 8.6
Operating System + Version: Ubuntu 22.04
Python Version (if applicable): 3.9
TensorFlow Version (if applicable): N/A
PyTorch Version (if applicable): N/A
Baremetal or Container (if container which image + tag): Baremetal deployment (using the one‑click Tokkio pipeline release vX.Y)
Relevant Files
- Tokkio documentation: Tokkio - NVIDIA Docs
- Animation Graph Microservice API: Animation Graph API
- Omniverse Renderer Microservice API: Omniverse Renderer API
Steps To Reproduce
- Deploy the Pipeline: Follow the one‑click deployment guide for Tokkio on a baremetal PC.
- Customize the Scene: Use the manual resource upload method to change the scene (this part works as expected).
- Access the Stream: Open the Tokkio UI in Google Chrome, which currently displays a video stream rendered via RTP over UDP.
- Attempt to Control: Try accessing the documented ports (e.g., via REST or WebSocket endpoints) to interact with the scene (e.g., adjust camera parameters or control movement), but note that no interactivity is available—the stream remains a passive video.
- Observation: There is no error traceback; rather, the system does not expose any interactive scene data or control endpoints that would allow browser‑based “walking” or other interactive maneuvers.
Any insights on how to modify the pipeline to allow interactive control from the web browser would be extremely helpful.