Running Isaac Sim in a browser on TCP-only clouds (RunPod): WebRTC livestream can't work — here's a noVNC approach that does

TL;DR

Isaac Sim’s WebRTC/native livestreaming cannot work on cloud hosts that only forward TCP (like RunPod and similar proxy-based GPU providers). I spent weeks troubleshooting this and built a solid workaround: rendering Isaac Sim into a virtual X display and serving that desktop over HTTP with noVNC.

The Problem

On RunPod, opening the WebRTC client gives a grey screen forever. The signaling connects, but no frames arrive. Root cause: Isaac Sim’s livestream media is SRTP over UDP. RunPod forwards no inbound UDP at all. The signaling rides TCP, but the video simply cannot arrive.

The Solution: noVNC

Instead of trying to force Isaac to stream WebRTC over TCP, I let it draw its normal GUI into a virtual screen, and shipped that screen over HTTP: Isaac Sim GUIXvfbx11vncwebsockify + noVNCBrowser

  • Pure TCP (HTTP + WebSocket): Exactly what these cloud proxies carry.

  • No NAT traversal or ICE needed.

  • Isaac Sim is not modified: It works across versions.

  • One HTTP port: Plus, you get real mouse/keyboard to load your own USD scenes easily.

(Note: VNC ships compressed framebuffer updates, so it’s less smooth than hardware-encoded WebRTC, but it actually works!)

🛠️ Deep Dive & Bug Reports (For NVIDIA Devs)

I tried everything to make the native streaming work before building the noVNC workaround:

  • UDP→TCP tunnel (socat): Impossible, WebRTC stack only speaks ICE/DTLS/SRTP.

  • Tailscale / WireGuard: Fails due to no /dev/net/tun in these containers.

  • TURN relay over TCP (coturn): This almost worked, but exposed several hardcoded issues in the current streaming setup.

Here is the actionable evidence for the dev team:

  1. Private ICE Candidates Only: Isaac Sim’s WebRTC server only advertises private ICE candidates (127.0.0.1 and 172.18.0.2). A remote browser can never reach those, so Chrome burns its timeout on unreachable pairs instead of using the relay. coturn logs showed the relay was allocated but never used (zero bytes reached Isaac).

  2. Fatal Port 80 Check: Forcing iceTransportPolicy: "relay" tripped Isaac’s watchdogs. One watchdog does a connectivity check against http://<server> on port 80 (never exposed on these hosts) and converts a recoverable state into a fatal stream stop.

  3. 0x0 Resolution Bug: Seen in the server logs, explaining black frames even on a connected session: [carb.livestream-rtc.plugin] Stream Server: streaming at 0 x 0.

Questions for the NVIDIA Team:

  1. Is there a supported way to make the WebRTC server gather relay/host candidates for a reachable address, so it can be used behind a TCP-only proxy?

  2. The client’s connectivity check hits port 80 and turns a failure into a fatal stream stop. Could this be made configurable?

  3. Isaac Sim 5.x/6.x appear to have removed the ICE/TURN configuration entirely. Is a TCP-only streaming path planned for the future?

I am happy to share full logs or test patches if it helps!

Hi, thanks for the exceptionally thorough writeup — the diagnosis and the noVNC fallback are both genuinely useful.

On Private-only ICE candidates, the WebRTC server does have a knob for the advertised address, it’s just not obvious. Set the public address explicitly:
–/exts/omni.kit.livestream.app/primaryStream/publicIp=<PUBLIC_IP>

In our container entrypoint (runheadless.sh) this is wired to the ISAACSIM_HOST env var, alongside ISAACSIM_SIGNAL_PORT (default 49100) and ISAACSIM_STREAM_PORT (default 47998). If publicIp is unset you get exactly the 127.0.0.1 / 172.x candidates you saw.
Setting it should make the advertised candidate reachable.
The real blocker is transport, not addressing. You’re right that even with a correct candidate, the media is SRTP over UDP, and a TCP-only proxy can’t carry it.

For the fatal port 80 check issue, can you share the relevant logs? Would like to do further check on this.

The port-80 connectivity check — Detailed Analysis

Environment:

  • Image: nvcr.io/nvidia/isaac-sim:4.0.0 (Isaac Sim 4.0.0-rc.21)

  • Host: RunPod GPU pod (container, Docker bridge net 172.18.0.2)

  • Client: /streaming/webrtc-demo/ served by the pod, Chrome 150

  • Pod public IP: 213.173.109.229

  • Reachable ports: Only these, via RunPod’s port mapping:

    • 213.173.109.229:19321 -> :3478 (coturn, my TURN relay)

    • 213.173.109.229:19322 -> :8211 (Isaac web client + REST)

    • 213.173.109.229:19323 -> :49100 (WebRTC signaling)

Nothing is listening on 213.173.109.229:80, and nothing can be. RunPod exposes services only on mapped ports or through its HTTPS proxy. This is normal for essentially every proxy-based cloud GPU provider.

What the client does

kit-player.js (the Ragnarok client) runs checkConnectivity() whenever a session error is reported. Deminified:

JavaScript

checkConnectivity(evt, session, cb) {
  if (utils.shouldRunConnectivityTest(session.error.code)) {

    let timeout = 1500; // connectivityCheckTimeout
    if (RagnarokSettings.ragnarokConfig.connectivityCheckTimeout) {
      timeout = RagnarokSettings.ragnarokConfig.connectivityCheckTimeout;
    }

    // ---- the probe URL: scheme + serverAddress, NO PORT ----
    const url = ("http:" === window.location.protocol ? "http://" : "https://")
              + this.initializeParams.serverAddress;

    Log.i(TAG, "connectivity url: " + url);

    utils.customFetch(url, timeout, { method: "OPTIONS" })
      .then(res => {
        if ((res.status >= 200 && res.status < 300) || res.status === 403) {
          evt.connectivity = "online(...)";
        } else {
          evt.connectivity = "offline_wrong_status(" + res.status + ")";
        }
        this.notifyClientWithError(evt, session, cb);
      })
      .catch(err => {
        if (err.name === "AbortError") {
          evt.connectivity = "timeout";
        } else {
          evt.connectivity = "offline(" + err.name + ":" + err.message + ")";
          // ---- the error code is REWRITTEN here ----
          session.error.code = utils.convertErrorOnConnectivityTest(session.error.code);
          evt.result = utils.GetHexString(session.error.code);
        }
        this.notifyClientWithError(evt, session, cb);
      });

  } else {
    this.notifyClientWithError(evt, session, cb);
  }
}


serverAddress is taken from the page’s ?server= parameter. In my case, that is 213.173.109.229 — a bare host with no port, so the probe resolves to [``http://213.173.109.229:80/``](``http://213.173.109.229:80/``). That address is correct as a host. The client is talking to it right now — it loaded the page from :19322 and is running signaling against :19323. But port 80 on that host is not, and never will be, a service.

Why this can never succeed here

The probe is hardcoded to the default port for the scheme. In any environment where the server is reached on a non-default port — port mapping, Docker publish, Kubernetes NodePort, an SSH tunnel, or a reverse proxy on a custom port — the probe targets an address that is not the service. The result is deterministic:

OPTIONS [http://213.173.109.229/](http://213.173.109.229/) net::ERR_CONNECTION_REFUSED

Two things are being conflated: “is the host reachable” and “is port 80 on the host serving HTTP with permissive CORS”. Only the first is what the check wants to know; only the second is what it actually tests.

The Network panel also shows the CORS preflight failing alongside the fetch. So even if something were listening on :80, a cross-origin OPTIONS from [http://213.173.109.229:19322](http://213.173.109.229:19322) would still need explicit CORS headers to be treated as success. The probe has two independent ways to fail before it ever measures connectivity.

Observed behaviour — reproduced twice, independently

Occurrence 1 (17:00:41):

Plaintext

[sleepdetector] sleep 5579 0
[gridapp]       connectivity url: http://213.173.109.229
[streamclient]  Stopping Stream with error 0xC0F22206
OPTIONS http://213.173.109.229/  net::ERR_CONNECTION_REFUSED      <- kit-player.js:7
[streamclient]  Couldn't send control channel message
[streamclient]  Audio track muted
[RWorker]       WebSocket connection closed


Occurrence 2 (17:19:12), same sequence:

Plaintext

[sleepdetector] sleep 5677 0
[gridapp]       connectivity url: http://213.173.109.229
[streamclient]  Stopping Stream with error 0xC0F22206
OPTIONS http://213.173.109.229/  net::ERR_CONNECTION_REFUSED


Stack, straight from the console:

Plaintext

t.customFetch       @ kit-player.js:7
checkConnectivity   @ kit-player.js:91
onSessionStart      @ kit-player.js:91
notifyStart         @ kit-player.js:109
stopStreamWithError @ kit-player.js:109
streamBeginTimeout  @ kit-player.js:109


I also saw the same probe on errors 0xC0F22210, 0xC0F22219, and 0x00F22003. Every failure I encountered, whatever its origin, ran this probe and it failed every time.

What the impact actually is — being precise

I want to correct my own original wording. In my forum post, I said the check “converts a recoverable state into a fatal stop.” Reading the code carefully, that overstates it, and I’d rather be exact:

The probe does not itself stop the stream. By the time checkConnectivity runs, stopStreamWithError has already been called — in my case from streamBeginTimeout, because no media ever arrived (the real, separate problem: SRTP over UDP on a TCP-only host). Teardown is already underway.

What it does do is three things, and the third is the harmful one:

  1. It is guaranteed to fail in any port-mapped deployment. Not flaky — deterministic.

  2. It mislabels every failure as a network outage. Every error I hit was ultimately reported as offline(...), regardless of true cause. As a user, this cost me a great deal of debugging time chasing a network problem that did not exist. The client was telling me the network was down while it was concurrently holding a healthy WebSocket to that very host.

  3. It rewrites the error code. In the .catch path:

    JavaScript

    session.error.code = utils.convertErrorOnConnectivityTest(session.error.code);
    
    
    

    And that function performs real mappings:

    JavaScript

    switch (e) {
      case 3237093906: if (g()) t = 15867908; break;   // 0xC0F22212 -> 0x00F22004
      case 3237093654: t = 15867909;             break;   // 0xC0F22116 -> 0x00F22005
    }
    
    
    

    The mutated code then feeds the client’s own resume logic:

    JavaScript

    if (t.error) {
      const e = utils.canResume(t.error.code);
      t.isResumable = e;
      Log.i(TAG, "canResume " + e);
    }
    
    
    

    So a probe that cannot succeed in this environment is silently reclassifying error codes on a path that decides whether the session is resumable. I have not fully mapped canResume()'s table, so I won’t assert that this always flips a resumable session to non-resumable — but the data flow is there, and it is driven by a probe whose failure carries no information about the network at all.

Suggested fix

The client already knows at least two endpoints that are known good — it was served from one and is speaking signaling to the other. Any of these removes the failure mode:

  • Probe the signaling endpoint it is already connected to — serverAddress plus the signal port. Most faithful to intent: “can I still reach the server I’m streaming from.”

  • Probe window.location.origin — the page it was loaded from, by definition reachable.

  • Make it configurable. RagnarokSettings.ragnarokConfig already carries connectivityCheckTimeout; a connectivityCheckUrl (or a disable flag) alongside it would be a natural, minimal change.

Even option 3 alone would have saved me hours, and would unblock every user on a port-mapped host.

Related, possibly the same root area

On sessions that did reach a connected state, the server logged:

Plaintext

[carb.livestream-rtc.plugin] Stream Server: connected stream 0x... on connection 0x...
[carb.livestream-rtc.plugin] Stream Server: configuration update packet from client
[carb.livestream-rtc.plugin] Stream Server: streaming at 0 x 0.
[carb.livestream-rtc.plugin] Client (nil) disconnected.


streaming at 0 x 0 — a zero-area render viewport — while the client had requested and logged Overriding stream resolution to : 1280x720. If the encoder is producing no pixels, that would independently explain black frames on otherwise-connected sessions. I never chased this one down; flagging it in case it’s known or points somewhere useful.

For completeness — how far the TURN path got

Relevant because it shows ICE itself is not the blocker. With a coturn relay in the pod and iceTransportPolicy: "relay" forced on the client, Chrome reached:

Plaintext

ICE connection state: new => checking => connected
transport (iceState=connected, dtlsState=connected)
candidate-pair (state=succeeded)
selected pair: relay(tcp) turn:213.173.109.229:19321?transport=tcp  <=>  172.18.0.2:1024


So ICE completes fine over a TCP relay. The wall is exactly where you said it is: the media itself is SRTP over UDP, and no amount of correct addressing creates a UDP path through a TCP-only proxy.

Hi, thanks for the follow-up! I forwarded your findings to our streaming client team for review.

Internal ticket 6467268