Hi, I am facing an issue with AVP and CloudXR 6.2 .After carefully testing, I am pretty sure there is a bug; the thing is that official documentation claim that everything should be compatible in my setup. Here is a detailed description of what is happening:
Bug report for NVIDIA (Isaac Teleop / CloudXR). Everything below was observed on a single workstation. Pico (Quest-style, auto-webrtc, AV1) streams fine; Apple Vision Pro (HEVC) connects then immediately drops because the server video-encode pipeline never produces a frame.
This is not specific to one device profile. It reproduces identically on both the generic auto-native profile and the AVP-specific AppleVisionPro profile. The common factor is CloudXR 6.2.x’s HEVC server-encode path on Linux.
Summary
Streaming to Apple Vision Pro establishes signaling and negotiates all video/audio streams, then the server-side HEVC encoder fails to produce any frames and the session is torn down (the AVP app closes right after “Connect”). This happens under every device profile we tried (auto-native and the AVP-specific AppleVisionPro), so it is not a profile-selection problem — it is the 6.2.x HEVC encoder-init path itself.
The same workstation streams perfectly to a Pico 4 Ultra (auto-webrtc, AV1), and — critically — ffmpeg’s hevc_nvenc encodes fine at the AVP resolution (2400×1920), including 4 concurrent sessions. So HEVC encode works at the GPU/driver level; the failure is inside the CloudXR 6.2.x runtime’s HEVC encode path on Linux (independent of the device profile — see below).
Environment
| Component | Version |
|---|---|
| GPU | NVIDIA GeForce RTX 5090 (Blackwell, GB202) |
| NVIDIA driver | 580.159.03 (requirement is 573.42+ / 580.95.05+ → met) |
| OS | Ubuntu 24.04.4 LTS |
| CUDA | 12.8 (container) |
| CloudXR runtime | 6.2.0 (also reproduced on 6.2.1) |
isaacteleop |
1.3.131 (also reproduced on 1.4.89rc1) |
| Omniverse Kit XR | engineVersion 107039000, OpenXR apiVersion 1.1.45 |
| NVENC API header (server) | 12.0 |
| Headset | Apple Vision Pro, isaac-xr-teleop-sample-client-apple |
| Device profile | auto-native and AppleVisionPro (AVP — both fail identically) — vs auto-webrtc (Pico), which works |
| Runtime launch | python.sh -m isaacteleop.cloudxr --accept-eula (in-container, --network host, --ipc host, --gpus all) |
GPU memory during the failure: used ≈ 11.4 GB, free ≈ 20.7 GB / 32 GB (not OOM). NVENC concurrent-session limit on current consumer drivers is 8; the AVP path uses 4.
Steps to reproduce
- Start the CloudXR runtime with either AVP profile — both reproduce the failure:
NV_DEVICE_PROFILE=auto-native python.sh -m isaacteleop.cloudxr --accept-eulaorNV_DEVICE_PROFILE=AppleVisionPro python.sh -m isaacteleop.cloudxr --accept-eula - Launch an Isaac Lab XR app (
xr=True), Start AR. - Connect the Apple Vision Pro client.
Expected
Video streams to the AVP (as it does for Pico / as CloudXR 5.0.1 does — see Workaround).
Actual
Client connects, all streams negotiate, then encode fails and the session drops.
Server log (cxr_server.*.log)
INFO [verifyAndLogStreamSizes] Client connected, color stream size: 2400x1920
INFO [verifyAndLogStreamSizes] Client connected, alpha/depth stream size: 2400x1920 (x2 views -> 4 streams)
INFO [onStreamEventRaised] Video stream connected: color_0 / color_1 / alpha_depth_0 / alpha_depth_1
ERROR [pushStreamData] VideoStream: nvstPushStreamData() failed: NVST_R_ERROR_PIPELINE_INITIALIZATION_FAILED
ERROR [pushStreamData] VideoStream: nvstPushStreamData() failed: NVST_R_BUSY
Stream-SDK log (cxr_streamsdk.*.log)
(W) <VideoSender> Client requested an unknown streaming format 2400x1920@90hz! Results may be unpredictable.
(M) <NvEnc12VideoEnco> Stream:0 HEVC video format selected (Streams 0-3, all HEVC, PRESET_NVENC_P2, CBR)
(E) <VideoEncoder> Could not get encoded frame.
(W) <NvEnc12VideoEnco> Failed to allocate completion event for end of input stream
(M) <AdapterContext> GPU memory usage: used = 11384 MB, free = 20721 MB, total = 32106 MB
(E) <ClientControl> Control channel operation canceled by a session event. Error: 80000013
Relevant disconnect codes present in the runtime libs: NVST_DISCONN_SERVER_VIDEO_ENCODER_INIT_HEVC_NOT_SUPPORTED, NVST_DISCONN_SERVER_VIDEO_ENCODER_ASYNC_MODE_EVENT_WAIT_FAILED.
Key evidence: this is NOT a hardware/driver limitation
ffmpeg NVENC, run inside the same container, bypassing CloudXR:
hevc_nvenc @ 2400x1920 -> OK
h264_nvenc @ 2400x1920 -> OK
4x concurrent hevc_nvenc @ 2400x1920 -> OK (all 4 succeed)
av1_nvenc -> "No capable devices found" (stock-ffmpeg build quirk; CloudXR itself uses AV1 fine)
So the GPU + driver + NVENC encode HEVC at the exact AVP resolution and stream count. The failure is specific to CloudXR 6.2.x’s HEVC server encode path (under both AVP profiles — see Not profile-specific).
Working comparison — Pico on the same machine (cxr_streamsdk from a good session):
deviceProfile: auto-webrtc
Stream:0 AV1 video format selected (2 streams, 4096x4032@90)
=> 0 encode errors, streams for minutes
The "unknown streaming format" warning is benign — it also appears in the working AV1 session (at 4096×4032).
Direct 5.0.1 vs 6.2.x comparison (same machine, same client, same app)
Running the identical Isaac app + Apple Vision Pro client against a CloudXR 5.0.1 runtime (XR_RUNTIME_JSON pointed at a shared /openxr volume) works — video + finger tracking stream for minutes. Inputs are identical to the failing 6.2.x case (auto-native, 4 HEVC streams at 2400×1920, same “unknown streaming format” warning). The only variable is the CloudXR runtime version.
| Signature | CloudXR 5.0.1 (works) | CloudXR 6.2.x (fails) |
|---|---|---|
| Device/encoder configuration timing | up front at session start | Device configuration deferred until client connects |
NVENC_CODEC_HEVC is supported logged (caps probe) |
yes (also H264, AV1, 10-bit) | none |
| pre-generates SPS/PPS headers for standard resolutions | yes | none |
| Pre-headset PC state | session active → VR view + “connected” | “waiting for connection”, no view |
Could not get encoded frame |
0 | many |
PIPELINE_INITIALIZATION_FAILED |
0 | yes |
| encode activity / frames streamed | yes (minutes) | none |
Failed to allocate completion event ... end of input stream |
yes (benign, at teardown) | yes |
Leading mechanism: 6.2.x logs Device configuration deferred until client connects and skips the up-front NVENC codec capability probe / SPS/PPS pre-generation that 5.0.1 performs. The 6.2.x encoder is therefore configured lazily, from the connecting client’s requested format (the AVP’s 2400×1920, 4-stream HEVC), and that deferred HEVC init is what fails (Could not get encoded frame / PIPELINE_INITIALIZATION_FAILED). 5.0.1 validates and configures the encoder up front and streams the identical format cleanly.
Note: the Failed to allocate completion event line appears in both the working (5.0.1, at session teardown) and failing (6.2.x) logs, so it is a benign teardown warning, not the cause.
Tried the specific AppleVisionPro profile (vs auto-native) on 6.2.x: the runtime accepts it (deviceProfile: apple-vision-pro) and now pre-configures the device up front (Device configuration created from profile, and Isaac Sim renders the VR view before the client connects — unlike auto-native’s deferred config). But the HEVC encode still fails identically (Could not get encoded frame → NVST_R_BUSY → drop), and 6.2.x still never runs the up-front NVENC caps probe (NVENC_CODEC_HEVC is supported) that 5.0.1 does. So the failure is independent of the deferred-config / device profile — it is specifically the 6.2.x HEVC encoder-init path that cannot produce frames on this GPU, whereas 5.0.1 (and ffmpeg hevc_nvenc) can.
What we tried (all server-side, no effect)
NV_GPU_INDEX=0(single GPU is index 0) — no change.NV_CXR_DOWNSAMPLE_AMOUNT=2— no effect on encoded resolution (still 2400×1920).NV_CXR_RUNTIME_FOVEATION=1— no change.NV_CXR_STREAMING_BITS_PER_CHANNEL=8+NV_CXR_RUNTIME_BITS_PER_CHANNEL=8(force 8-bit, in case of HEVC Main10) — no change.- Upgraded
isaacteleop1.3.131 → 1.4.89rc1 (CloudXR 6.2.0 → 6.2.1) — identical failure. - No server-side codec-preference / disable-HEVC / force-sync knob found in the runtime.
Workaround (confirms a regression)
CloudXR 5.0.1 (nvcr.io/nvidia/cloudxr-runtime:5.0.1) streams to the same Apple Vision Pro on this same workstation + driver — confirmed working: with the Isaac app’s XR_RUNTIME_JSON pointed at a 5.0.1 runtime (shared /openxr volume), the AVP enters teleop and video + finger tracking stream normally, using the same client that fails against 6.2.x. This confirms a CloudXR 5.0.1 → 6.2.x regression on the Linux auto-native (HEVC) path — the only variable changed between working and failing is the CloudXR runtime version.
Questions for NVIDIA
- Why does 6.2.x defer device/encoder configuration until the client connects (and skip the up-front NVENC caps probe + SPS/PPS pre-generation that 5.0.1 does)? The deferred HEVC init, driven by the AVP’s 2400×1920 format, is where it fails.
- Is
auto-nativeHEVC encode expected to work on Linux with CloudXR 6.2.x, or is the native profile effectively Windows-only? Is there a server-side flag to pre-configure/validate the encoder, or to select AV1/H.264 for the native path? - Is this a known 5.0.1 → 6.2.x regression for Apple Vision Pro on Linux + Blackwell? (
ffmpeg hevc_nvencand CloudXR 5.0.1 both encode this exact format fine on the same machine — only the 6.2.x runtime fails.)