Hi everyone,
I run a robotics company developing autonomous UGV/rover-class robots that operate in GPS-denied environments. Each robot carries multiple camera feeds, LiDAR, depth sensors, thermal imagers, and environment detection models running on Jetson Orin Nano Super at the edge.
We’re designing our training infrastructure and would love feedback from anyone who has deployed Isaac Sim and Isaac Lab at scale.
Proposed Setup
- 4× DGX Spark (expanding to 8× later) connected via Mikrotik CRS804-DDQ (4× 400GbE, supports up to 8× Sparks from a single switch)
- 1× RTX 5090 workstation dedicated to Isaac Sim GUI authoring (environment design, sensor rig setup)
- Jetson Orin Nano Super on each robot for edge inference
- Sim-to-real pipeline: author in Isaac Sim → train headless on Spark cluster → validate HIL → deploy to Orin
Rationale for DGX Spark over RTX Pro 6000 Blackwell
Locally, the RTX Pro 6000 Blackwell with a workstation server runs to approximately ₹14–15 lakhs, while each DGX Spark is ₹4.3 lakhs. Four Sparks at ₹17.2L gives us 512 GB unified memory, comparable FP32 throughput (~124 TFLOPS), and a modular path to 8 nodes, all under the cost of the Pro 6000 workstation. For our workload (headless, long-running training iterations numbering in the hundreds of thousands), the memory capacity and scalability felt more relevant than the Pro 6000’s bandwidth advantage.
Questions
- Isaac Lab multi-node on DGX Spark: Has anyone validated Isaac Lab’s PyTorch multi-node training (via `torchrun` / `torch.distributed`) across DGX Spark nodes connected over 200GbE ConnectX-7? What kind of scaling efficiency are you seeing? are gradient sync overheads significant at 4 or 8 nodes?
- Sensor-heavy environments: Our sim environments involve multi-camera (RGB + depth + thermal), LiDAR point clouds, and real-time particulate/dust occlusion in confined geometry. Are there known memory or performance bottlenecks in Isaac Sim when running these sensor configurations headless at scale?
- Sim-to-real with Orin Nano Super: Any specific recommendations for maintaining policy fidelity when transferring from Isaac Lab (trained on Spark cluster) to Jetson Orin Nano Super? Domain randomization parameters, sensor noise modeling, or export format recommendations (TensorRT, ONNX) would be very helpful.
- RTX 5090 for Isaac Sim authoring: Is the RTX 5090 (non-Pro, consumer Blackwell) a practical choice for the authoring workstation specifically for RT-based rendering, Omniverse viewport, and sensor rig visualization? or are there driver/feature limitations versus the Pro 6000 that would affect the Isaac Sim workflow?
- Network topology: Is the 200GbE direct-link clustering (as described in the DGX Spark cluster playbook) sufficient for Isaac Lab’s inter-node gradient communication, or would InfiniBand-class interconnects make a meaningful difference for RL workloads at 4–8 nodes?
Any insights from people running similar setups, especially in robotics, physical AI, or Isaac Lab at multi-node scale, it would be genuinely appreciated.
Thanks in advance.
Hi @adityaharsh, thanks for the detailed write-up — this is a well-structured question and a compelling use case. Let me address each of your five questions as best I can based on current knowledge:
1. Isaac Lab multi-node on DGX Spark (200GbE ConnectX-7)
Isaac Lab’s multi-node support via torchrun / torch.distributed is functional but still maturing. The primary bottleneck in RL workloads is typically environment throughput (physics + rendering steps per second) rather than gradient synchronization, since RL gradient tensors are relatively small compared to supervised learning. At 4–8 nodes, 200GbE ConnectX-7 is generally sufficient for gradient sync — InfiniBand would matter more for large-batch supervised learning or very high-frequency all-reduce. That said, DGX Spark’s NVLink-C2C within each node is a significant advantage for intra-node memory bandwidth. We are not aware of published community benchmarks specifically on DGX Spark clusters for Isaac Lab; if you run benchmarks, sharing them would be valuable to the community. Make sure NCCL is configured to prefer the ConnectX-7 interface (NCCL_SOCKET_IFNAME, NCCL_IB_DISABLE=0 if using RoCE).
2. Sensor-heavy headless environments
Multi-camera + LiDAR + thermal in a single Isaac Sim environment is memory-intensive. Known considerations:
- RTX sensor rendering (depth, RGB) is GPU-memory-bound; each camera adds significant VRAM pressure per environment instance. With 128 GB unified memory per Spark node, you have headroom, but watch for CPU↔GPU transfer overhead if sensors are read back to host frequently.
- LiDAR point cloud generation (especially RTX Lidar) is expensive per-step. Consider using pattern-based (non-RTX) lidar for training if full ray-traced fidelity isn’t required, and reserve RTX lidar for validation.
- Dust/particulate occlusion via Replicator or custom shaders adds rendering cost; profile per-step render time vs. physics step time to find your bottleneck.
- Running headless with
--headless and disabling the viewport entirely is essential for throughput.
3. Sim-to-real with Jetson Orin Nano Super
Key recommendations:
- Domain randomization: Randomize camera intrinsics, exposure, lighting, LiDAR return intensity, and sensor noise (Gaussian + dropout) aggressively. For GPS-denied/confined environments, also randomize geometry clutter and surface reflectance.
- Export: Export policies to ONNX first, then convert to TensorRT on the Orin using
trtexec targeting the Orin’s Ampere GPU. Use FP16 or INT8 quantization with calibration data from your sim distribution. Isaac Lab’s export_policy_as_onnx utility is the recommended path.
- Observation space alignment: Ensure your sim observation normalization and sensor preprocessing exactly match what runs on the Orin at inference time — mismatches here are a common source of sim-to-real gap.
- Consider running a small HIL loop where the Orin receives sim sensor streams over ROS 2 before full physical deployment.
4. RTX 5090 for Isaac Sim authoring
The RTX 5090 (consumer Ada/Blackwell) is a practical choice for Isaac Sim GUI authoring — RT rendering, Omniverse viewport, and sensor rig visualization all work well on consumer RTX cards. The main differences vs. RTX Pro 6000 Blackwell for this workflow are: (a) ECC memory (Pro only, less relevant for authoring), (b) driver certification for enterprise Omniverse (Pro cards use Studio/enterprise drivers), and (c) slightly higher memory bandwidth on the Pro. For authoring-only use (not production rendering or 24/7 uptime), the RTX 5090 is fine. Use the latest Game Ready or Studio driver and verify it appears on the Isaac Sim platform requirements matrix for your target Isaac Sim version.
5. Network topology: 200GbE vs. InfiniBand for RL
For Isaac Lab RL workloads at 4–8 nodes, 200GbE RoCE (ConnectX-7) is almost certainly sufficient. RL gradient payloads are small (policy networks for locomotion/navigation are typically <10M parameters), and the dominant cost is simulation throughput, not communication. InfiniBand would provide lower latency and higher bandwidth but would likely yield <5% end-to-end training speedup for typical Isaac Lab RL tasks at this node count. Your Mikrotik CRS804-DDQ topology is reasonable; ensure jumbo frames (MTU 9000) are enabled end-to-end and DCQCN/ECN is configured for RoCE to avoid congestion drops.
Overall your architecture is well-reasoned. The main open question is whether Isaac Lab’s multi-node distributed mode has been validated specifically on DGX Spark — we’d recommend opening a follow-up with the Isaac Lab GitHub team (GitHub - isaac-sim/IsaacLab: Unified framework for robot learning built on NVIDIA Isaac Sim · GitHub) for the most current status on multi-node support and any known issues. Good luck with the deployment!
Hi @VickNV as per your suggestion I opened a new issue, and this is what I was replied with [Question] Multi-Node Distributed RL Validation Status on DGX Spark · Issue #5707 · isaac-sim/IsaacLab · GitHub
“The main thing I would watch on DGX Spark is memory bandwidth. The 128GB vram on the spark is excellent for inference of large models and high Lab env count, but the memory bandwidth will likely be the limiting factor for training.”
My Query after this,
One follow-up on the memory bandwidth point: could you help clarify whether bandwidth constraints affect training wall-clock time (steps/sec throughput) or whether there are scenarios where it could also affect convergence quality (e.g. forcing smaller batch sizes, fewer parallel environments, or truncated rollouts that change the effective sample distribution)? Trying to understand whether it’s purely a speed concern or whether it can have downstream effects on policy quality at scale.
Also, we’re evaluating several hardware configurations and would appreciate any guidance on relative training performance for our Isaac Lab RL workload (headless, sensor-heavy: multi-camera RGB+depth+thermal, RTX LiDAR, GPS denied confined geometry):
| Configuration |
VRAM |
Notes |
| RTX 5090 (single) |
32GB |
Consumer Blackwell, PCIe |
| RTX Pro 5000 Blackwell (single) |
48GB |
Pro Blackwell workstation |
| RTX Pro 6000 Blackwell (single) |
96GB |
Pro Blackwell workstation |
| DGX Spark × 1 |
128GB unified |
GB10 |
| DGX Spark × 2 cluster |
256GB unified |
200GbE ConnectX-7 |
| DGX Spark × 4 cluster |
512GB unified |
200GbE ConnectX-7 |
Specific questions:
-
For the GPU-based workstation options (RTX 5090, Pro 5000, Pro 6000), what system RAM would you recommend alongside the GPU? Is there a minimum to avoid the CPU & GPU transfer becoming a bottleneck when reading back sensor observations?
-
How does the DGX Spark’s unified memory architecture compare to a discrete GPU + system RAM setup for Isaac Lab specifically does the unified model meaningfully reduce transfer overhead for sensor heavy environments?
-
At what point in scaling (single Spark to 2-node to 4-node) does the inter-node communication overhead start to noticeably offset the gains from additional parallel environments?
This would really help us calibrate whether the cluster investment is justified vs a high-end single node workstation for our workload. Thanks in advance!
Hi @adityaharsh,
Thanks for the detailed write-up, first to answer your question, the memory bandwidth should mostly affect runtime performance and not training quality. VRAM size would affect how many environments/sensors your setup can support.
I’d like to flip the question around, because I think it’ll get you a much more useful answer. Right now the proposal is framed around a hardware budget (“is this rig good enough?”), but the thing that actually determines whether any given setup will work is the simulation workload you intend to run. If we can pin that down, we can help you determine whether a compute hardware can meet the requirements.
To do that, could you share a rough estimate of what you’re trying to simulate/train? The things that drive the hardware requirements most are:
- Parallel environments — roughly how many Isaac Lab envs do you want running concurrently per node? (This is usually the single biggest VRAM/throughput driver for RL training.)
- Sensors per robot — how many cameras, what type (RGB, depth, segmentation), and at what resolution and frame rate? Rendering-based sensors are far more expensive than state-only observations, so even a rough count changes the picture a lot.
- Scene complexity — approximate triangle count / asset detail of your environments, and whether the dust/occlusion effects you mentioned are rendered or simulated as sensor noise.
- Training target — are you doing pure RL policy training (often state-based, very GPU-efficient) or are cameras/LiDAR in the observation loop (vision-based, much heavier)?
- Throughput goal — any target for steps/sec or wall-clock time to convergence?
Even ballpark numbers for the above will let us estimate whether your proposed cluster is sized appropriately, over-spec’d, or short — and where the bottleneck is likely to be (render, physics, or gradient sync).
Once we have that, we can address your specific questions (multi-node scaling, RTX 5090 authoring, sim-to-real on Orin Nano) in the context of your actual numbers.
@adityaharsh
We noticed that this topic hasn’t received any recent updates from anyone reporting this issue, so we are closing it for now to help keep the forum organized.
If you are still experiencing this issue or have related questions, please create a GitHub Discussion or Issue in the Isaac Sim repository and include a link to this topic along with updated details. Mentioning or linking to this original topic provides helpful context and makes it easier for others to assist you.
Thank you for being part of the NVIDIA Isaac Sim community.
Best regards,
The NVIDIA Isaac Sim Forum Team