I’m evaluating my hardware’s performance using the following guide:
When I run the SDG images per second (complex) benchmark:
./python.sh standalone_examples/benchmarks/benchmark_sdg.py \
--num-cameras 2 --resolution 1280 720 --asset-count 100 \
--annotators all --disable-viewport-rendering \
--skip-write --headless
I get a mean FPS of 1.56. Do you know why the FPS is so low?
Spec:
Ubuntu 22.04
GPUs: 2ADA6000 + 2 A5000
CPU: AMD® Ryzen threadripper pro 3975wx 32-cores × 64
RAM: 256 Go
Here is the output from my terminal:
2025-03-24 13:10:12 [33,979ms] [Warning] [omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.
2025-03-24 13:10:16 [38,128ms] [Warning] [isaacsim.benchmark.services.recorders] Detected multiple GPU types: ['NVIDIA RTX 6000 Ada Generation', 'NVIDIA RTX 6000 Ada Generation', 'NVIDIA RTX A5000', 'NVIDIA RTX A5000'].
2025-03-24 13:10:16 [38,128ms] [Warning] [isaacsim.benchmark.services.recorders] Only recording GPU 0 type: NVIDIA RTX 6000 Ada Generation
[SDG Benchmark] Starting SDG..
2025-03-24 13:10:16 [38,259ms] [INFO] [isaacsim.benchmark.services.base_isaac_benchmark] Starting phase: benchmark
Module omni.replicator.core.ogn.python._impl.nodes.OgnSemanticSegmentation a0ffa74 load on device 'cuda:0' took 0.47 ms (cached)
Module omni.replicator.core.ogn.python._impl.nodes.OgnSemanticSegmentation a0ffa74 load on device 'cuda:2' took 0.38 ms (cached)
Module omni.replicator.core.ogn.python._impl.nodes.OgnPointCloudGenerator 1f4060b load on device 'cuda:0' took 0.43 ms (cached)
Module omni.replicator.core.ogn.python._impl.nodes.OgnPointCloudGenerator 1f4060b load on device 'cuda:2' took 0.35 ms (cached)
2025-03-24 13:16:44 [425,942ms] [Warning] [isaacsim.benchmark.services.recorders] Detected multiple GPU types: ['NVIDIA RTX 6000 Ada Generation', 'NVIDIA RTX 6000 Ada Generation', 'NVIDIA RTX A5000', 'NVIDIA RTX A5000'].
2025-03-24 13:16:44 [425,942ms] [Warning] [isaacsim.benchmark.services.recorders] Only recording GPU 0 type: NVIDIA RTX 6000 Ada Generation
2025-03-24 13:16:44 [426,094ms] [INFO] [isaacsim.benchmark.services.base_isaac_benchmark] Stopping
2025-03-24 13:16:44 [426,095ms] [INFO] [isaacsim.benchmark.services.base_isaac_benchmark] Writing metrics data.
2025-03-24 13:16:44 [426,095ms] [INFO] [isaacsim.benchmark.services.base_isaac_benchmark] Metrics type = OmniPerfKPIFile
2025-03-24 13:16:44 [426,095ms] [INFO] [isaacsim.benchmark.services.metrics.backend]
loading Metrics:
workflow_name: benchmark_sdg
num_frames: 600
num_cameras: 2
width: 1280
height: 720
asset_count: 100
annotators: all
num_gpus: 4
phase: loading
System Memory RSS: 10.263 GB
System Memory VMS: 209.412 GB
System Memory USS: 10.212 GB
GPU Memory Tracked: 7.521 GB
GPU Memory Dedicated: 2.487 GB
System CPU iowait: 0.0 %
System CPU system: 4.0 %
System CPU user: 9.0 %
System CPU idle: 87.0 %
num_cpus: 64
gpu_device_name: NVIDIA RTX 6000 Ada Generation
Runtime: 19454.56 ms
2025-03-24 13:16:44 [426,095ms] [INFO] [isaacsim.benchmark.services.metrics.backend]
benchmark Metrics:
workflow_name: benchmark_sdg
num_frames: 600
num_cameras: 2
width: 1280
height: 720
asset_count: 100
annotators: all
num_gpus: 4
phase: benchmark
System Memory RSS: 10.497 GB
System Memory VMS: 258.225 GB
System Memory USS: 10.444 GB
GPU Memory Tracked: 7.506 GB
GPU Memory Dedicated: 2.476 GB
System CPU iowait: 0.0 %
System CPU system: 9.0 %
System CPU user: 27.0 %
System CPU idle: 64.0 %
num_cpus: 64
gpu_device_name: NVIDIA RTX 6000 Ada Generation
Mean App_Update Frametime: 640.85 ms
Stdev App_Update Frametime: 7.12 ms
Min App_Update Frametime: 626.13 ms
Max App_Update Frametime: 654.89 ms
Mean Physics Frametime: 0.18 ms
Stdev Physics Frametime: 0.01 ms
Min Physics Frametime: 0.16 ms
Max Physics Frametime: 0.19 ms
Mean GPU Frametime: 12.23 ms
Stdev GPU Frametime: 0.78 ms
Min GPU Frametime: 10.82 ms
Max GPU Frametime: 13.98 ms
Mean FPS: 1.56 FPS
Real Time Factor: 1.008
Runtime: 384643.526 ms
2025-03-24 13:16:44 [426,096ms] [INFO] [isaacsim.benchmark.services.metrics.backend] Writing metrics to /tmp/kpis_benchmark_sdg.json
|----------------------------------------------------|
| Summary Report |
|----------------------------------------------------|
| workflow_name: benchmark_sdg |
| num_frames: 600 |
| num_cameras: 2 |
| width: 1280 |
| height: 720 |
| asset_count: 100 |
| annotators: all |
| num_gpus: 4 |
| num_cpus: 64 |
| gpu_device_name: NVIDIA RTX 6000 Ada Generation |
|----------------------------------------------------|
| Phase: loading |
| System Memory RSS: 10.263 GB |
| System Memory VMS: 209.412 GB |
| System Memory USS: 10.212 GB |
| GPU Memory Tracked: 7.521 GB |
| Runtime: 19454.56 ms |
|----------------------------------------------------|
| Phase: benchmark |
| System Memory RSS: 10.497 GB |
| System Memory VMS: 258.225 GB |
| System Memory USS: 10.444 GB |
| GPU Memory Tracked: 7.506 GB |
| Mean FPS: 1.56 FPS |
| Real Time Factor: 1.008 |
| Runtime: 384643.526 ms |
| Frametimes (ms): mean | stdev | min | max |
| App_Update 640.85 | 7.12 | 626.13 | 654.89 |
| Physics 0.18 | 0.01 | 0.16 | 0.19 |
| GPU 12.23 | 0.78 | 10.82 | 13.98 |
|----------------------------------------------------|
2025-03-24 13:16:45 [427,270ms] [Warning] [carb] Plugin interface for a client: omni.hydratexture.plugin was already released.