"headless mode" has lower rewords and is slower than "non-headless mode" (Preview3)

[DDPG7]

Hi
“headless mode” has lower rewords and is slower in my task.
Camera sensors are not used.
I repeated it several times and the results are reproducible.
Does it make a difference originally?

• non-headless mode (CHECKED “Show only selected env” box)
  If you don’t check “Show only selected env” box, “headless mode” is a little faster.

(isaac) sa@Fossa:~/wsp/isaac_3/IsaacGymEnvs-main/isaacgymenvs$ python3 train.py task=TASK num_envs=512 headless=False
.
.
.
RunningMeanStd:  (18,)
fps step: 4180.0 fps step and policy inference: 4133.7  fps total: 3950.8
fps step: 5193.4 fps step and policy inference: 5124.5  fps total: 4843.9
fps step: 5198.8 fps step and policy inference: 5130.4  fps total: 4859.5
fps step: 5223.9 fps step and policy inference: 5155.5  fps total: 4880.9
fps step: 5177.4 fps step and policy inference: 5106.6  fps total: 4801.2
fps step: 5177.9 fps step and policy inference: 5109.5  fps total: 4840.8
fps step: 5206.9 fps step and policy inference: 5138.2  fps total: 4864.6
fps step: 5189.3 fps step and policy inference: 5120.2  fps total: 4852.1
fps step: 5167.0 fps step and policy inference: 5098.8  fps total: 4830.5
fps step: 5147.7 fps step and policy inference: 5078.8  fps total: 4814.3
fps step: 5159.8 fps step and policy inference: 5092.3  fps total: 4825.8
fps step: 5142.4 fps step and policy inference: 5074.5  fps total: 4809.1
fps step: 5146.6 fps step and policy inference: 5080.1  fps total: 4816.4
fps step: 5137.0 fps step and policy inference: 5069.9  fps total: 4806.9
fps step: 5163.6 fps step and policy inference: 5094.7  fps total: 4796.6
fps step: 5201.1 fps step and policy inference: 5131.7  fps total: 4862.9
saving next best rewards:  [501.86]
=> saving checkpoint 'runs/TASK/nn/TASK.pth'

• step time

fps step: 4915.9 fps step and policy inference: 4851.4  fps total: 4604.1
step time:  0.09714865684509277
step time:  0.09894943237304688
step time:  0.09816908836364746
step time:  0.09942102432250977
step time:  0.09969496726989746
step time:  0.09773898124694824
step time:  0.0977473258972168
step time:  0.09895539283752441
step time:  0.09899091720581055
step time:  0.09884309768676758
step time:  0.09858965873718262
step time:  0.09852981567382812
step time:  0.09790897369384766
step time:  0.09991931915283203
step time:  0.09973311424255371
step time:  0.09845280647277832
step time:  0.0972590446472168
step time:  0.09922337532043457
step time:  0.10511302947998047
step time:  0.10057592391967773
step time:  0.10012102127075195
step time:  0.09983587265014648
step time:  0.10001063346862793
step time:  0.1003265380859375
step time:  0.1001579761505127
step time:  0.10090065002441406
step time:  0.0988917350769043
step time:  0.09976768493652344
step time:  0.09978723526000977
step time:  0.10020017623901367
step time:  0.09939861297607422
step time:  0.09771966934204102
fps step: 5150.3 fps step and policy inference: 5080.1  fps total: 4810.5
saving next best rewards:  [501.86]

• headless mode

(isaac) sa@Fossa:~/wsp/isaac_3/IsaacGymEnvs-main/isaacgymenvs$ python3 train.py task=TASK num_envs=512 headless=True
.
.
.
RunningMeanStd:  (18,)
fps step: 3818.4 fps step and policy inference: 3780.7  fps total: 3631.3
fps step: 3845.5 fps step and policy inference: 3808.6  fps total: 3658.6
fps step: 3812.6 fps step and policy inference: 3776.7  fps total: 3629.0
fps step: 3783.7 fps step and policy inference: 3747.4  fps total: 3601.9
fps step: 3785.2 fps step and policy inference: 3749.7  fps total: 3601.1
fps step: 3792.7 fps step and policy inference: 3756.3  fps total: 3608.1
fps step: 3784.5 fps step and policy inference: 3748.0  fps total: 3602.0
fps step: 3732.3 fps step and policy inference: 3696.1  fps total: 3554.7
fps step: 3795.1 fps step and policy inference: 3759.6  fps total: 3597.5
fps step: 3716.4 fps step and policy inference: 3678.7  fps total: 3537.4
fps step: 3645.1 fps step and policy inference: 3609.8  fps total: 3468.1
fps step: 3666.9 fps step and policy inference: 3630.3  fps total: 3487.1
fps step: 3643.4 fps step and policy inference: 3608.3  fps total: 3469.7
fps step: 3589.3 fps step and policy inference: 3555.3  fps total: 3422.4
fps step: 3660.7 fps step and policy inference: 3625.3  fps total: 3481.5
fps step: 3625.8 fps step and policy inference: 3590.5  fps total: 3454.8
saving next best rewards:  [487.94]
=> saving checkpoint 'runs/TASK/nn/TASK.pth'

• step time

fps step: 3760.0 fps step and policy inference: 3679.6  fps total: 3537.8
step time:  0.13702750205993652
step time:  0.1348130702972412
step time:  0.13611578941345215
step time:  0.13491511344909668
step time:  0.12049508094787598
step time:  0.1370859146118164
step time:  0.14050817489624023
step time:  0.13720059394836426
step time:  0.1355893611907959
step time:  0.13829541206359863
step time:  0.14522981643676758
step time:  0.14173007011413574
step time:  0.14649486541748047
step time:  0.13808751106262207
step time:  0.1344773769378662
step time:  0.13517355918884277
step time:  0.13549304008483887
step time:  0.13528800010681152
step time:  0.13478851318359375
step time:  0.13663125038146973
step time:  0.1363523006439209
step time:  0.13636279106140137
step time:  0.1353926658630371
step time:  0.13617706298828125
step time:  0.15598702430725098
step time:  0.13667941093444824
step time:  0.13668012619018555
step time:  0.1377854347229004
step time:  0.13624310493469238
step time:  0.13564300537109375
step time:  0.1336984634399414
step time:  0.13290858268737793
fps step: 3733.5 fps step and policy inference: 3697.3  fps total: 3553.5
saving next best rewards:  [487.94]

[gstate]

Hi @DDPG7,

That’s generally quite a low framerate for training, though you’re training with relatively few environments (we typically train complex environments like ShadowHand with 8k or 16k environments). It’s definitely surprising to see results like this.

Do you get this kind of performance with the example tasks, or only with your own?

If you are using a camera sensor in your scene, you won’t be able to get high frame rates in general, as CPU readback of physics state is required before rendering. We hope to improve this in the future in Omniverse.

Take care,
-Gav

[DDPG7]

@gstate

Do you get this kind of performance with the example tasks, or only with your own?

My own task. I created it with reference to “Franka Cabinet”.

But, I understand the cause.
I put the following “if statement” in the “compute_observations” function. I misunderstood “cfg[“env”][“headless”]” as a replacement for “command line argument”.
I thought it would replace “headless” as well as “numEnvs” in the “yaml” file.
Even if I set “headless” to “True” on the command line, “self.cfg [” env “] [” headless “]” remains “False” as the “yaml file”.

In the case of “Preview2”, I set “command line argument” was “–headless”. And I set “self.cfg [” headless “]” correctly.

I think there are probably people who do the same thing as I do.

if not self.cfg["env"]["headless"]:
    self.gym.clear_lines(self.viewer)
    for i in range(self.num_envs):
         ...

Also I ran “Franka Cabinet”, “Headless mode” is faster.

• Headless mode

python train.py task=FrankaCabinet headless=True
.
.
RunningMeanStd:  (23,)
time:  0.032996177673339844
time:  0.018712282180786133
time:  0.021856307983398438
time:  0.019181251525878906
time:  0.018357515335083008
time:  0.019512176513671875
time:  0.019088268280029297
time:  0.017313718795776367
time:  0.019658565521240234
time:  0.018935680389404297
time:  0.017935991287231445
time:  0.019405603408813477
time:  0.017279624938964844
time:  0.018705129623413086
time:  0.019415616989135742
time:  0.017787694931030273
fps step: 206214.8 fps step and policy inference: 191194.8  fps total: 113829.9

• NON-Headless mode

python train.py task=FrankaCabinet headless=False
.
.
RunningMeanStd:  (23,)
time:  0.5187857151031494
time:  0.26894044876098633
time:  0.2727789878845215
time:  0.2745029926300049
time:  0.2680506706237793
time:  0.26672935485839844
time:  0.26893138885498047
time:  0.059188127517700195←CHECKED “Show only selected env” box
time:  0.056133270263671875
time:  0.05603981018066406
time:  0.05620884895324707
time:  0.05616950988769531
time:  0.05588388442993164
time:  0.0568387508392334
time:  0.056227922439575195
time:  0.056523799896240234
fps step: 24734.5 fps step and policy inference: 24529.2  fps total: 22545.3