How to increase the exclusive lock of carb.datastore.plugin？

Omniverse Isaac Sim
1

Isaac Sim Version

4.5.0
4.2.0
[✅] 4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

[✅] Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model: NVIDIA GeForce RTX 4090
  • Driver Version: 550.90.07

Topic Description

When I launch SimulationApp using a conda environment with isaac-sim 4.1.0 on a Slurm cluster with multiple 4090 GPUs (> 64), I get the following error. When the number of tasks is less than 64, this situation will not occur. It looks like carb.datastore.plugin has a file lock, and the maximum is set to 64. How can I increase this value?

2025-02-27 14:00:50 [55,135ms] [Error] [carb.datastore.plugin] Failed to acquire exclusive lock to data store (64>=64)
2025-02-27 14:00:50 [55,138ms] [Error] [carb.datastore.plugin] Failed to create local file data store at '~/.conda/envs/name/lib/python3.10/site-packages/omni/cache/Kit/106.0/3a7abd1c/PhysXLocalMeshCache/LocalDS0'
2025-02-27 14:00:51 [56,185ms] [Error] [carb.datastore.plugin] Failed to acquire exclusive lock to data store (64>=64)
2025-02-27 14:00:51 [56,186ms] [Error] [carb.datastore.plugin] Failed to create local file data store at '~/.conda/envs/name/lib/python3.10/site-packages/omni/cache/Kit/106.0/3a7abd1c/PhysXLocalMeshCache/LocalDS0'
slurmstepd: error: *** JOB 1184 ON g1052 CANCELLED AT 2025-02-27T22:11:45 ***
3

We’re currently checking with our team regarding your issue. In the meantime, could you please try using Isaac Sim version 4.5.0?

4

Please try running the application with the following option to disable UJITSO-based physX cooking and see if the issue persists.
--/physics/cooking/ujitsoCollisionCooking=false

6

I add this simulation_app._carb_settings.set(“/physics/cooking/ujitsoCollisionCooking”, False)
after launching SimulationApp, but this error still exsit.

7

Please try launching your application with the command line option directly. Setting it after launching may not work.

8

Like this? or where insert the command?

CONFIG = {"/physics/cooking/ujitsoCollisionCooking": False}
simulation_app = SimulationApp(launch_config=CONFIG)
9

No, running your command with --/physics/cooking/ujitsoCollisionCooking=false.

10

Does the option resolve this problem?

11

Sorry to have kept you waiting, I did not start with --/physics/cooking/ujitsoCollisionCooking=false. I checked my program log again according to the original startup method. Although the error mentioned above will be reported when more than 64 isaac sims are started at one time, it will not freeze and will be abnormally normal. The program can run normally.

12

Closing this thread based on the findings here.

The carb.datastore.plugin / omni.datastore lock error is non-fatal. When a process cannot acquire the shared cache lock, it falls back to local PhysX cooking and the simulation continues normally. As @yioutpi confirmed in post #11, programs run fine despite the error.

Summary for anyone landing here:

  • The lock limit is 64 in Isaac Sim 4.x and 256 in 5.x.
  • If you exceed the limit, you’ll see the error in logs but your simulation will still work.
  • To suppress the error, launch with --/physics/cooking/ujitsoCollisionCooking=false (disables the shared PhysX cooking cache entirely).
  • Making the lock limit user-configurable has been noted as a feature request internally.

A related thread for Isaac Sim 5.1.0 is tracked here: Omni.datastore error.