Problem with CUDA Memory

Isaac Sim Version

4.0.0

Operating System

Ubuntu 22.04

Topic Description

Detailed Description

I tried to rerun my program but I can’t the program were working before and now they are not anymore. I don’t remeber all the things I have done before m program ne fonctionne plus but there is a list :

  • I followed the tutorial to use curobo with moveit2
  • I have downloaded and installed the IsaacSim v.4.5.0 maybe there is a conflict etween thise two.

Error Messages

When I run my program after a restart everything works fine but If I stopped it this message is shown :

2025-02-04 08:53:57 [1,270,888ms] [Warning] [omni.usd] Unexpected reference count of 3 for UsdStage 'anon:0x28d05cc0:World0.usd' while being closed in UsdContext (this may indicate it is still resident in memory).
[363.478s] Simulation App Shutting Down
2025-02-04T09:19:06.516473Z  INFO hub-002 ThreadId(13) hub::rpc::server_msgpack: hub/src/rpc/server_msgpack.rs:172: Connection from 127.0.0.1:46744 closed

and then when I restart the program:

2025-02-04 08:54:12 [5,316ms] [Warning] [carb.omniclient.plugin]  16194: OmniHub: ThreadId(03) Hub encountered error. Trying to reconnect to Hub. retry_reason="Hub failed to launch: Io(\"waited for file \\\"/tmp/hub-theobloesch-32343ABF.config.json\\\"\")"
2025-02-04 08:54:17 [10,358ms] [Warning] [carb.omniclient.plugin]  16194: OmniHub: ThreadId(03) Hub encountered error. Trying to reconnect to Hub. retry_reason="Hub failed to launch: Io(\"waited for file \\\"/tmp/hub-theobloesch-32343ABF.config.json\\\"\")"
2025-02-04 08:54:22 [15,407ms] [Warning] [carb.omniclient.plugin]  16194: OmniHub: ThreadId(03) Hub encountered error. Trying to reconnect to Hub. retry_reason="Hub failed to launch: Io(\"waited for file \\\"/tmp/hub-theobloesch-32343ABF.config.json\\\"\")"
2025-02-04 08:54:27 [20,465ms] [Warning] [carb.omniclient.plugin]  16194: OmniHub: ThreadId(03) Hub encountered error. Trying to reconnect to Hub. retry_reason="Hub failed to launch: Io(\"waited for file \\\"/tmp/hub-theobloesch-32343ABF.config.json\\\"\")"
2025-02-04 08:54:32 [25,532ms] [Warning] [carb.omniclient.plugin]  16194: OmniHub: ThreadId(03) Hub encountered error. Trying to reconnect to Hub. retry_reason="Hub failed to launch: Io(\"waited for file \\\"/tmp/hub-theobloesch-32343ABF.config.json\\\"\")"
2025-02-04T08:54:37.702598Z  INFO main ThreadId(01) hub: hub/src/hub.rs:54: return=Err(error sending request for url (http://127.0.0.1:14090/api/1.0/service/config)

Caused by:
    operation timed out)
Error: error sending request for url (http://127.0.0.1:14090/api/1.0/service/config)

Caused by:
    operation timed out
2025-02-04 08:54:37 [30,610ms] [Warning] [carb.omniclient.plugin]  16194: OmniHub: ThreadId(03) Hub encountered error. Trying to reconnect to Hub. retry_reason="Hub failed to launch: Io(\"waited for file \\\"/tmp/hub-theobloesch-32343ABF.config.json\\\"\")"
2025-02-04T08:54:42.751291Z  INFO main ThreadId(01) hub: hub/src/hub.rs:54: return=Err(error sending request for url (http://127.0.0.1:14090/api/1.0/service/config)

Or the program restart but very slowly:

2025-02-04T09:09:02.236100Z  INFO hub-002 ThreadId(13) lru_task:lru_update: hub_cache::block::cache: hub-cache/src/block/cache.rs:1320: return=Ok(())
2025-02-04T09:09:02.236141Z  INFO hub-002 ThreadId(13) lru_task:lru_update: hub_cache::block::cache: hub-cache/src/block/cache.rs:1320: return=Ok(())
2025-02-04T09:09:02.236168Z  INFO hub-002 ThreadId(13) lru_task:lru_update: hub_cache::block::cache: hub-cache/src/block/cache.rs:1320: return=Ok(())
warming up...
/home/theobloesch/curobo/src/curobo/rollout/dynamics_model/tensor_step.py:538: UserWarning: Applied workaround for CuDNN issue, install nvrtc.so (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:80.)
  return filter_signal_jit(signal, self._sma_kernel)
Curobo is Ready
2025-02-04 09:09:10 [18,868ms] [Warning] [omni.hydra] Mesh '/World/obstacles/limit_3_mesh' has corrupted data in primvar 'displayColor': buffer size 8 doesn't match expected size 36 in faceVarying primvars
2025-02-04 09:09:10 [18,868ms] [Warning] [omni.hydra] Mesh '/World/obstacles/rack_sp_mesh' has corrupted data in primvar 'displayColor': buffer size 8 doesn't match expected size 36 in faceVarying primvars
2025-02-04 09:09:10 [18,868ms] [Warning] [omni.hydra] Mesh '/World/obstacles/rack_rp_mesh' has corrupted data in primvar 'displayColor': buffer size 8 doesn't match expected size 36 in faceVarying primvars
2025-02-04 09:09:10 [18,869ms] [Warning] [omni.hydra] Mesh '/World/obstacles/rack_tp_mesh' has corrupted data in primvar 'displayColor': buffer size 8 doesn't match expected size 36 in faceVarying primvars
2025-02-04 09:09:10 [18,869ms] [Warning] [omni.hydra] Mesh '/World/obstacles/rack_stp_mesh' has corrupted data in primvar 'displayColor': buffer size 8 doesn't match expected size 36 in faceVarying primvars
2025-02-04 09:09:10 [18,869ms] [Warning] [omni.hydra] Mesh '/World/obstacles/rack_lp_mesh' has corrupted data in primvar 'displayColor': buffer size 8 doesn't match expected size 36 in faceVarying primvars
2025-02-04 09:09:10 [18,869ms] [Warning] [omni.hydra] Mesh '/World/obstacles/table_mesh' has corrupted data in primvar 'displayColor': buffer size 8 doesn't match expected size 36 in faceVarying primvars
2025-02-04 09:09:12 [21,152ms] [Warning] [omni.physx.tensors.plugin] Cannot assign transform to non-root articulation link at '/World/UF_ROBOT/root_joint/xarm6link_tcp'
2025-02-04 09:09:12 [21,152ms] [Warning] [omni.physx.tensors.plugin] Cannot assign velocities to rigid body at '/World/UF_ROBOT/root_joint/xarm6link_tcp'
2025-02-04 09:09:12 [21,152ms] [Warning] [omni.physx.tensors.plugin] Cannot assign transform to non-root articulation link at '/World/UF_ROBOT/root_joint/xarm6link_tcp'
2025-02-04 09:09:12 [21,152ms] [Warning] [omni.physx.tensors.plugin] Cannot assign velocities to rigid body at '/World/UF_ROBOT/root_joint/xarm6link_tcp'
Updating world, reading w.r.t. /World/UF_ROBOT
Obstacles read from stage 19
Updated World
[0.59958845 0.07874151]
[0.59958845 0.07874151]
[0.02631525 0.02381778]
[0.00159558 0.00159486]

Additional Information

I had also a warning message that the Cuda memory run out of place but it doesn’t appear anymore.

What I’ve Tried

I have tried to restart my PC to run older program that worked and example from Curobo but nothings works.

Thanks you in advane for all the help you would provide
Best regards

Please move on to the latest release (Isaac Sim 4.5)

Sorry, after rereading Curobo’s documentation, I realized that it is only compatible with Isaac Sim 4.0.0—that’s where my issue comes from. Do you know if Curobo will be available soon for the latest Isaac Sim release?"

Hi,

After retesting my code with the correct version of Isaac Sim (4.0.0) compatible with Curobo, I’m still facing the same issue. Could you provide some guidance to troubleshoot this problem?

Thank you in advance for your help—I really appreciate it!

Error message :

Traceback (most recent call last):
  File "/home/theobloesch/xArm6_Pick_Place/with_motion_gen/Pick_Place_demo_real.py", line 780, in <module>
    main()
  File "/home/theobloesch/xArm6_Pick_Place/with_motion_gen/Pick_Place_demo_real.py", line 625, in main
    curobo.config_motion_gen()
  File "/home/theobloesch/xArm6_Pick_Place/with_motion_gen/Pick_Place_demo_real.py", line 269, in config_motion_gen
    self.motion_gen.warmup(enable_graph=True, warmup_js_trajopt=False)
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1892, in warmup
    self.plan_single(
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1533, in plan_single
    result = self._plan_attempts(
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/motion_gen.py", line 2997, in _plan_attempts
    result = self._plan_from_solve_state(
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/motion_gen.py", line 3450, in _plan_from_solve_state
    traj_result = self._solve_trajopt_from_solve_state(
  File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/motion_gen.py", line 2819, in _solve_trajopt_from_solve_state
    traj_result = trajopt_instance.solve_any(
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/trajopt.py", line 805, in solve_any
    return self.solve_single(
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/trajopt.py", line 978, in solve_single
    return self._solve_from_solve_state(
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/trajopt.py", line 920, in _solve_from_solve_state
    traj_result = self._get_result(
  File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/theobloesch/curobo/src/curobo/wrap/reacher/trajopt.py", line 1337, in _get_result
    metrics = self.interpolate_rollout.get_metrics_cuda_graph(interpolated_trajs)
  File "/home/theobloesch/curobo/src/curobo/rollout/arm_base.py", line 461, in get_metrics_cuda_graph
    self._cu_out_metrics = self.get_metrics(self._cu_metrics_state_in)
  File "/home/theobloesch/curobo/src/curobo/rollout/arm_base.py", line 436, in get_metrics
    out_metrics = self.constraint_fn(state)
  File "/home/theobloesch/curobo/src/curobo/rollout/arm_base.py", line 396, in constraint_fn
    coll_constraint = self.primitive_collision_constraint.forward(
  File "/home/theobloesch/curobo/src/curobo/rollout/cost/primitive_collision_cost.py", line 173, in discrete_fn
    self._collision_query_buffer.update_buffer_shape(
  File "/home/theobloesch/curobo/src/curobo/geom/sdf/world.py", line 270, in update_buffer_shape
    self.create_from_shape(shape, tensor_args, collision_types)
  File "/home/theobloesch/curobo/src/curobo/geom/sdf/world.py", line 243, in create_from_shape
    self.primitive_collision_buffer = CollisionBuffer.initialize_from_shape(
  File "/home/theobloesch/curobo/src/curobo/geom/sdf/world.py", line 67, in initialize_from_shape
    distance_buffer = torch.zeros(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 28.00 MiB. GPU 0 has a total capacity of 5.77 GiB of which 17.94 MiB is free. Process 6314 has 2.89 GiB memory in use. Including non-PyTorch memory, this process has 2.81 GiB memory in use. Of the allocated memory 385.36 MiB is allocated by PyTorch, and 96.64 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Best regards,
Théo Bloesch

The branch isaac-3.2 of curobo works with Isaac Sim 4.2.
Could you please give it a try? Thanks!

Hi thanks for your response !

I’have tried to install the new brach of curobo with this command :
```omni_python_4.2 -m pip install -e .[isaacsim] --no-build-isolation``

But this Error shows up :

ERROR: Command errored out with exit status 1:
     command: /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/theobloesch/curobo3.2/setup.py'"'"'; __file__='"'"'/home/theobloesch/curobo3.2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /home/theobloesch/curobo3.2/
    Complete output (94 lines):
    /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'long-description-content-type' will not be supported in future versions. Please use the underscore name 'long_description_content_type' instead
      warnings.warn(
    /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
      warnings.warn(msg, warning_class)
    /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'long-description-content-type' will not be supported in future versions. Please use the underscore name 'long_description_content_type' instead
      warnings.warn(
    /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
      warnings.warn(msg, warning_class)
    running develop
    /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running egg_info
    writing src/nvidia_curobo.egg-info/PKG-INFO
    writing dependency_links to src/nvidia_curobo.egg-info/dependency_links.txt
    writing requirements to src/nvidia_curobo.egg-info/requires.txt
    writing top-level names to src/nvidia_curobo.egg-info/top_level.txt
    reading manifest template 'MANIFEST.in'
    warning: no directories found matching 'src/curobo/curobolib/*.so'
    adding license file 'LICENSE'
    writing manifest file 'src/nvidia_curobo.egg-info/SOURCES.txt'
    running build_ext
    /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/utils/cpp_extension.py:414: UserWarning: The detected CUDA version (11.5) has a minor version mismatch with the version that was used to compile PyTorch (11.8). Most likely this shouldn't be a problem.
      warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
    building 'curobo.curobolib.lbfgs_step_cu' extension
    /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/utils/cpp_extension.py:1965: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
    If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
      warnings.warn(
    Emitting ninja build file /home/theobloesch/curobo3.2/build/temp.linux-x86_64-cpython-310/build.ninja...
    Compiling objects...
    Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
    [1/2] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output /home/theobloesch/curobo3.2/build/temp.linux-x86_64-cpython-310/src/curobo/curobolib/cpp/lbfgs_step_kernel.o.d -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/torch/csrc/api/include -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/TH -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/THC -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/include/python3.10 -c -c /home/theobloesch/curobo3.2/src/curobo/curobolib/cpp/lbfgs_step_kernel.cu -o /home/theobloesch/curobo3.2/build/temp.linux-x86_64-cpython-310/src/curobo/curobolib/cpp/lbfgs_step_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' --threads=8 -O3 --ftz=true --fmad=true --prec-div=false --prec-sqrt=false -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=lbfgs_step_cu -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 -std=c++17
    FAILED: /home/theobloesch/curobo3.2/build/temp.linux-x86_64-cpython-310/src/curobo/curobolib/cpp/lbfgs_step_kernel.o
    /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output /home/theobloesch/curobo3.2/build/temp.linux-x86_64-cpython-310/src/curobo/curobolib/cpp/lbfgs_step_kernel.o.d -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/torch/csrc/api/include -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/TH -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/THC -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/include/python3.10 -c -c /home/theobloesch/curobo3.2/src/curobo/curobolib/cpp/lbfgs_step_kernel.cu -o /home/theobloesch/curobo3.2/build/temp.linux-x86_64-cpython-310/src/curobo/curobolib/cpp/lbfgs_step_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' --threads=8 -O3 --ftz=true --fmad=true --prec-div=false --prec-sqrt=false -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=lbfgs_step_cu -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 -std=c++17
    nvcc fatal   : Unsupported gpu architecture 'compute_89'
    [2/2] c++ -MMD -MF /home/theobloesch/curobo3.2/build/temp.linux-x86_64-cpython-310/src/curobo/curobolib/cpp/lbfgs_step_cuda.o.d -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/torch/csrc/api/include -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/TH -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/include/THC -I/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/include/python3.10 -c -c /home/theobloesch/curobo3.2/src/curobo/curobolib/cpp/lbfgs_step_cuda.cpp -o /home/theobloesch/curobo3.2/build/temp.linux-x86_64-cpython-310/src/curobo/curobolib/cpp/lbfgs_step_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=lbfgs_step_cu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/utils/cpp_extension.py", line 2105, in _run_ninja_build
        subprocess.run(
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/subprocess.py", line 526, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/theobloesch/curobo3.2/setup.py", line 88, in <module>
        setuptools.setup(
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
        return distutils.core.setup(**attrs)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
        self.run_command(cmd)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
        super().run_command(command)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.run()
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/command/develop.py", line 114, in install_for_development
        self.run_command('build_ext')
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
        self.distribution.run_command(command)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
        super().run_command(command)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.run()
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
        _build_ext.run(self)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
        self.build_extensions()
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/utils/cpp_extension.py", line 866, in build_extensions
        build_ext.build_extensions(self)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
        self._build_extensions_serial()
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
        self.build_extension(ext)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
        _build_ext.build_extension(self, ext)
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 547, in build_extension
        objects = self.compiler.compile(
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/utils/cpp_extension.py", line 679, in unix_wrap_ninja_compile
        _write_ninja_file_and_compile_objects(
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/utils/cpp_extension.py", line 1785, in _write_ninja_file_and_compile_objects
        _run_ninja_build(
      File "/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/exts/omni.isaac.ml_archive/pip_prebundle/torch/utils/cpp_extension.py", line 2121, in _run_ninja_build
        raise RuntimeError(message) from e
    RuntimeError: Error compiling objects for extension
    ----------------------------------------
  Rolling back uninstall of nvidia-curobo
  Moving to /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/lib/python3.10/site-packages/nvidia-curobo.egg-link
   from /tmp/pip-uninstall-4lnmhb58/nvidia-curobo.egg-link
ERROR: Command errored out with exit status 1: /home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/theobloesch/curobo3.2/setup.py'"'"'; __file__='"'"'/home/theobloesch/curobo3.2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
WARNING: You are using pip version 21.2.1+nv1; however, version 25.0.1 is available.
You should consider upgrading via the '/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/bin/python3 -m pip install --upgrade pip' command.
There was an error running python

Thanks in advance
Best regards

Hi theo,

This is not an answer but I’m running the same issue after I gave a try to IsaacLab2 and IsaacSim4.5 and switching back to IsaacLab1.2.0 and IsaacSim4.2.0 with the original conda env.

Watching the logs I’ve noticed that the issue is related to CONDA_ENV_PATH/lib/python3.10/site-packages/omni/extscore/omni.client

Unluckily we won’t get any help since IsaacSim older than 4.5 is no longer supported.

I’ll let you know if I discover something more,
Elia

1 Like

I have already reached out to our internal team about extending the support of curobo to Isaac Sim 4.5.0.

Also I am not really sure how to reproduce the issue. I have Isaac Sim 4.0.0, 4.2.0 and 4.5.0 installed. For curobo, I am on branch isaac-3.2. I tried running curobo example with Isaac Sim 4.5.0 and it failed (because it is not supported yet). Then I switched back to Isaac Sim 4.2.0 and the example works fine.

This is the command I used to launch a curobo example:

omni_python examples/isaac_sim/collision_checker.py

@theo.bloesch In your log file, it suggests you to upgrade your pip

/home/theobloesch/.local/share/ov/pkg/isaac-sim-4.2.0/kit/python/bin/python3 -m pip install --upgrade pip

Have you tried that?