Isaac Sim Docker container crashes from TAO Quickstart for centerpose with WSL2 Ubuntu 22.04 and Vulkan support through kisak-mesa ppa

Omniverse Isaac Sim
, , , ,
1

When trying out the tutorial notebook for centerpose synthetic data generation with

  • WSL2, Kernel 5.15.153.1-microsoft-standard-WSL2
  • Ubuntu 22.04
  • Cuda 12.6
  • Driver 561.09
  • conda env configured as shown in tutorial with python 3.10.14
    from Beginners - NVIDIA Docs under notebooks/tao_launcher_starter_kit/centerpose/ the docker container used to start isaac sim will crash constantly when executing
!nvidia-smi
!nvcc -V
!docker run --gpus device=0 -it \
    --entrypoint /bin/bash \
    --network host \
    -v $HOST_DATA_DIR:/tmpsrc \
    -e OMNI_USER='$omni-api-token' \
    -e OMNI_PASS=$OMNI_PASS \
    nvcr.io/nvidia/isaac-sim:4.1.0 \
    -c "apt-get update && apt-get install zenity libglib2.0-dev -y && bash isaac-sim.sh --no-window --allow-root --windowless=True --allow-root --/log/outputStreamLevel=fatal --/app/extensions/fastImporter/enabled=false --enable omni.replicator.object --/config/file=/tmpsrc/config.yaml"

with the following log from docker: (just the last few lines, full log is attached as a file)

2024-09-20 18:36:23 2024-09-20 16:36:23 [588ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osDistro = 'ubuntu'
2024-09-20 18:36:23 2024-09-20 16:36:23 [592ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osName = '22.04.3 LTS (Jammy Jellyfish)'
2024-09-20 18:36:23 2024-09-20 16:36:23 [596ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osVersion = '22.04.3'
2024-09-20 18:36:23 2024-09-20 16:36:23 [600ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  portableMode = '0'
2024-09-20 18:36:23 2024-09-20 16:36:23 [604ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  runEnvironment = 'Individual'
2024-09-20 18:36:23 2024-09-20 16:36:23 [609ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  runningInContainer = '1'
2024-09-20 18:36:23 2024-09-20 16:36:23 [613ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  systemInfo = ''
2024-09-20 18:36:23 2024-09-20 16:36:23 [617ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  telemetrySessionId = '16503640176256581015'
2024-09-20 18:36:23 2024-09-20 16:36:23 [622ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  terminatedByAbort = '0'
2024-09-20 18:36:23 2024-09-20 16:36:23 [626ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  userId = 'default'
2024-09-20 18:36:23 2024-09-20 16:36:23 [631ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  workingDirectory = '/isaac-sim'
2024-09-20 18:36:23 2024-09-20 16:36:23 [635ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Thread 1 backtrace follows:
2024-09-20 18:36:23 2024-09-20 16:36:23 [646ms] [Fatal] [carb.crashreporter-breakpad.plugin] 000: libc.so.6!__sigaction+0x50
2024-09-20 18:36:23 2024-09-20 16:36:23 [653ms] [Fatal] [carb.crashreporter-breakpad.plugin] 001: libimgui.so!ImGui::PushStyleColor(int, ImVec4 const&)+0x2a
2024-09-20 18:36:23 2024-09-20 16:36:23 [660ms] [Fatal] [carb.crashreporter-breakpad.plugin] 002: libcarb.imgui.plugin.so!_init+0x366f
2024-09-20 18:36:23 2024-09-20 16:36:23 [667ms] [Fatal] [carb.crashreporter-breakpad.plugin] 003: imgui.cpython-310-x86_64-linux-gnu.so!std::vector<std::string, std::allocator<std::string> >::~vector()+0x12154
2024-09-20 18:36:23 2024-09-20 16:36:23 [674ms] [Fatal] [carb.crashreporter-breakpad.plugin] 004: imgui.cpython-310-x86_64-linux-gnu.so!std::vector<std::string, std::allocator<std::string> >::~vector()+0x159e5
2024-09-20 18:36:23 2024-09-20 16:36:23 [683ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: libpython3.10.so.1.0!PyBuffer_ToContiguous+0x9a7
2024-09-20 18:36:23 2024-09-20 16:36:23 [693ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: libpython3.10.so.1.0!_PyObject_MakeTpCall+0x9a
2024-09-20 18:36:23 2024-09-20 16:36:23 [703ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: libpython3.10.so.1.0!PyCell_Set+0x8b5
2024-09-20 18:36:23 2024-09-20 16:36:23 [713ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: libpython3.10.so.1.0!_init+0xbaed
2024-09-20 18:36:23 2024-09-20 16:36:23 [723ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: libpython3.10.so.1.0!_PyEval_EvalFrameDefault+0x6e78
2024-09-20 18:36:23 2024-09-20 16:36:23 [733ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: libpython3.10.so.1.0!PyEval_EvalFrameEx+0xb24
2024-09-20 18:36:23 2024-09-20 16:36:23 [742ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: libpython3.10.so.1.0!_init+0xbaed
2024-09-20 18:36:23 2024-09-20 16:36:23 [751ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: libpython3.10.so.1.0!_PyEval_EvalFrameDefault+0x38bc
2024-09-20 18:36:23 2024-09-20 16:36:23 [760ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: libpython3.10.so.1.0!PyEval_EvalFrameEx+0xb24
2024-09-20 18:36:23 2024-09-20 16:36:23 [769ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: libpython3.10.so.1.0!PyVectorcall_Call+0x54
2024-09-20 18:36:23 2024-09-20 16:36:23 [778ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: libpython3.10.so.1.0!_PyEval_EvalFrameDefault+0x5745
2024-09-20 18:36:23 2024-09-20 16:36:23 [787ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: libpython3.10.so.1.0!PyEval_EvalFrameEx+0xb24
2024-09-20 18:36:23 2024-09-20 16:36:23 [796ms] [Fatal] [carb.crashreporter-breakpad.plugin] 017: libpython3.10.so.1.0!_init+0xbaed
2024-09-20 18:36:23 2024-09-20 16:36:23 [806ms] [Fatal] [carb.crashreporter-breakpad.plugin] 018: libpython3.10.so.1.0!_PyEval_EvalFrameDefault+0x38bc
2024-09-20 18:36:23 2024-09-20 16:36:23 [816ms] [Fatal] [carb.crashreporter-breakpad.plugin] 019: libpython3.10.so.1.0!PyEval_EvalFrameEx+0xb24
2024-09-20 18:36:23 2024-09-20 16:36:23 [826ms] [Fatal] [carb.crashreporter-breakpad.plugin] 020: libpython3.10.so.1.0!_init+0xbaed
2024-09-20 18:36:23 2024-09-20 16:36:23 [836ms] [Fatal] [carb.crashreporter-breakpad.plugin] 021: libpython3.10.so.1.0!_PyEval_EvalFrameDefault+0x38bc
2024-09-20 18:36:23 2024-09-20 16:36:23 [845ms] [Fatal] [carb.crashreporter-breakpad.plugin] 022: libpython3.10.so.1.0!PyEval_EvalFrameEx+0xb24
2024-09-20 18:36:23 2024-09-20 16:36:23 [855ms] [Fatal] [carb.crashreporter-breakpad.plugin] 023: libpython3.10.so.1.0!_init+0xbaed
2024-09-20 18:36:23 2024-09-20 16:36:23 [864ms] [Fatal] [carb.crashreporter-breakpad.plugin] 024: libpython3.10.so.1.0!_PyEval_EvalFrameDefault+0x6e78
2024-09-20 18:36:23 2024-09-20 16:36:23 [873ms] [Fatal] [carb.crashreporter-breakpad.plugin] 025: libpython3.10.so.1.0!PyEval_EvalCode+0x504
2024-09-20 18:36:23 2024-09-20 16:36:23 [882ms] [Fatal] [carb.crashreporter-breakpad.plugin] 026: libcarb.scripting-python.plugin.so!carbOnPluginShutdown+0x98c
2024-09-20 18:36:23 2024-09-20 16:36:23 [892ms] [Fatal] [carb.crashreporter-breakpad.plugin] 027: libcarb.scripting-python.plugin.so!carbOnPluginShutdown+0x117f
2024-09-20 18:36:23 2024-09-20 16:36:23 [902ms] [Fatal] [carb.crashreporter-breakpad.plugin] 028: libomni.ext.plugin.so!std::string& std::vector<std::string, std::allocator<std::string> >::emplace_back<char const*&>(char const*&)+0xd158
2024-09-20 18:36:23 2024-09-20 16:36:23 [911ms] [Fatal] [carb.crashreporter-breakpad.plugin] 029: libomni.ext.plugin.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int (*)(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, ...)+0x1022a
2024-09-20 18:36:23 2024-09-20 16:36:23 [921ms] [Fatal] [carb.crashreporter-breakpad.plugin] 030: libomni.ext.plugin.so!carbOnPluginPreStartup+0x1223
2024-09-20 18:36:23 2024-09-20 16:36:23 [930ms] [Fatal] [carb.crashreporter-breakpad.plugin] 031: libomni.ext.plugin.so!carbOnPluginPreStartup+0xd6cc
2024-09-20 18:36:23 2024-09-20 16:36:23 [940ms] [Fatal] [carb.crashreporter-breakpad.plugin] 032: libomni.ext.plugin.so!carbOnPluginPreStartup+0xe733
2024-09-20 18:36:23 2024-09-20 16:36:23 [949ms] [Fatal] [carb.crashreporter-breakpad.plugin] 033: libomni.kit.app.plugin.so!std::string::compare(std::string const&) const+0x96f3
2024-09-20 18:36:23 2024-09-20 16:36:23 [958ms] [Fatal] [carb.crashreporter-breakpad.plugin] 034: libomni.kit.app.plugin.so!std::string::compare(std::string const&) const+0xa0cf
2024-09-20 18:36:23 2024-09-20 16:36:23 [967ms] [Fatal] [carb.crashreporter-breakpad.plugin] 035: libomni.kit.app.plugin.so!_init+0x64a7
2024-09-20 18:36:23 2024-09-20 16:36:23 [971ms] [Fatal] [carb.crashreporter-breakpad.plugin] 036: kit!_init+0x6eb
2024-09-20 18:36:23 2024-09-20 16:36:23 [980ms] [Fatal] [carb.crashreporter-breakpad.plugin] 037: libc.so.6!__libc_init_first+0x90
2024-09-20 18:36:23 2024-09-20 16:36:23 [990ms] [Fatal] [carb.crashreporter-breakpad.plugin] 038: libc.so.6!__libc_start_main+0x80
2024-09-20 18:36:23 2024-09-20 16:36:23 [995ms] [Fatal] [carb.crashreporter-breakpad.plugin] 039: kit!_start+0x29

the error occurs just after isaac sim starting:

[...]
2024-09-20 18:36:21 [23.132s] [ext: omni.isaac.examples-2.2.1] startup
2024-09-20 18:36:22 [23.547s] [ext: omni.graph.window.generic-1.24.0] startup
2024-09-20 18:36:22 [23.609s] [ext: omni.kit.converter.cad-201.1.0-rc.4] startup
2024-09-20 18:36:22 [23.611s] [ext: omni.kit.window.stats-0.1.6] startup
2024-09-20 18:36:22 [23.633s] [ext: omni.isaac.app.setup-0.13.2] startup
2024-09-20 18:36:22 [23.647s] Isaac Sim Version: 4.1.0-rc.7
2024-09-20 18:36:22 2024-09-20 16:36:22 [0ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred.  If a debugger should be attached, please set the '/crashreporter/debuggerAttachTimeoutMs' setting to a timeout in milliseconds.  This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
[...]

Can you give me hints on how to fix this issue? I dont think WSL2 is the issue here, as Vulkan does work, see for example vkcube:

vkcube
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
Selected GPU 0: Microsoft Direct3D12 (NVIDIA GeForce RTX 4090 Laptop GPU), type: 2

and the cube is displayed correctly. Vulkaninfo --summary shows:

vulkaninfo --summary
WARNING: [Loader Message] Code 0 : terminator_CreateInstance: Failed to CreateInstance in ICD 6.  Skipping ICD.
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
==========
VULKANINFO
==========

Vulkan Instance Version: 1.3.204


Instance Extensions: count = 21
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 4
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6

Instance Layers: count = 3
--------------------------
VK_LAYER_INTEL_nullhw       INTEL NULL HW                1.1.73   version 1
VK_LAYER_MESA_device_select Linux device selection layer 1.3.211  version 1
VK_LAYER_MESA_overlay       Mesa Overlay layer           1.3.211  version 1

Devices:
========
GPU0:
        apiVersion         = 4202785 (1.2.289)
        driverVersion      = 100671491 (0x6002003)
        vendorID           = 0x10de
        deviceID           = 0x2757
        deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
        deviceName         = Microsoft Direct3D12 (NVIDIA GeForce RTX 4090 Laptop GPU)
        driverID           = UNKNOWN_VkDriverId_value23
        driverName         = Dozen
        driverInfo         = Mesa 24.2.3 - kisak-mesa PPA
        conformanceVersion = 0.0.0.0
        deviceUUID         = 55be0f0c-5858-347d-d5e3-9a6ce003a404
        driverUUID         = 6ee2028a-2063-39d1-a78b-ff50217faeb8
GPU1:
        apiVersion         = 4206881 (1.3.289)
        driverVersion      = 1 (0x0001)
        vendorID           = 0x10005
        deviceID           = 0x0000
        deviceType         = PHYSICAL_DEVICE_TYPE_CPU
        deviceName         = llvmpipe (LLVM 15.0.7, 256 bits)
        driverID           = DRIVER_ID_MESA_LLVMPIPE
        driverName         = llvmpipe
        driverInfo         = Mesa 24.2.3 - kisak-mesa PPA (LLVM 15.0.7)
        conformanceVersion = 1.3.1.1
        deviceUUID         = 6d657361-3234-2e32-2e33-202d206b6900
        driverUUID         = 6c6c766d-7069-7065-5555-494400000000
GPU2:
        apiVersion         = 4206881 (1.3.289)
        driverVersion      = 1 (0x0001)
        vendorID           = 0x10005
        deviceID           = 0x0000
        deviceType         = PHYSICAL_DEVICE_TYPE_CPU
        deviceName         = llvmpipe (LLVM 15.0.7, 256 bits)
        driverID           = DRIVER_ID_MESA_LLVMPIPE
        driverName         = llvmpipe
        driverInfo         = Mesa 24.2.3 (LLVM 15.0.7)
        conformanceVersion = 1.3.1.1
        deviceUUID         = 6d657361-3234-2e32-2e33-000000000000
        driverUUID         = 6c6c766d-7069-7065-5555-494400000000

Best regards

2

I’m curious about your use case for the Isaac Sim Docker container. Could you share more about why you chose to use the Isaac Sim Docker container instead of the TAO container?

3

Hey Vick,

I got inspired heavily by the TAO Quickstarter Notebooks, especially centerpose_synthetic which does the same and calls ISAAC SIM for SDG first. Secondly, the TAO Toolkit is executed next to use the synthetic training data for fine-tuning/training the centerpose model.

Is this not the optimal workflow?

Best regards

5

Please follow this guide to verify that CUDA applications work in your system. Also, refer to the section on known limitations. For example, for Docker 19.03, only --gpus all is supported.

1 Like
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.