Isaac sim unable to load after restart [ VkResult: ERROR_OUT_OF_DEVICE_MEMORY]

zheng.zhang314 · January 12, 2024, 6:54am

Hi,

I used Isaac sim to do some simulation on my Ubuntu laptop for a while and it worked fine. But after once I restarted my computer. I was not able to launch Isaac Sim in Omniverse. I tried to launch Isaac sim using provided isaac_sim.sh and it showed:

[0.357s] [ext: omni.kit.renderer.init-0.0.0] startup
2024-01-12 06:32:13 [1,539ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: Intel(R) Graphics (ADL GT2)
2024-01-12 06:32:13 [1,539ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: Intel(R) Graphics (ADL GT2)

|---------------------------------------------------------------------------------------------|
| Driver Version: 545.29.06     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 3070 Ti Lap.. | Yes: 0 |     | 8438    MB | 10de      | 0          |
|     |                                  |        |     |            | 24e0      | a8ffd185.. |
|     |                                  |        |     |            | 1         |            |
|---------------------------------------------------------------------------------------------|
| 1   | Intel(R) Graphics (ADL GT2)      |        |     | 48018   MB | 8086      | 0          |
|     |                                  |        |     |            | 46a6      | 3fc767c9.. |
|     |                                  |        |     |            | 0         |            |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 6.5.0-14-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101004 (1.21.1.4)
| Processor: 12th Gen Intel(R) Core(TM) i7-12700H | Cores: 14 | Logical: 20
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 64024 | Free Memory: 59934
| Total Page/Swap (MB): 9999 | Free Page/Swap: 9999
|---------------------------------------------------------------------------------------------|
[1.866s] [ext: omni.kit.pipapi-0.0.0] startup
[1.867s] [ext: omni.kit.pip_archive-0.0.0] startup
[1.867s] [ext: omni.kit.telemetry-0.5.0] startup
[1.880s] [ext: omni.kit.loop-default-0.2.0] startup
[1.880s] [ext: omni.kit.test-0.0.0] startup
[1.921s] [ext: omni.mtlx-0.1.0] startup
[1.921s] [ext: omni.usd.config-1.0.3] startup
[1.923s] [ext: omni.gpucompute.plugins-0.0.0] startup
[1.924s] [ext: omni.usd.libs-1.0.0] startup
[2.018s] [ext: omni.pip.compute-1.2.0] startup
[2.018s] [ext: omni.pip.cloud-1.0.1] startup
[2.022s] [ext: omni.isaac.core_archive-2.2.1] startup
[2.022s] [ext: omni.pip.torch-2_0_1-2.0.2] startup
[2.023s] [ext: omni.usd.schema.anim-0.0.0] startup
[2.169s] [ext: omni.usd.schema.audio-0.0.0] startup
[2.172s] [ext: omni.appwindow-1.1.5] startup
[2.229s] [ext: omni.kit.renderer.core-0.0.0] startup
2024-01-12 06:32:13 [2,287ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2024-01-12 06:32:13 [2,287ms] [Error] [carb.graphics-vulkan.plugin] vkCreateSwapchainKHR failed.
2024-01-12 06:32:13 [2,287ms] [Error] [carb.graphics-vulkan.plugin] Failed to create a swapchain.
2024-01-12 06:32:13 [2,287ms] [Error] [omni.kit.renderer.plugin] createSwapchain failed. Width: 1440, height: 900, Format: 9
2024-01-12 06:32:13 [2,287ms] [Error] [omni.kit.renderer.plugin] Failed to initialize graphics environment.
[2.314s] [ext: omni.kit.renderer.capture-0.0.0] startup
[2.317s] [ext: omni.kit.renderer.imgui-0.0.0] startup
2024-01-12 06:32:14 [0ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred.  If a debugger should be attached, please set the '/crashreporter/debuggerAttachTimeoutMs' setting to a timeout in milliseconds.  This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
2024-01-12 06:32:14 [35ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash] compressing 3 files into '/home/zheng/.local/share/ov/data/Kit/Isaac-Sim/2023.1/ab03b37a-c678-409b-4b8347a9-cadabfca.dmp.zip' for upload.
'crashreport.gui' exited with code 1272024-01-12 06:32:14 [180ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] Wrote dump file '/home/zheng/.local/share/ov/data/Kit/Isaac-Sim/2023.1/ab03b37a-c678-409b-4b8347a9-cadabfca.dmp.zip'
2024-01-12 06:32:14 [185ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash] Uploading minidump: file:'/home/zheng/.local/share/ov/data/Kit/Isaac-Sim/2023.1/ab03b37a-c678-409b-4b8347a9-cadabfca.dmp.zip' svr:'https://services.nvidia.com/submit'
2024-01-12 06:32:14 [190ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash] Crash metadata for upload:
2024-01-12 06:32:14 [195ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   CarbSdkVersion = '156.0+release156.tc9247.e85106ac'
2024-01-12 06:32:14 [201ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   CrashTime = 'Fri Jan 12 06:32:14 2024 GMT'
2024-01-12 06:32:14 [206ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   DriverShaderCacheWrapper = 'disabled'
2024-01-12 06:32:14 [211ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   DumpId = 'ab03b37a-c678-409b-4b8347a9-cadabfca'
2024-01-12 06:32:14 [216ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   ProductName = 'OmniverseKit'
2024-01-12 06:32:14 [221ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   PythonTracebackStatus = ''py-spy' successfully wrote info to '/home/zheng/.local/share/ov/data/Kit/Isaac-Sim/2023.1/ab03b37a-c678-409b-4b8347a9-cadabfca.py.txt''
2024-01-12 06:32:14 [226ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   RetryCount = '0'
2024-01-12 06:32:14 [231ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   StartupTime = '1705041131'
2024-01-12 06:32:14 [237ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   UploadSuccessful = '0'
2024-01-12 06:32:14 [242ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   UptimeSeconds = '3'
2024-01-12 06:32:14 [247ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   UserStory = ''
2024-01-12 06:32:14 [252ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   UserStoryStatus = ''crashreport.gui' exited with code 127'
2024-01-12 06:32:14 [258ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   Version = '105.1+release.130815.04872a84.tc'
2024-01-12 06:32:14 [263ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   aftermath_enabled = '1'
2024-01-12 06:32:14 [268ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   aftermath_flags = '3'
2024-01-12 06:32:14 [273ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   aftermath_status = 'auto-enabled'
2024-01-12 06:32:14 [279ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   aftermath_version = '2022.2'
2024-01-12 06:32:14 [284ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   appName = 'Isaac-Sim'
2024-01-12 06:32:14 [289ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   appState = 'startup'
2024-01-12 06:32:14 [294ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   appVersion = '2023.1.0'
2024-01-12 06:32:14 [300ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   autoloadExts = 'omni.isaac.quadruped-1.3.2'
2024-01-12 06:32:14 [305ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildBranch = 'release'
2024-01-12 06:32:14 [310ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildCi = 'tc'
2024-01-12 06:32:14 [315ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildHash = '04872a84'
2024-01-12 06:32:14 [320ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildId = '18000955'
2024-01-12 06:32:14 [325ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildMajor = '105'
2024-01-12 06:32:14 [331ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildMinor = '1'
2024-01-12 06:32:14 [336ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildMr = '0'
2024-01-12 06:32:14 [341ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildNumber = '130815'
2024-01-12 06:32:14 [347ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildPatch = '0'
2024-01-12 06:32:14 [352ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   buildVersion = '105.1.0'
2024-01-12 06:32:14 [357ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   commandLine = './kit/kit ./apps/omni.isaac.sim.kit --ext-folder ./apps'
2024-01-12 06:32:14 [362ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   cpuId = 'Intel64 Family 6 Model 154 Stepping 3'
2024-01-12 06:32:14 [368ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   cpuName = '12th Gen Intel(R) Core(TM) i7-12700H'
2024-01-12 06:32:14 [373ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   cpuVendor = 'GenuineIntel'
2024-01-12 06:32:14 [379ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   desktopOrigin = '(0, 0)'
2024-01-12 06:32:14 [384ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   desktopSize = '2560x1600'
2024-01-12 06:32:14 [389ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   displayCount = '1'
2024-01-12 06:32:14 [395ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   displayRes_0 = '2560x1600x32bit@165Hz'
2024-01-12 06:32:14 [400ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   environmentName = 'Individual'
2024-01-12 06:32:14 [405ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   experience = 'Isaac Sim'
2024-01-12 06:32:14 [411ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   externalBuild = '1'
2024-01-12 06:32:14 [417ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   gpuDriver_0 = '545.29'
2024-01-12 06:32:14 [422ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   gpuDriver_1 = '0x5c00004'
2024-01-12 06:32:14 [427ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   gpuVRAM_0 = '8847884288'
2024-01-12 06:32:14 [433ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   gpuVRAM_1 = '50350728192'
2024-01-12 06:32:14 [438ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   gpu_0 = 'NVIDIA GeForce RTX 3070 Ti Laptop GPU'
2024-01-12 06:32:14 [443ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   gpu_1 = 'Intel(R) Graphics (ADL GT2)'
2024-01-12 06:32:14 [449ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   kitRendererDriverVersion = '545.29'
2024-01-12 06:32:14 [454ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   memoryStats = '(avail/total) RAM: 57.2758/62.5237GB, Swap: 9.76562/9.76562GB, VM: 1.71799e+10/1.71799e+10GB'
2024-01-12 06:32:14 [459ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   nvidia-smi = 'Fri Jan 12 14:32:13 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3070 ...    Off | 00000000:01:00.0 Off |                  N/A |
| N/A   37C    P3             752W /  50W |      1MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
'
2024-01-12 06:32:14 [465ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   osDistro = 'ubuntu'
2024-01-12 06:32:14 [470ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   osName = '22.04.3 LTS (Jammy Jellyfish)'
2024-01-12 06:32:14 [476ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   osVersion = '22.04.3'
2024-01-12 06:32:14 [482ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   portableMode = '0'
2024-01-12 06:32:14 [487ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   primaryDisplayRes = '2560x1600x32bit@165Hz'
2024-01-12 06:32:14 [493ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   runEnvironment = 'Individual'
2024-01-12 06:32:14 [499ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   shaderdb_debugSymbols = '0'
2024-01-12 06:32:14 [505ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   shaderdb_dumpIncludeOverrides = '0'
2024-01-12 06:32:14 [510ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   shaderdb_dumpIntermediates = '0'
2024-01-12 06:32:14 [516ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   shaderdb_obfuscateCode = '1'
2024-01-12 06:32:14 [521ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   shaderdb_optimizationLevel = '1'
2024-01-12 06:32:14 [526ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]   systemInfo = '

It worn ERROR_OUT_OF_DEVICE_MEMORY, but nothing was running. I tried to kill all processes used nvidia, but issue still remains.

What I did before I restarted computer:
Installed Isaac Orbit and ran some demo using it.
Installed SimpleScreenRecorder and recorded the screen while running Isaac sim demos. Meanwhile It showed some out of memory warnings, but Isaac sim can still run or relaunch from the omniverse.
Then I restarted the computer and this issue appeared.

May I know how to fix this?

Best regards!

zheng.zhang314 · January 12, 2024, 7:04am

kit_20240112_143211.log (574.8 KB)
a most recent log

zheng.zhang314 · January 12, 2024, 7:06am

The log for the last time I successfully run isaac sim
kit_20240112_114853.log (1.3 MB)

Do need your help, thanks a lot!

Sheikh_Dawood · January 12, 2024, 3:33pm

Hi. Thanks for including your log files.
It looks like the 8GB VRAM on the 3070 Ti is the bare minimum to run Isaac Sim. It still shows the memory error even when Isaac Sim runs.

Do you see the same error when running USD Composer or Code?

Try running Isaac Sim right after a reboot. You could also try disabling some extensions that run with Isaac Sim.

Another thing to try is to use the 525 drivers instead. We have not fully tested with the new 545 drivers.

zheng.zhang314 · January 15, 2024, 1:35am

Thanks for your instant replay.
I tried to reboot and ran:
./python.sh standalone_examples/api/omni.isaac.franka/follow_target_with_rmpflow.py
but it shows the same error as before.

However when I tried to run
./jupyter_notebook.sh standalone_examples/notebooks/hello_world.ipynb
It seems to work-ish with some errors. The cube pose is printed in the cell, but GUI is not popped out.

Error shows in the first cell:
2024-01-15 01:10:58 [10,374ms] [Error] [omni.kit.app.plugin] Can’t delay app ready event, it was already sent. Requester name: omni.usd
Error shows in the last cell
2023-12-22 03:43:02 [497,153ms] [Error] [carb.graphics-vulkan.plugin] pipelineLayouts: 2 or descriptorSetLayouts: 9 is leaking memory. Missing call to destroyResourceBindingSignature()
log:
kit_20240115_091048.log (782.6 KB)

Any comments on that?

I am trying to dowload the usd composer and reinstall the drivers and see how it works. Can you provide any guidance to fully uninstall the old driver and reinstall the new one? And how to disable the extensions run with isaac sim in command line without isaac sim to be opened?

Best

Simplychenable · January 15, 2024, 1:52am

not sure if this will help?

https://docs.omniverse.nvidia.com/dev-guide/latest/linux-troubleshooting.html#q1-how-to-install-a-driver

zheng.zhang314 · January 15, 2024, 4:17am

Thanks for your reply. That do helps.

However, when I try to reinstall the driver, it shows error: You appear to be running an X server; please exit X before installing, which was not shown last time I installed. Don’t know if this could be the problem cause Isaac unable to launch. I am trying to fix that to reinstall the driver.

Sheikh_Dawood · January 22, 2024, 6:33pm

Are you able to reinstall the drivers with the linux trouble shooting link above?
Run this command to get into an environment without X server running:
sudo systemctl isolate multi-user.target

Topic		Replies	Views
Isaac SIM in omniverse not launching Isaac Sim setup	62	8932	August 31, 2022
Failed to install isaac sim 2021.2.0 version General Discussion isaacsim	14	4964	September 15, 2022
Error occured during the installation of IsaaC SIM Isaac Sim setup	10	2845	April 5, 2024
Isaac sim 2023.1.1 with omniverse 1.9.11 on ubuntu 22.04 failed to launch Isaac Sim	1	129	July 16, 2024
ISAAC SIM 2021.1 Launch from Omniverse fail, TypeError: t is not iterable Isaac Sim installation	21	1897	April 5, 2024
Isaac Sim (2021.1.1) Wont Launch (Omniverse 1.3.4) Isaac Sim aws , setup	23	4155	April 5, 2024
I started isaac-sim and got segmentation fault Isaac Sim ubuntu , isaac-sim-v4-2-0	2	58	November 3, 2024
Issues abot Installing Isaac Sim on Ubuntu20.04 on laptop Isaac Sim isaacsim	6	1029	April 5, 2024
Isaac Sim 2022.2.1 crashes after opening USD file Isaac Sim	19	1912	September 27, 2023
Isaac Sim 2022.2.1 stuck opening Hello World or any .USD saved file Isaac Sim	7	671	October 12, 2023

Isaac sim unable to load after restart [ VkResult: ERROR_OUT_OF_DEVICE_MEMORY]

Related topics