CUDA error 999; Failed to query CUDA device count; CUDA deviceOrdinal is invalid

[ShawnZhou997]

2023-11-30 02:13:54 [Error] [carb.cudainterop.plugin] CUDA error 999: cudaErrorUnknown - unknown error)
2023-11-30 02:13:54 [Error] [carb.cudainterop.plugin] Failed to query CUDA device count.
2023-11-30 02:13:54 [Error] [carb.cudainterop.plugin] Could not query CUDA device.

Hi, my Isaac Sim seems to meet some new issues after my CUDA installation.
Now, it is dark all the time and the console shows a lot errors.
Have you ever met with the same issue before? Cam I fix it with not unistalling the CUDA?

[ShawnZhou997]

It happened after I installed environments for Deep Learing such as GPU Driver, CUDA, cuDNN and so on.

[ShawnZhou997]

I solved it by annotating the lines with regard to CUDA in .bashrc file and rebooting my Ubuntu.
Then I installed another version of CUDA and export the $PATH to it
Not pretty sure whether i solved the issue permanently or temporarily.

[ShawnZhou997]

After using Isaac Sim for a week, the CUDA error showed up again.
Then I rebooted my PC for a makeshift, it works now but not sure about how long can it work.
I’ve updated my NVIDIA Driver and its CUDA version to 525 and 11.7 since last broke.
What else can I do now?

[rthaker]

Hi @ShawnZhou997 - Let us know if you are still issues after updating to latest Isaac Sim release.

[ShawnZhou997]

@rthaker Hi, it still occured in the neweast release version.

And after reboot my PC, it’ll be fixed for a short time.

[ShawnZhou997]

Hi, the issue above happens more frequently than before. Should I reinstall the cuda version?

[rthaker]

Hi @ShawnZhou997 - Yes, please try that and let us know if you still see similar issue.