Running OptiX on WSL (2026 version)

Gaming and Visualization Technologies Visualization OptiX
1

Hi all,

While we’re still working (slowly) on official OptiX support in the WSL drivers, Kyle has refreshed the instructions for manually getting OptiX to work in WSL. We hope this helps people, but please remember these are unofficial instructions and come with no support and no guarantee they’ll even work, so your mileage may vary. Make backups and be careful!

Running OptiX on WSL

[!CAUTION]
This is an experimental, unsupported workaround. It combines Linux user-mode driver libraries with the Windows NVIDIA driver used by WSL. An incompatible combination can cause OptiX errors, WSL instability, or a Windows display-driver crash. Use native Windows or native Linux for production workloads.

These instructions apply to OptiX 7 and later. These instructions do not apply to OptiX 6 or earlier, since those versions used different runtime components and had different compatibility requirements, and they are no longer supported.

Install a CUDA Toolkit for WSL

Follow the canonical methods for installing WSL, a Windows display driver, and CUDA Toolkit within WSL.

  • As mentioned in the docs, do not install a Linux display driver in WSL, as this will break the mapping of libcuda. There are CUDA Toolkit downloads specifically for WSL that will not attempt to install the driver.
  • If using NVIDIA’s regular Ubuntu repository, install only a versioned cuda-toolkit-X-Y package. Do not install the cuda, cuda-X-Y, or cuda-drivers metapackages under WSL, because they attempt to install the driver. If the driver component is selected during installation, make sure to deselect it.
  • Select a CUDA Toolkit version supported by the installed Windows driver, as listed in the CUDA driver compatibility tables.
  • You will likely need to install build-essential for the relevant C/C++ compiler
  • A good checkpoint here is to build and run deviceQuery from the CUDA Samples.

Obtain OptiX/RTCore/GPU compiler libraries for Linux

Check the minimum driver required by the selected OptiX SDK release. Extract the required libraries from a Linux display driver without installing it (you can unpack the driver from a runfile in WSL using ./<driver>.run -x). Use a Linux driver from the same driver branch as the installed Windows driver and as close to its version as possible.

  • Copy libnvoptix.so.<linux_driver_version>, libnvidia-rtcore.so.<linux_driver_version>, libnvidia-gpucomp.so.<linux_driver_version>, and nvoptix.bin into the WSL driver store.
    • For a persistent installation using Windows File Explorer, use C:/Windows/System32/lxss/lib as the destination. This is a Windows-managed driver directory and requires Administrator access. After copying here, you will need to quit all WSL sessions and run wsl --shutdown from PowerShell before starting a new WSL session to refresh the projected driver directory.
    • For a temporary test from within WSL, use /usr/lib/wsl/lib as the destination. Files copied there may be restored to their original state when WSL shuts down.
  • Rename libnvoptix.so.<linux_driver_version> to libnvoptix.so.1, overwriting the existing file only after backing it up.

Notes

  • Before modifying either driver-store location, back up every file that will be replaced. For example, there might already exist a libnvoptix.so.1 (with a libnvoptix_loader.so.1 symlink) and a libnvidia-gpucomp.so (with a libnvidia-gpucomp.so.<windows_driver_version> symlink).
  • Whenever the Windows display driver is updated, remove the copied Linux files and repeat this procedure with a compatible Linux driver runfile.

Building the OptiX SDK samples

Download and build the OptiX SDK samples for Linux.

  • As of writing this, the current version is OptiX SDK 9.1, which requires CMake 3.20.
  • You’ll need to install OpenGL, along with other necessary packages (libxi-dev, libxcursor-dev, libxrandr-dev, libxinerama-dev)
  • You may need to add /usr/local/cuda/bin to your PATH to access NVCC. Do not add /usr/local/cuda/lib64/stubs to the runtime LD_LIBRARY_PATH, because its libcuda stub must not override the WSL driver library.

Running an OptiX SDK sample

  • You may need to add /usr/lib/wsl/lib to your LD_LIBRARY_PATH to ensure your copied files are loaded. Also add the location of libsutil to LD_LIBRARY_PATH (usually <sdk_location>/SDK/build/lib).
  • You may verify loading paths using strace -f -e trace=file -o trace.log <application> and searching the output for the library names. LD_DEBUG=libs is another useful loader diagnostic.
    • For example, with the 595.71.05 Linux driver, you should see these libraries being loaded:
/usr/lib/wsl/lib/libnvoptix.so.1
/usr/lib/wsl/lib/libnvidia-rtcore.so.595.71.05
/usr/lib/wsl/lib/libnvidia-gpucomp.so.595.71.05
  • Run a denoiser sample if your application uses the OptiX denoiser, and verify that it can load nvoptix.bin.
  • OpenGL-CUDA interop is not supported under WSL. Run SDK samples with the --no-gl-interop option when available.

Rollback procedure

  • For files copied temporarily into /usr/lib/wsl/lib, quit all WSL sessions and run wsl --shutdown, then confirm that the original projected files have been restored.
  • For files copied persistently into C:/Windows/System32/lxss/lib, run wsl --shutdown, remove only the files that were copied from the Linux driver, restore any overwritten files from the backup, and reboot Windows.
  • If Windows is freezing or the display driver is crashing, boot into Windows Safe Mode before removing the copied files. If instability continues after rollback, keep WSL shut down and clean-install the Windows NVIDIA display driver.
  • After rollback, confirm that nvidia-smi and the CUDA Samples deviceQuery test work in WSL.

Common issues

  • If you see OPTIX_ERROR_LIBRARY_NOT_FOUND, verify that the application can find the copied libnvoptix.so.1 in the loader path.
  • If you see OPTIX_ERROR_ENTRY_SYMBOL_NOT_FOUND, verify that your application is loading the libnvoptix copied from the Linux package and not the shim library installed by the Windows driver, and check that the OptiX SDK and driver versions are compatible.
  • If you see Error initializing RTX library, verify that you have copied and are loading libnvidia-rtcore.
  • If you see OPTIX_ERROR_INTERNAL_COMPILER_ERROR, verify you are loading the copied libnvidia-gpucomp and confirm that it came from the same runfile as the other copied libraries. Also check the SDK/driver compatibility and the application input.
  • If you see minor Version newer than tool or minor NvvmIRVersion newer than tool, either your CUDA Toolkit version is too new for the driver, or the copied libnvoptix, libnvidia-rtcore, and libnvidia-gpucomp are not all from the same Linux driver package.
  • If you see “OS call failed or operation not supported on this OS” for an CUDA + OpenGL function, you may need to run OptiX SDK samples with the --no-gl-interop option.
  • If you see DENOISER: Unable to load denoiser weights, verify that the matching nvoptix.bin was copied and is being loaded.
  • If the application behaves differently under sudo, check for duplicate libnvoptix.so.1 files and different library search paths.
  • If you get a segmentation fault during library loading, try restarting either WSL or Windows and verifying that the libraries/symlinks are still correct in the driver store.
1 Like