Nvidia-container-toolkit not finding items on PATH

I have been working through setup of Dell XPS15 (RTX 3050) W11 WSL2-Ubuntu22.04LTS with CUDA 12.2 to run GPU workloads in containers.

Steps to reproduce the behavior:

  1. Update Windows, Dell, NVIDIA Graphics driver
  2. Allow access to all users to the GPU performance counters in NVIDIA Control Panel Developer Mode
  3. Update WSL, Configure default version to WSL2
  4. Follow many steps (will add details in follow-up comment)

install ubuntu 22.04 in WSL2
install cuda 12.2 in WSL2
append relevant directories to PATH and LD_LIBRARY_PATH
install nvidia-cuda-toolkit
install docker
install nvidia-container-toolkit
generate the “Container Device Interface” config json
run vespa container and install vespa-onnxruntime-cuda
try to deploy an app

Expected behavior
The app should read LD_LIBRARY_PATH, load libcublasLt.so.12, converge and deploy.

Actual behaviour
Vespa app in container could not load library libcublasLt.so.12.
nvidia-container-toolkit.log shows a searching and not finding things.

I followed steps to delete something in system32 and make the symlinks. search for windows subsystem for linux - WSL libcuda is not a symbolic link - Super User to see details.

and also added to my ~/.profile

export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

Environment :
OS Name: Microsoft Windows 11 Pro
Version: 10.0.22621 Build 22621
System Model: XPS 15 9520
Processor: 12th Gen Intel(R) Core™ i7-12700H, 2300 Mhz, 14 Core(s), 20 Logical Processor(s)
BIOS Version/Date: Dell Inc. 1.13.1, 4/17/2023
SMBIOS Version: 3.4
Total Physical Memory: 31.7 GB

NVIDIA Studio Driver: 536.67
NVIDIA-SMI: 535.86.01

WSL Linux kernel:
WSL Distro Description: Ubuntu 22.04.2 LTS

“CUDA Version”: 12.2
nvcc: Cuda compilation tools, release 12.2, V12.2.91 Build cuda_12.2.r12.2/compiler.32965470_0

Vespa version

I am inspecting the (see attached) /var/log/nvidia-container-toolkit.log and I would like assistance in interpreting all the missing libraries in the log. What is normal and abnormal about what it contains? How can I get it to find these libraries?
nvidia-container-toolkit.log (20 KB)

Here is a detailed step-by-step document on my build process
system_buildC.txt (4.3 KB)