NVIDIA Parabricks v4.3.1-1run germline pipeline error: cudaGetDevice() failed in geting device ID

,

We have an error: “cudaGetDevice() failed in geting device ID. Status: forward compatibility was attempted on non supported HW, exiting.” when we run germline pipeline by NVIDIA Parabricks v4.3.1-1.
This is our code:

docker run --rm --gpus all --volume /home/sunjingchun:/workdir --volume /home/sunjingchun:/outputdir \
	--workdir /workdir \
	--env TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=268435456 \
	nvcr.io/nvidia/clara/clara-parabricks:4.3.1-1 \
	pbrun germline \
	--ref /workdir/1.nxpig/1.genome/Sus_scrofa.Sscrofa11.1.dna.toplevel.fa \
	--in-fq /workdir/1.nxpig/2.WGS_data/2.clean_data/B917_L2_377X77.R1.clean.fastq.gz /workdir/1.nxpig/2.WGS_data/2.clean_data/B917_L2_377X77.R2.clean.fastq.gz "@RG\tID:917\tLB:lib1\tPL:bar\tSM:917\tPU:917"\
	--out-bam /outputdir/1.nxpig/2.WGS_data/3.bam/917.rmdup.bam \
        --tmp-dir /workdir/1.nxpig \
        --num-cpu-threads-per-stage 64 \
        --bwa-cpu-thread-pool 64 \
	--out-variants /outputdir/1.nxpig/2.WGS_data/4.gvcf/917.rmdup.g.vcf \
        --run-partition \
        --read-from-tmp-dir \
        --gpusort \
        --gpuwrite \
	--fq2bamfast \
	--low-memory

This is nvidia-smi:
NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------±---------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:41:00.0 Off | Off |
| 30% 32C P8 30W / 450W | 11MiB / 24564MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
| 1 NVIDIA GeForce RTX 4090 Off | 00000000:A1:00.0 Off | Off |
| 30% 30C P8 20W / 450W | 11MiB / 24564MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+

±--------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 7133 G /usr/lib/xorg/Xorg 4MiB |
| 1 N/A N/A 7133 G /usr/lib/xorg/Xorg 4MiB

From documentation on forward compatibility:

Forward Compatibility is applicable only for systems with NVIDIA Data Center GPUs or select NGC Server Ready SKUs of RTX cards. It’s mainly intended to support applications built on newer CUDA Toolkits to run on systems installed with an older NVIDIA Linux GPU driver from different major release families.

This container is based off of CUDA Toolkit 12.3 (you can see this by doing echo $NV_CUDA_CUDART_VERSION from within the container). You have the driver corresponding to CUDA 12.2 installed. Forward compatibility libraries are not supported on GeForce GPUs. Since you have GeForce GPUs, you will need to update your driver to something >=12.3 for compatibility with the container.