Parabricks 4.2.0-1 haplotypecaller error - cudaSafeCall() failed - out of memory

[avenkatraman]

Hi

I am running haplotypecaller nvcr.io/nvidia/clara/clara-parabricks:4.2.0-1 on AWS g4dn.12xlarge and I get this error message below. Interestingly, the job exits with Exit Code 0

[PB Info 2023-Oct-17 02:52:14] ProgressMeter -  Current-Locus   Elapsed-Minutes Regions-Processed       Regions/Minute
[PB Debug 2023-Oct-17 02:52:15][src/assembleRegions_GPU.cu:333] The gpu memory used by Stage 0 is 3097207948
[PB Debug 2023-Oct-17 02:52:15][src/assembleRegions_GPU.cu:414] The cpu memory used by Stage 0 is 810178340
[PB Error 2023-Oct-17 02:52:15][src/likehood_test.cu:951] cudaSafeCall() failed: out of memory, exiting.
aSafeCall() failed: out of memory, exiting.
023-Oct-17 02:52:15][src/likehood_test.cu:951] cudaSafeCall() failed: out of memory, exiting.

The command used with haplotypecaller is shown below and the --in-bam file is the file that was output from fq2bam

pbrun haplotypecaller \
    --ref hs38DH.fa \
    --in-bam HA2WPADXX_2.pb.bam \
    --in-recal-file HA2WPADXX_2.pb.BQSR-REPORT.txt \
    --out-variants "HA2WPADXX_2.HAPLOTYPECALLER.pb.vcf" \
    --logfile "HA2WPADXX_2.HAPLOTYPECALLER.log.txt" \
    --run-partition \
    --gpu-num-per-partition 1 \
    --verbose --x3 \
    --num-gpus 4

fq2bam had a --low-memory option. Is there something similar for haplotypecaller? Also, do I need to change/enable --htvc-low-memory and/or --num-htvc-threads to overcome this error?

haplotypecaller - NVIDIA Docs

FWIW, the same --in-bam file works well with deepvariant

pbrun deepvariant \
    --ref hs38DH.fa \
    --in-bam HA2WPADXX_2.pb.bam \
    --out-variants "HA2WPADXX_2.DEEPVARIANT.pb.vcf" \
    --logfile "HA2WPADXX_2.DEEPVARIANT.log.txt" \
    --num-streams-per-gpu 4 \
    --run-partition \
    --gpu-num-per-partition 1 \
    --verbose --x3 \
    --num-gpus 4

Thanks in advance.

[avenkatraman]

Hi Parabricks developers,

Just checking in on the above - any suggestions on how to circumvent this. Thanks in advance.

The haplotypecaller link in my original post does not work - here is the changed link - haplotypecaller - NVIDIA Docs

[gburnett]

for resolution see: post