Problem using interval file and parabricks deepvariant in 4.10 and 4.0.0

Healthcare Parabricks
1

I am getting an error, very long run times, and stalling on the same chromsomal region when using the --interval-file option with parabricks deepvariant in version 4.1.0 and 4.0.0. When running on whole exome bam files using 4 GPUs the run usually takes less than 5 minutes when no bed file is specified. However, specifiying a bed file with --interval-file option slows it down and leads to an error:

[src/PBBamFile.cpp:619] Loop went past the last block., expected off == iter->n_off - 1, exiting.

I have attached log files for 4.0.0 and 4.1.0 runs showing the error.

Below are the details on the GPUs that we are using

Fri Jun  9 10:56:27 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A30                      On | 00000000:3B:00.0 Off |                    0 |
| N/A   42C    P0               78W / 165W|  11645MiB / 24576MiB |     99%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A30                      On | 00000000:5E:00.0 Off |                    0 |
| N/A   36C    P0               70W / 165W|  11645MiB / 24576MiB |     97%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A30                      On | 00000000:86:00.0 Off |                    0 |
| N/A   37C    P0               72W / 165W|  11645MiB / 24576MiB |     73%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A30                      On | 00000000:AF:00.0 Off |                    0 |
| N/A   39C    P0              108W / 165W|  11645MiB / 24576MiB |     99%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   2263814      C   .../local/parabricks/binaries//bin/bwa    11610MiB |
|    1   N/A  N/A   2263814      C   .../local/parabricks/binaries//bin/bwa    11610MiB |
|    2   N/A  N/A   2263814      C   .../local/parabricks/binaries//bin/bwa    11610MiB |
|    3   N/A  N/A   2263814      C   .../local/parabricks/binaries//bin/bwa    11610MiB |
+---------------------------------------------------------------------------------------+

4.0.0.log (3.1 KB)
4.1.0.log (480.4 KB)

2

Hello @padraic.corcoran,

Do you have the commands you used to run this? We can have the engineering team take a look.

Thank you!

3

Below is the command:

pbrun deepvariant --ref /beegfs-storage/data/ref_genomes/GRCh38/reference_grasnatter/homo_sapiens.fasta --in-bam parabricks/pbrun_fq2bam/NA12878_N.bam --num-gpus 4 --out-variants parabricks/pbrun_deepvariant/NA12878_N.g.vcf --pb-model-file /beegfs-storage/data/parabricks/4.0.0-1/A30/wes/deepvariant_wes.eng --interval-file /beegfs-storage/projects/wp3/Reference_files/Manifest/Clinical_research_exome/TWIST/Twist_Comprehensive_Exome_Covered_Targets_hg38_20bp_pad.bed --disable-use-window-selector-model --gvcf  --tmp-dir parabricks/pbrun_deepvariant/NA12878

I should also add that this was run as part of a snakemake pipeline using singularity. Notably, the error was not detected as failed job in the pipeline and I only became aware of it when checking the parabricks deepvariant log files.