I am getting an error, very long run times, and stalling on the same chromsomal region when using the --interval-file option with parabricks deepvariant in version 4.1.0 and 4.0.0. When running on whole exome bam files using 4 GPUs the run usually takes less than 5 minutes when no bed file is specified. However, specifiying a bed file with --interval-file option slows it down and leads to an error:
[src/PBBamFile.cpp:619] Loop went past the last block., expected off == iter->n_off - 1, exiting.
I have attached log files for 4.0.0 and 4.1.0 runs showing the error.
Below are the details on the GPUs that we are using
Fri Jun 9 10:56:27 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02 Driver Version: 530.30.02 CUDA Version: 12.1 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A30 On | 00000000:3B:00.0 Off | 0 |
| N/A 42C P0 78W / 165W| 11645MiB / 24576MiB | 99% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A30 On | 00000000:5E:00.0 Off | 0 |
| N/A 36C P0 70W / 165W| 11645MiB / 24576MiB | 97% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA A30 On | 00000000:86:00.0 Off | 0 |
| N/A 37C P0 72W / 165W| 11645MiB / 24576MiB | 73% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA A30 On | 00000000:AF:00.0 Off | 0 |
| N/A 39C P0 108W / 165W| 11645MiB / 24576MiB | 99% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2263814 C .../local/parabricks/binaries//bin/bwa 11610MiB |
| 1 N/A N/A 2263814 C .../local/parabricks/binaries//bin/bwa 11610MiB |
| 2 N/A N/A 2263814 C .../local/parabricks/binaries//bin/bwa 11610MiB |
| 3 N/A N/A 2263814 C .../local/parabricks/binaries//bin/bwa 11610MiB |
+---------------------------------------------------------------------------------------+