Problem with gpu

Healthcare Parabricks
1

According to this userguide Running Parabricks on AWS - NVIDIA Docs, the ec2 instance type should be “g4dn.4xlarge”. I followed and when executing the germline pipeline, I received this error:

WARNING
The system has 62 GB, however recommended RAM with 1 GPU is 64 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-15 17:13:23] GPU-BWA mem
[PB Info 2023-Jun-15 17:13:23] ProgressMeter    Reads           Base Pairs Aligned
[PB Warning 2023-Jun-15 17:13:28][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Error 2023-Jun-15 17:13:28][ParaBricks/src/check_error.cu:44] No GPUs active, shutting down due to previous error., exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...

So I change the instance type to “g4dn.8xlarge”, and got a new error:

[PB Warning 2023-Jun-15 17:06:34][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Error 2023-Jun-15 17:06:34][ParaBricks/src/check_error.cu:44] No GPUs active, shutting down due to previous error., exiting.

What should I do now?

2

Hey @huyen.nguyen,

There could be a few causes for this error. You are using the g4dn.4xlarge machine which has GPUs so that is all correct. Which AMI are you using? It could be that there is no driver or an incompatible version of cuda is on the machine.

Can you also run nvidia-smi on this machine and verify that the GPUs are usable?

Thank you!

3

Hi Gburnett,

I used a “Deep Learning’ AMI but I don’t remember which one but I am sure it supports g4 instances.

However, if I type the command ‘nvidia-smi’ it says the command does not exit. Does that mean the gpu driver is not installed?

Thanks

4

Can you share output of:
sudo lspci | grep -i nvidia

And this command:
sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

5

Hi

I am trying to run fq2bam using this tutorial here: Tutorials - NVIDIA Docs

I wanted to check if the moderator @gburnett and/or @andjoseph have tried out the stuff as mentioned in those tutorials on an AWS g4dn.12xlarge or any AWS g4 machine.

As experienced by @huyen.nguyen, I am also getting a similar error - I am using g4dn.12xlarge and nvidia/clara/clara-parabricks:4.1.1-1

Here is the nvidia-smi info

nvidia-smi
Fri Jun 23 06:06:22 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03   Driver Version: 470.182.03   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   46C    P0    28W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   43C    P0    27W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   45C    P0    28W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   42C    P0    27W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

This is the stderr of the fq2bam job

pbrun fq2bam \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --ref  $INDEX \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --out-duplicate-metrics "Tutorial_Sample_1.duplicates_metrics.txt" \
    --num-gpus 4
[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for parabricks_example_data/sample_1.fq.gz
and parabricks_example_data/sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[PB Info 2023-Jun-23 06:06:29] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:06:29] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-23 06:06:29] ||                              Version 4.1.1-1                             ||
[PB Info 2023-Jun-23 06:06:29] ||                       GPU-BWA mem, Sorting Phase-I                       ||
[PB Info 2023-Jun-23 06:06:29] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2023-Jun-23 06:06:42][ParaBricks/src/pbOpts.cu:245] 

WARNING
The system has 186 GB, however recommended RAM with 4 GPU is 196 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-23 06:06:43] GPU-BWA mem
[PB Info 2023-Jun-23 06:06:43] ProgressMeter	Reads		Base Pairs Aligned
[PB Warning 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Warning 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Warning 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Warning 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB e[31mErrore[0m 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:44] No GPUs active, shutting down due to previous error., exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation

Could not run fq2bam
Exiting pbrun ...

However, if I use nvidia/clara/clara-parabricks:4.0.1-1, the above runs fine. Hence, something is wrong with 4.1.1-1

For the sake of completeness, here is the output when using nvidia/clara/clara-parabricks:4.0.1-1

nvidia-smi
Fri Jun 23 06:46:14 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03   Driver Version: 470.182.03   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   47C    P0    27W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   46C    P0    26W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   48C    P0    27W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   44C    P0    26W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

pbrun fq2bam \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --ref  $INDEX \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --out-duplicate-metrics "Tutorial_Sample_1.duplicates_metrics.txt" \
    --num-gpus 4

[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for parabricks_example_data/sample_1.fq.gz
and parabricks_example_data/sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[PB Info 2023-Jun-23 06:46:21] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:46:21] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-23 06:46:21] ||                              Version 4.0.1-1                             ||
[PB Info 2023-Jun-23 06:46:21] ||                       GPU-BWA mem, Sorting Phase-I                       ||
[PB Info 2023-Jun-23 06:46:21] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2023-Jun-23 06:46:35][ParaBricks/src/pbOpts.cu:316] 

WARNING
The system has 186 GB, however recommended RAM with 4 GPU is 196 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-23 06:46:36] GPU-BWA mem
[PB Info 2023-Jun-23 06:46:36] ProgressMeter	Reads		Base Pairs Aligned
[PB Info 2023-Jun-23 06:46:50] 5043564		560000000
[PB Info 2023-Jun-23 06:46:56] 10087128	1180000000
[PB Info 2023-Jun-23 06:47:03] 15130692	1720000000
[PB Info 2023-Jun-23 06:47:09] 20174256	2340000000
[PB Info 2023-Jun-23 06:47:16] 25217820	2890000000
[PB Info 2023-Jun-23 06:47:22] 30261384	3460000000
[PB Info 2023-Jun-23 06:47:28] 35304948	4060000000
[PB Info 2023-Jun-23 06:47:35] 40348512	4650000000
[PB Info 2023-Jun-23 06:47:41] 45392076	5200000000
[PB Info 2023-Jun-23 06:47:48] 50435640	5820000000
[PB Info 2023-Jun-23 06:47:58] 
GPU-BWA Mem time: 82.209722 seconds
[PB Info 2023-Jun-23 06:47:58] GPU-BWA Mem is finished.


[main] CMD: /usr/local/parabricks/binaries//bin/bwa mem -Z ./pbOpts.txt parabricks_reference_data/fasta/Homo_sapiens_assembly38.fasta parabricks_example_data/sample_1.fq.gz parabricks_example_data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[main] Real time: 96.616 sec; CPU: 3141.947 sec
[PB Info 2023-Jun-23 06:47:58] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:47:58] ||        Program:                      GPU-BWA mem, Sorting Phase-I        ||
[PB Info 2023-Jun-23 06:47:58] ||        Version:                                           4.0.1-1        ||
[PB Info 2023-Jun-23 06:47:58] ||        Start Time:                       Fri Jun 23 06:46:21 2023        ||
[PB Info 2023-Jun-23 06:47:58] ||        End Time:                         Fri Jun 23 06:47:58 2023        ||
[PB Info 2023-Jun-23 06:47:58] ||        Total Time:                            1 minute 37 seconds        ||
[PB Info 2023-Jun-23 06:47:58] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:00] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:00] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-23 06:48:00] ||                              Version 4.0.1-1                             ||
[PB Info 2023-Jun-23 06:48:00] ||                             Sorting Phase-II                             ||
[PB Info 2023-Jun-23 06:48:00] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:00] progressMeter - Percentage
[PB Info 2023-Jun-23 06:48:00] 0.0	 0.00 GB
[PB Info 2023-Jun-23 06:48:10] Sorting and Marking: 10.000 seconds
[PB Info 2023-Jun-23 06:48:10] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:10] ||        Program:                                  Sorting Phase-II        ||
[PB Info 2023-Jun-23 06:48:10] ||        Version:                                           4.0.1-1        ||
[PB Info 2023-Jun-23 06:48:10] ||        Start Time:                       Fri Jun 23 06:48:00 2023        ||
[PB Info 2023-Jun-23 06:48:10] ||        End Time:                         Fri Jun 23 06:48:10 2023        ||
[PB Info 2023-Jun-23 06:48:10] ||        Total Time:                                     10 seconds        ||
[PB Info 2023-Jun-23 06:48:10] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:10] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:10] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-23 06:48:10] ||                              Version 4.0.1-1                             ||
[PB Info 2023-Jun-23 06:48:10] ||                         Marking Duplicates, BQSR                         ||
[PB Info 2023-Jun-23 06:48:10] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:10] progressMeter -	Percentage
[PB Info 2023-Jun-23 06:48:20] 55.3	 8.58 GB
[PB Info 2023-Jun-23 06:48:30] 100.0	 0.00 GB
[PB Info 2023-Jun-23 06:48:30] BQSR and writing final BAM:  20.035 seconds
[PB Info 2023-Jun-23 06:48:30] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:30] ||        Program:                          Marking Duplicates, BQSR        ||
[PB Info 2023-Jun-23 06:48:30] ||        Version:                                           4.0.1-1        ||
[PB Info 2023-Jun-23 06:48:30] ||        Start Time:                       Fri Jun 23 06:48:10 2023        ||
[PB Info 2023-Jun-23 06:48:30] ||        End Time:                         Fri Jun 23 06:48:30 2023        ||
[PB Info 2023-Jun-23 06:48:30] ||        Total Time:                                     20 seconds        ||
[PB Info 2023-Jun-23 06:48:30] ------------------------------------------------------------------------------
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation

As evident from nvidia-smi from both the above runs:

  • nvidia/clara/clara-parabricks:4.0.1-1 is using CUDA 11.7

  • nvidia/clara/clara-parabricks:4.1.1-1 is using CUDA 12.0

so - it could very well be that the CUDA version 12.0 is causing these particular issues with nvidia/clara/clara-parabricks:4.1.1-1

6

Hi,

The latest version of Parabricks’s fq2bam requires 24GB of memory.
Can you please try running with --low-memory option?

Best

7

Hi @mdemouth

Based on your suggestion, I tried this below with nvcr.io/nvidia/clara/clara-parabricks:4.1.1-1 and get this error

/home/jenkins/agent/workspace/parabricks-branch-build//sortcommon/inc/compressfile.h:439] LZ4 decompression returned an error code of -1, expected decBytes > 0, exiting.
Received signal: 11

pbrun fq2bam \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --ref  $INDEX \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --out-duplicate-metrics "Tutorial_Sample_1.duplicates_metrics.txt" \
    --num-gpus 4 \
    --num-cpu-threads 16 \
    --low-memory \
    --gpusort \
    --gpuwrite

[PB Info 2023-Jun-23 18:56:17] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:56:17] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-23 18:56:17] ||                              Version 4.1.1-1                             ||
[PB Info 2023-Jun-23 18:56:17] ||                       GPU-BWA mem, Sorting Phase-I                       ||
[PB Info 2023-Jun-23 18:56:17] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2023-Jun-23 18:56:30][ParaBricks/src/pbOpts.cu:245] 

WARNING
The system has 186 GB, however recommended RAM with 4 GPU is 196 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-23 18:56:31] GPU-BWA mem
[PB Info 2023-Jun-23 18:56:31] ProgressMeter	Reads		Base Pairs Aligned
[PB Info 2023-Jun-23 18:56:46] 5043564		560000000
[PB Info 2023-Jun-23 18:56:55] 10087128	1180000000
[PB Info 2023-Jun-23 18:57:02] 15130692	1730000000
[PB Info 2023-Jun-23 18:57:10] 20174256	2270000000
[PB Info 2023-Jun-23 18:57:18] 25217820	2840000000
[PB Info 2023-Jun-23 18:57:25] 30261384	3500000000
[PB Info 2023-Jun-23 18:57:33] 35304948	4050000000
[PB Info 2023-Jun-23 18:57:40] 40348512	4620000000
[PB Info 2023-Jun-23 18:57:47] 45392076	5210000000
[PB Info 2023-Jun-23 18:57:55] 50435640	5780000000
[PB Info 2023-Jun-23 18:58:05] 
GPU-BWA Mem time: 94.789358 seconds
[PB Info 2023-Jun-23 18:58:05] GPU-BWA Mem is finished.

[main] CMD: /usr/local/parabricks/binaries//bin/bwa mem -Z ./pbOpts.txt -l -F 0 Homo_sapiens_assembly38.fasta sample_1.fq.gz sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[main] Real time: 108.345 sec; CPU: 4074.803 sec
[PB Info 2023-Jun-23 18:58:05] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:58:05] ||        Program:                      GPU-BWA mem, Sorting Phase-I        ||
[PB Info 2023-Jun-23 18:58:05] ||        Version:                                           4.1.1-1        ||
[PB Info 2023-Jun-23 18:58:05] ||        Start Time:                       Fri Jun 23 18:56:17 2023        ||
[PB Info 2023-Jun-23 18:58:05] ||        End Time:                         Fri Jun 23 18:58:05 2023        ||
[PB Info 2023-Jun-23 18:58:05] ||        Total Time:                            1 minute 48 seconds        ||
[PB Info 2023-Jun-23 18:58:05] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:58:07] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:58:07] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-23 18:58:07] ||                              Version 4.1.1-1                             ||
[PB Info 2023-Jun-23 18:58:07] ||                             Sorting Phase-II                             ||
[PB Info 2023-Jun-23 18:58:07] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:58:07] Using CPU-mode for sort when duplicate metrics file is requested.
[PB Info 2023-Jun-23 18:58:07] Performance may be slower than GPU-mode.
[PB Info 2023-Jun-23 18:58:07] progressMeter - Percentage
[PB Info 2023-Jun-23 18:58:07] 0.0	 0.00 GB
[PB e[31mErrore[0m 2023-Jun-23 18:58:09][/home/jenkins/agent/workspace/parabricks-branch-build//sortcommon/inc/compressfile.h:439] LZ4 decompression returned an error code of -1, expected decBytes > 0, exiting.
[PB e[31mErrore[0m 2023-Jun-23 18:58:09][-unknown-:0] [PB e[31mErrore[0m 2023-Jun-23 18:58:09][-unknown-:0] Received signal: 11
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html, exiting.
Received signal: 11
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html, exiting.
[PB e[31mErrore[0m 2023-Jun-23 18:58:09][-unknown-:0] Received signal: 11
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html, exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation
Could not run fq2bam
Exiting pbrun ...
8

Hi @mdemouth @gburnett

I now tried this with nvcr.io/nvidia/clara/clara-parabricks:4.1.0-1 and I get this error [src/PBCuBGZFWriter.cu:590] CUDA_CHECK() failed with out of memory (2), exiting

Full log message:

pbrun fq2bam \
    --ref  $INDEX \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --num-gpus 4 \
    --low-memory --gpusort --gpuwrite

[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for sample_1.fq.gz and sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[PB Info 2023-Jun-24 23:07:07] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:07:07] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-24 23:07:07] ||                              Version 4.1.0-1                             ||
[PB Info 2023-Jun-24 23:07:07] ||                       GPU-BWA mem, Sorting Phase-I                       ||
[PB Info 2023-Jun-24 23:07:07] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2023-Jun-24 23:07:18][ParaBricks/src/pbOpts.cu:245] 

WARNING
The system has 186 GB, however recommended RAM with 4 GPU is 196 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-24 23:07:19] GPU-BWA mem
[PB Info 2023-Jun-24 23:07:19] ProgressMeter	Reads		Base Pairs Aligned
[PB Info 2023-Jun-24 23:07:31] 5043564		580000000
[PB Info 2023-Jun-24 23:07:36] 10087128	1170000000
[PB Info 2023-Jun-24 23:07:41] 15130692	1740000000
[PB Info 2023-Jun-24 23:07:47] 20174256	2320000000
[PB Info 2023-Jun-24 23:07:52] 25217820	2900000000
[PB Info 2023-Jun-24 23:07:57] 30261384	3470000000
[PB Info 2023-Jun-24 23:08:03] 35304948	4060000000
[PB Info 2023-Jun-24 23:08:08] 40348512	4640000000
[PB Info 2023-Jun-24 23:08:13] 45392076	5220000000
[PB Info 2023-Jun-24 23:08:18] 50435640	5820000000
[PB Info 2023-Jun-24 23:08:27] 
GPU-BWA Mem time: 67.698456 seconds
[PB Info 2023-Jun-24 23:08:27] GPU-BWA Mem is finished.

[main] CMD: /usr/local/parabricks/binaries//bin/bwa mem -Z ./pbOpts.txt -l -F 0 Homo_sapiens_assembly38.fasta sample_1.fq.gz sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[main] Real time: 79.572 sec; CPU: 2585.088 sec
[PB Info 2023-Jun-24 23:08:27] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:27] ||        Program:                      GPU-BWA mem, Sorting Phase-I        ||
[PB Info 2023-Jun-24 23:08:27] ||        Version:                                           4.1.0-1        ||
[PB Info 2023-Jun-24 23:08:27] ||        Start Time:                       Sat Jun 24 23:07:07 2023        ||
[PB Info 2023-Jun-24 23:08:27] ||        End Time:                         Sat Jun 24 23:08:27 2023        ||
[PB Info 2023-Jun-24 23:08:27] ||        Total Time:                            1 minute 20 seconds        ||
[PB Info 2023-Jun-24 23:08:27] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:29] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:29] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-24 23:08:29] ||                              Version 4.1.0-1                             ||
[PB Info 2023-Jun-24 23:08:29] ||                             Sorting Phase-II                             ||
[PB Info 2023-Jun-24 23:08:29] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:29] progressMeter - Percentage
[PB Info 2023-Jun-24 23:08:29] 0.0	 0.00 GB
[PB Info 2023-Jun-24 23:08:34] Sorting and Marking: 5.000 seconds
[PB Info 2023-Jun-24 23:08:34] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:34] ||        Program:                                  Sorting Phase-II        ||
[PB Info 2023-Jun-24 23:08:34] ||        Version:                                           4.1.0-1        ||
[PB Info 2023-Jun-24 23:08:34] ||        Start Time:                       Sat Jun 24 23:08:29 2023        ||
[PB Info 2023-Jun-24 23:08:34] ||        End Time:                         Sat Jun 24 23:08:34 2023        ||
[PB Info 2023-Jun-24 23:08:34] ||        Total Time:                                      5 seconds        ||
[PB Info 2023-Jun-24 23:08:34] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:34] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:34] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jun-24 23:08:34] ||                              Version 4.1.0-1                             ||
[PB Info 2023-Jun-24 23:08:34] ||                         Marking Duplicates, BQSR                         ||
[PB Info 2023-Jun-24 23:08:34] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:34] CuBamWriter using CUDA device 3
[PB Info 2023-Jun-24 23:08:37] Using PBCuBamWriter for BAM writing
[PB e[31mErrore[0m 2023-Jun-24 23:08:37][src/PBCuBGZFWriter.cu:590] CUDA_CHECK() failed with out of memory (2), exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation

Could not run fq2bam
Exiting pbrun ...

Let me know if you need additional info.

9

Hi @avenkatraman,

Please try without using--gpusort --gpuwrite with T4.

pbrun fq2bam \
    --ref  $INDEX \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --num-gpus 4 \
    --low-memory
12

HI @mdemouth ,

Does use --low-memory affect the output?

14

Hi @mdemouth

Where does --gpusort --gpuwrite come in handy then? The nvidia docs for fq2bam best performance suggest their usage

I am trying this out on a g4dn.12xlarge which has 4gpus 48cpus 192GiB mem and 900GB NVME SSD - I am using docker and nvcr.io/nvidia/clara/clara-parabricks:4.1.1-1

Sat Jul 29 00:36:16 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03   Driver Version: 470.182.03   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   41C    P0    27W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   39C    P0    26W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla T4            Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   39C    P0    25W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   40C    P0    26W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

As you can see from the below pb_fq2bam finishes the BWA-MEM portion of it, but errs out during MarkDuplicates, BQSR with some CUDA errors

[PB Info 2023-Jul-29 01:08:55] 577766824    85240000000
[PB Info 2023-Jul-29 01:09:12] 582788250    85990000000
[PB Info 2023-Jul-29 01:09:28] 587809088    86740000000
[PB Info 2023-Jul-29 01:09:44] 592828882    87450000000
[PB Info 2023-Jul-29 01:09:54]
GPU-BWA Mem time: 1990.227598 seconds
[PB Info 2023-Jul-29 01:09:54] GPU-BWA Mem is finished.
[main] CMD: /usr/local/parabricks/binaries//bin/bwa mem -Z ./pbOpts.txt -M -l -F 0 . <other_options>

[main] Real time: 2008.670 sec; CPU: 86784.898 sec
[PB Info 2023-Jul-29 01:09:54] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:09:54] ||        Program:                      GPU-BWA mem, Sorting Phase-I        ||
[PB Info 2023-Jul-29 01:09:54] ||        Version:                                           4.1.1-1        ||
[PB Info 2023-Jul-29 01:09:54] ||        Start Time:                       Sat Jul 29 00:36:25 2023        ||
[PB Info 2023-Jul-29 01:09:54] ||        End Time:                         Sat Jul 29 01:09:54 2023        ||
[PB Info 2023-Jul-29 01:09:54] ||        Total Time:                          33 minutes 29 seconds        ||
[PB Info 2023-Jul-29 01:09:54] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:09:56] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:09:56] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jul-29 01:09:56] ||                              Version 4.1.1-1                             ||
[PB Info 2023-Jul-29 01:09:56] ||                             Sorting Phase-II                             ||
[PB Info 2023-Jul-29 01:09:56] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:09:56] progressMeter - Percentage
[PB Info 2023-Jul-29 01:09:56] 0.0     0.00 GB
[PB Info 2023-Jul-29 01:10:01] 10.1     2.00 GB
[PB Info 2023-Jul-29 01:10:06] 37.7     2.00 GB
[PB Info 2023-Jul-29 01:10:11] 64.3     1.00 GB
[PB Info 2023-Jul-29 01:10:16] 94.0     1.00 GB
[PB Info 2023-Jul-29 01:10:21] Sorting and Marking: 25.001 seconds
[PB Info 2023-Jul-29 01:10:21] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:10:21] ||        Program:                                  Sorting Phase-II        ||
[PB Info 2023-Jul-29 01:10:21] ||        Version:                                           4.1.1-1        ||
[PB Info 2023-Jul-29 01:10:21] ||        Start Time:                       Sat Jul 29 01:09:56 2023        ||
[PB Info 2023-Jul-29 01:10:21] ||        End Time:                         Sat Jul 29 01:10:21 2023        ||
[PB Info 2023-Jul-29 01:10:21] ||        Total Time:                                     25 seconds        ||
[PB Info 2023-Jul-29 01:10:21] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:10:22] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:10:22] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2023-Jul-29 01:10:22] ||                              Version 4.1.1-1                             ||
[PB Info 2023-Jul-29 01:10:22] ||                         Marking Duplicates, BQSR                         ||
[PB Info 2023-Jul-29 01:10:22] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:10:22] CuBamWriter using CUDA device 3
[PB Info 2023-Jul-29 01:10:22] BQSR using CUDA device(s): { 0 1 2 }
[PB Info 2023-Jul-29 01:10:24] Using PBCuBamWriter for BAM writing
[PB Error 2023-Jul-29 01:10:25][src/PBCuBGZFWriter.cu:590] CUDA_CHECK() failed with out of memory (2), exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...

Thanks.