# Problem with gpu

**URL:** <https://forums.developer.nvidia.com/t/problem-with-gpu/256825>\
**Category:** Parabricks\
**Tags:** ai\
**Created:** [June 15, 2023, 5:15pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825 "2023-06-15T17:15:11Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![huyen.nguyen](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@huyen.nguyen](https://forums.developer.nvidia.com/u/huyen.nguyen)\
**Post date:** [June 15, 2023, 5:15pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/1 "2023-06-15T17:15:12Z")

</div>

According to this userguide [Running Parabricks on AWS - NVIDIA Docs](https://docs.nvidia.com/clara/parabricks/4.1.0/tutorials/cloudguides/aws.html), the ec2 instance type should be “g4dn.4xlarge”. I followed and when executing the germline pipeline, I received this error:

```auto
WARNING
The system has 62 GB, however recommended RAM with 1 GPU is 64 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-15 17:13:23] GPU-BWA mem
[PB Info 2023-Jun-15 17:13:23] ProgressMeter Reads Base Pairs Aligned
[PB Warning 2023-Jun-15 17:13:28][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Error 2023-Jun-15 17:13:28][ParaBricks/src/check_error.cu:44] No GPUs active, shutting down due to previous error., exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...

```

So I change the instance type to “g4dn.8xlarge”, and got a new error:

```auto
[PB Warning 2023-Jun-15 17:06:34][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Error 2023-Jun-15 17:06:34][ParaBricks/src/check_error.cu:44] No GPUs active, shutting down due to previous error., exiting.

```

What should I do now?

---

<div class="post-metadata">

**Author:** ![gburnett](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/gburnett/32/14047_2.png) [@gburnett](https://forums.developer.nvidia.com/u/gburnett)\
**Post date:** [June 21, 2023, 6:17pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/2 "2023-06-21T18:17:28Z")

</div>

Hey @huyen.nguyen,

There could be a few causes for this error. You are using the g4dn.4xlarge machine which has GPUs so that is all correct. Which AMI are you using? It could be that there is no driver or an incompatible version of cuda is on the machine.

Can you also run nvidia-smi on this machine and verify that the GPUs are usable?

Thank you!

---

<div class="post-metadata">

**Author:** ![huyen.nguyen](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@huyen.nguyen](https://forums.developer.nvidia.com/u/huyen.nguyen)\
**Post date:** [June 22, 2023, 5:36pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/3 "2023-06-22T17:36:40Z")

</div>

Hi Gburnett,

I used a “Deep Learning’ AMI but I don’t remember which one but I am sure it supports g4 instances.

However, if I type the command ‘nvidia-smi’ it says the command does not exit. Does that mean the gpu driver is not installed?

Thanks

---

<div class="post-metadata">

**Author:** ![andjoseph](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@andjoseph](https://forums.developer.nvidia.com/u/andjoseph)\
**Post date:** [June 22, 2023, 7:19pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/4 "2023-06-22T19:19:47Z")

</div>

Can you share output of:  
sudo lspci | grep -i nvidia

And this command:  
sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

---

<div class="post-metadata">

**Author:** ![avenkatraman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@avenkatraman](https://forums.developer.nvidia.com/u/avenkatraman)\
**Post date:** [June 23, 2023, 6:21am UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/5 "2023-06-23T06:21:05Z")

</div>

Hi

I am trying to run fq2bam using this tutorial here: [Tutorials - NVIDIA Docs](https://docs.nvidia.com/clara/parabricks/4.1.0/tutorials.html#tutorials)

I wanted to check if the moderator @gburnett and/or @andjoseph have tried out the stuff as mentioned in those tutorials on an AWS g4dn.12xlarge or any AWS g4 machine.

As experienced by @huyen.nguyen, I am also getting a similar error - I am using g4dn.12xlarge and nvidia/clara/clara-parabricks:4.1.1-1

Here is the nvidia-smi info

```auto
nvidia-smi
Fri Jun 23 06:06:22 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:1B.0 Off | 0 |
| N/A 46C P0 28W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:1C.0 Off | 0 |
| N/A 43C P0 27W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 Tesla T4 Off | 00000000:00:1D.0 Off | 0 |
| N/A 45C P0 28W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 Tesla T4 Off | 00000000:00:1E.0 Off | 0 |
| N/A 42C P0 27W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

```

This is the stderr of the `fq2bam` job

```auto
pbrun fq2bam \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --ref $INDEX \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --out-duplicate-metrics "Tutorial_Sample_1.duplicates_metrics.txt" \
    --num-gpus 4
[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for parabricks_example_data/sample_1.fq.gz
and parabricks_example_data/sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[PB Info 2023-Jun-23 06:06:29] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:06:29] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-23 06:06:29] || Version 4.1.1-1 ||
[PB Info 2023-Jun-23 06:06:29] || GPU-BWA mem, Sorting Phase-I ||
[PB Info 2023-Jun-23 06:06:29] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2023-Jun-23 06:06:42][ParaBricks/src/pbOpts.cu:245] 

WARNING
The system has 186 GB, however recommended RAM with 4 GPU is 196 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-23 06:06:43] GPU-BWA mem
[PB Info 2023-Jun-23 06:06:43] ProgressMeter	Reads Base Pairs Aligned
[PB Warning 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Warning 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Warning 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB Warning 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:41] cudaSafeCall() failed at ParaBricks/src/samGenerator.cu/771: out of memory
[PB e[31mErrore[0m 2023-Jun-23 06:06:55][ParaBricks/src/check_error.cu:44] No GPUs active, shutting down due to previous error., exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation

Could not run fq2bam
Exiting pbrun ...

```

However, if I use nvidia/clara/clara-parabricks:4.0.1-1, the above runs fine. Hence, something is wrong with 4.1.1-1

For the sake of completeness, here is the output when using nvidia/clara/clara-parabricks:4.0.1-1

```auto
nvidia-smi
Fri Jun 23 06:46:14 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:1B.0 Off | 0 |
| N/A 47C P0 27W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:1C.0 Off | 0 |
| N/A 46C P0 26W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 Tesla T4 Off | 00000000:00:1D.0 Off | 0 |
| N/A 48C P0 27W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 Tesla T4 Off | 00000000:00:1E.0 Off | 0 |
| N/A 44C P0 26W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

pbrun fq2bam \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --ref $INDEX \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --out-duplicate-metrics "Tutorial_Sample_1.duplicates_metrics.txt" \
    --num-gpus 4

[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for parabricks_example_data/sample_1.fq.gz
and parabricks_example_data/sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[PB Info 2023-Jun-23 06:46:21] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:46:21] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-23 06:46:21] || Version 4.0.1-1 ||
[PB Info 2023-Jun-23 06:46:21] || GPU-BWA mem, Sorting Phase-I ||
[PB Info 2023-Jun-23 06:46:21] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2023-Jun-23 06:46:35][ParaBricks/src/pbOpts.cu:316] 

WARNING
The system has 186 GB, however recommended RAM with 4 GPU is 196 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-23 06:46:36] GPU-BWA mem
[PB Info 2023-Jun-23 06:46:36] ProgressMeter	Reads Base Pairs Aligned
[PB Info 2023-Jun-23 06:46:50] 5043564 560000000
[PB Info 2023-Jun-23 06:46:56] 10087128	1180000000
[PB Info 2023-Jun-23 06:47:03] 15130692	1720000000
[PB Info 2023-Jun-23 06:47:09] 20174256	2340000000
[PB Info 2023-Jun-23 06:47:16] 25217820	2890000000
[PB Info 2023-Jun-23 06:47:22] 30261384	3460000000
[PB Info 2023-Jun-23 06:47:28] 35304948	4060000000
[PB Info 2023-Jun-23 06:47:35] 40348512	4650000000
[PB Info 2023-Jun-23 06:47:41] 45392076	5200000000
[PB Info 2023-Jun-23 06:47:48] 50435640	5820000000
[PB Info 2023-Jun-23 06:47:58] 
GPU-BWA Mem time: 82.209722 seconds
[PB Info 2023-Jun-23 06:47:58] GPU-BWA Mem is finished.

[main] CMD: /usr/local/parabricks/binaries//bin/bwa mem -Z ./pbOpts.txt parabricks_reference_data/fasta/Homo_sapiens_assembly38.fasta parabricks_example_data/sample_1.fq.gz parabricks_example_data/sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[main] Real time: 96.616 sec; CPU: 3141.947 sec
[PB Info 2023-Jun-23 06:47:58] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:47:58] || Program: GPU-BWA mem, Sorting Phase-I ||
[PB Info 2023-Jun-23 06:47:58] || Version: 4.0.1-1 ||
[PB Info 2023-Jun-23 06:47:58] || Start Time: Fri Jun 23 06:46:21 2023 ||
[PB Info 2023-Jun-23 06:47:58] || End Time: Fri Jun 23 06:47:58 2023 ||
[PB Info 2023-Jun-23 06:47:58] || Total Time: 1 minute 37 seconds ||
[PB Info 2023-Jun-23 06:47:58] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:00] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:00] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-23 06:48:00] || Version 4.0.1-1 ||
[PB Info 2023-Jun-23 06:48:00] || Sorting Phase-II ||
[PB Info 2023-Jun-23 06:48:00] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:00] progressMeter - Percentage
[PB Info 2023-Jun-23 06:48:00] 0.0 0.00 GB
[PB Info 2023-Jun-23 06:48:10] Sorting and Marking: 10.000 seconds
[PB Info 2023-Jun-23 06:48:10] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:10] || Program: Sorting Phase-II ||
[PB Info 2023-Jun-23 06:48:10] || Version: 4.0.1-1 ||
[PB Info 2023-Jun-23 06:48:10] || Start Time: Fri Jun 23 06:48:00 2023 ||
[PB Info 2023-Jun-23 06:48:10] || End Time: Fri Jun 23 06:48:10 2023 ||
[PB Info 2023-Jun-23 06:48:10] || Total Time: 10 seconds ||
[PB Info 2023-Jun-23 06:48:10] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:10] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:10] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-23 06:48:10] || Version 4.0.1-1 ||
[PB Info 2023-Jun-23 06:48:10] || Marking Duplicates, BQSR ||
[PB Info 2023-Jun-23 06:48:10] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:10] progressMeter -	Percentage
[PB Info 2023-Jun-23 06:48:20] 55.3 8.58 GB
[PB Info 2023-Jun-23 06:48:30] 100.0 0.00 GB
[PB Info 2023-Jun-23 06:48:30] BQSR and writing final BAM: 20.035 seconds
[PB Info 2023-Jun-23 06:48:30] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 06:48:30] || Program: Marking Duplicates, BQSR ||
[PB Info 2023-Jun-23 06:48:30] || Version: 4.0.1-1 ||
[PB Info 2023-Jun-23 06:48:30] || Start Time: Fri Jun 23 06:48:10 2023 ||
[PB Info 2023-Jun-23 06:48:30] || End Time: Fri Jun 23 06:48:30 2023 ||
[PB Info 2023-Jun-23 06:48:30] || Total Time: 20 seconds ||
[PB Info 2023-Jun-23 06:48:30] ------------------------------------------------------------------------------
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation

```

As evident from `nvidia-smi` from both the above runs:

- `nvidia/clara/clara-parabricks:4.0.1-1` is using CUDA 11.7

- `nvidia/clara/clara-parabricks:4.1.1-1` is using CUDA 12.0

so - it could very well be that the CUDA version 12.0 is causing these particular issues with `nvidia/clara/clara-parabricks:4.1.1-1`

---

<div class="post-metadata">

**Author:** ![mdemouth](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mdemouth/32/16591_2.png) [@mdemouth](https://forums.developer.nvidia.com/u/mdemouth)\
**Post date:** [June 23, 2023, 8:36am UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/6 "2023-06-23T08:36:50Z")

</div>

Hi,

The latest version of Parabricks’s fq2bam requires 24GB of memory.  
Can you please try running with `--low-memory` option?

Best

---

<div class="post-metadata">

**Author:** ![avenkatraman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@avenkatraman](https://forums.developer.nvidia.com/u/avenkatraman)\
**Post date:** [June 23, 2023, 7:46pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/7 "2023-06-23T19:46:05Z")

</div>

Hi @mdemouth

Based on your suggestion, I tried this below with `nvcr.io/nvidia/clara/clara-parabricks:4.1.1-1` and get this error

```auto
/home/jenkins/agent/workspace/parabricks-branch-build//sortcommon/inc/compressfile.h:439] LZ4 decompression returned an error code of -1, expected decBytes > 0, exiting.
Received signal: 11

```

- Background on the command line and params used - see [Best Performance - NVIDIA Docs](https://docs.nvidia.com/clara/parabricks/4.1.0/bestperformance.html#best-performance-for-fq2bam)

```auto
pbrun fq2bam \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --ref $INDEX \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --out-duplicate-metrics "Tutorial_Sample_1.duplicates_metrics.txt" \
    --num-gpus 4 \
    --num-cpu-threads 16 \
    --low-memory \
    --gpusort \
    --gpuwrite

[PB Info 2023-Jun-23 18:56:17] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:56:17] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-23 18:56:17] || Version 4.1.1-1 ||
[PB Info 2023-Jun-23 18:56:17] || GPU-BWA mem, Sorting Phase-I ||
[PB Info 2023-Jun-23 18:56:17] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2023-Jun-23 18:56:30][ParaBricks/src/pbOpts.cu:245] 

WARNING
The system has 186 GB, however recommended RAM with 4 GPU is 196 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-23 18:56:31] GPU-BWA mem
[PB Info 2023-Jun-23 18:56:31] ProgressMeter	Reads Base Pairs Aligned
[PB Info 2023-Jun-23 18:56:46] 5043564 560000000
[PB Info 2023-Jun-23 18:56:55] 10087128	1180000000
[PB Info 2023-Jun-23 18:57:02] 15130692	1730000000
[PB Info 2023-Jun-23 18:57:10] 20174256	2270000000
[PB Info 2023-Jun-23 18:57:18] 25217820	2840000000
[PB Info 2023-Jun-23 18:57:25] 30261384	3500000000
[PB Info 2023-Jun-23 18:57:33] 35304948	4050000000
[PB Info 2023-Jun-23 18:57:40] 40348512	4620000000
[PB Info 2023-Jun-23 18:57:47] 45392076	5210000000
[PB Info 2023-Jun-23 18:57:55] 50435640	5780000000
[PB Info 2023-Jun-23 18:58:05] 
GPU-BWA Mem time: 94.789358 seconds
[PB Info 2023-Jun-23 18:58:05] GPU-BWA Mem is finished.

[main] CMD: /usr/local/parabricks/binaries//bin/bwa mem -Z ./pbOpts.txt -l -F 0 Homo_sapiens_assembly38.fasta sample_1.fq.gz sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[main] Real time: 108.345 sec; CPU: 4074.803 sec
[PB Info 2023-Jun-23 18:58:05] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:58:05] || Program: GPU-BWA mem, Sorting Phase-I ||
[PB Info 2023-Jun-23 18:58:05] || Version: 4.1.1-1 ||
[PB Info 2023-Jun-23 18:58:05] || Start Time: Fri Jun 23 18:56:17 2023 ||
[PB Info 2023-Jun-23 18:58:05] || End Time: Fri Jun 23 18:58:05 2023 ||
[PB Info 2023-Jun-23 18:58:05] || Total Time: 1 minute 48 seconds ||
[PB Info 2023-Jun-23 18:58:05] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:58:07] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:58:07] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-23 18:58:07] || Version 4.1.1-1 ||
[PB Info 2023-Jun-23 18:58:07] || Sorting Phase-II ||
[PB Info 2023-Jun-23 18:58:07] ------------------------------------------------------------------------------
[PB Info 2023-Jun-23 18:58:07] Using CPU-mode for sort when duplicate metrics file is requested.
[PB Info 2023-Jun-23 18:58:07] Performance may be slower than GPU-mode.
[PB Info 2023-Jun-23 18:58:07] progressMeter - Percentage
[PB Info 2023-Jun-23 18:58:07] 0.0 0.00 GB
[PB e[31mErrore[0m 2023-Jun-23 18:58:09][/home/jenkins/agent/workspace/parabricks-branch-build//sortcommon/inc/compressfile.h:439] LZ4 decompression returned an error code of -1, expected decBytes > 0, exiting.
[PB e[31mErrore[0m 2023-Jun-23 18:58:09][-unknown-:0] [PB e[31mErrore[0m 2023-Jun-23 18:58:09][-unknown-:0] Received signal: 11
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html, exiting.
Received signal: 11
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html, exiting.
[PB e[31mErrore[0m 2023-Jun-23 18:58:09][-unknown-:0] Received signal: 11
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html, exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation
Could not run fq2bam
Exiting pbrun ...

```

---

<div class="post-metadata">

**Author:** ![avenkatraman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@avenkatraman](https://forums.developer.nvidia.com/u/avenkatraman)\
**Post date:** [June 24, 2023, 11:17pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/8 "2023-06-24T23:17:44Z")

</div>

Hi @mdemouth @gburnett

I now tried this with `nvcr.io/nvidia/clara/clara-parabricks:4.1.0-1` and I get this error `[src/PBCuBGZFWriter.cu:590] CUDA_CHECK() failed with out of memory (2), exiting`

Full log message:

```auto
pbrun fq2bam \
    --ref $INDEX \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --num-gpus 4 \
    --low-memory --gpusort --gpuwrite

[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for sample_1.fq.gz and sample_2.fq.gz
[Parabricks Options Mesg]: @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[PB Info 2023-Jun-24 23:07:07] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:07:07] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-24 23:07:07] || Version 4.1.0-1 ||
[PB Info 2023-Jun-24 23:07:07] || GPU-BWA mem, Sorting Phase-I ||
[PB Info 2023-Jun-24 23:07:07] ------------------------------------------------------------------------------
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[PB Warning 2023-Jun-24 23:07:18][ParaBricks/src/pbOpts.cu:245] 

WARNING
The system has 186 GB, however recommended RAM with 4 GPU is 196 GB.
The run might not finish or might have less than expected performance.

[PB Info 2023-Jun-24 23:07:19] GPU-BWA mem
[PB Info 2023-Jun-24 23:07:19] ProgressMeter	Reads Base Pairs Aligned
[PB Info 2023-Jun-24 23:07:31] 5043564 580000000
[PB Info 2023-Jun-24 23:07:36] 10087128	1170000000
[PB Info 2023-Jun-24 23:07:41] 15130692	1740000000
[PB Info 2023-Jun-24 23:07:47] 20174256	2320000000
[PB Info 2023-Jun-24 23:07:52] 25217820	2900000000
[PB Info 2023-Jun-24 23:07:57] 30261384	3470000000
[PB Info 2023-Jun-24 23:08:03] 35304948	4060000000
[PB Info 2023-Jun-24 23:08:08] 40348512	4640000000
[PB Info 2023-Jun-24 23:08:13] 45392076	5220000000
[PB Info 2023-Jun-24 23:08:18] 50435640	5820000000
[PB Info 2023-Jun-24 23:08:27] 
GPU-BWA Mem time: 67.698456 seconds
[PB Info 2023-Jun-24 23:08:27] GPU-BWA Mem is finished.

[main] CMD: /usr/local/parabricks/binaries//bin/bwa mem -Z ./pbOpts.txt -l -F 0 Homo_sapiens_assembly38.fasta sample_1.fq.gz sample_2.fq.gz @RG\tID:HK3TJBCX2.1\tLB:lib1\tPL:bar\tSM:sample\tPU:HK3TJBCX2.1
[main] Real time: 79.572 sec; CPU: 2585.088 sec
[PB Info 2023-Jun-24 23:08:27] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:27] || Program: GPU-BWA mem, Sorting Phase-I ||
[PB Info 2023-Jun-24 23:08:27] || Version: 4.1.0-1 ||
[PB Info 2023-Jun-24 23:08:27] || Start Time: Sat Jun 24 23:07:07 2023 ||
[PB Info 2023-Jun-24 23:08:27] || End Time: Sat Jun 24 23:08:27 2023 ||
[PB Info 2023-Jun-24 23:08:27] || Total Time: 1 minute 20 seconds ||
[PB Info 2023-Jun-24 23:08:27] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:29] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:29] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-24 23:08:29] || Version 4.1.0-1 ||
[PB Info 2023-Jun-24 23:08:29] || Sorting Phase-II ||
[PB Info 2023-Jun-24 23:08:29] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:29] progressMeter - Percentage
[PB Info 2023-Jun-24 23:08:29] 0.0 0.00 GB
[PB Info 2023-Jun-24 23:08:34] Sorting and Marking: 5.000 seconds
[PB Info 2023-Jun-24 23:08:34] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:34] || Program: Sorting Phase-II ||
[PB Info 2023-Jun-24 23:08:34] || Version: 4.1.0-1 ||
[PB Info 2023-Jun-24 23:08:34] || Start Time: Sat Jun 24 23:08:29 2023 ||
[PB Info 2023-Jun-24 23:08:34] || End Time: Sat Jun 24 23:08:34 2023 ||
[PB Info 2023-Jun-24 23:08:34] || Total Time: 5 seconds ||
[PB Info 2023-Jun-24 23:08:34] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:34] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:34] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jun-24 23:08:34] || Version 4.1.0-1 ||
[PB Info 2023-Jun-24 23:08:34] || Marking Duplicates, BQSR ||
[PB Info 2023-Jun-24 23:08:34] ------------------------------------------------------------------------------
[PB Info 2023-Jun-24 23:08:34] CuBamWriter using CUDA device 3
[PB Info 2023-Jun-24 23:08:37] Using PBCuBamWriter for BAM writing
[PB e[31mErrore[0m 2023-Jun-24 23:08:37][src/PBCuBGZFWriter.cu:590] CUDA_CHECK() failed with out of memory (2), exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation

Could not run fq2bam
Exiting pbrun ...

```

Let me know if you need additional info.

---

<div class="post-metadata">

**Author:** ![mdemouth](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mdemouth/32/16591_2.png) [@mdemouth](https://forums.developer.nvidia.com/u/mdemouth)\
**Post date:** [July 6, 2023, 9:30am UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/9 "2023-07-06T09:30:01Z")

</div>

Hi @avenkatraman,

Please try without using`--gpusort --gpuwrite` with T4.

```auto
pbrun fq2bam \
    --ref $INDEX \
    --in-fq "sample_1.fq.gz" "sample_2.fq.gz" \
    --out-bam "Tutorial_Sample_1.pb.bam" \
    --logfile "Tutorial_Sample_1.FQ2BAM.log.txt" \
    --num-gpus 4 \
    --low-memory
```

---

<div class="post-metadata">

**Author:** ![huyen.nguyen](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@huyen.nguyen](https://forums.developer.nvidia.com/u/huyen.nguyen)\
**Post date:** [July 6, 2023, 5:13pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/12 "2023-07-06T17:13:33Z")

</div>

HI @mdemouth ,

Does use --low-memory affect the output?

---

<div class="post-metadata">

**Author:** ![avenkatraman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@avenkatraman](https://forums.developer.nvidia.com/u/avenkatraman)\
**Post date:** [July 29, 2023, 2:12am UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/14 "2023-07-29T02:12:26Z")

</div>

Hi @mdemouth

Where does `--gpusort --gpuwrite` come in handy then? The [nvidia docs for fq2bam best performance](https://docs.nvidia.com/clara/parabricks/4.1.0/bestperformance.html#best-performance-for-fq2bam) suggest their usage

I am trying this out on a [g4dn.12xlarge](https://aws.amazon.com/ec2/instance-types/g4/) which has `4gpus 48cpus 192GiB mem and 900GB NVME SSD` - I am using docker and `nvcr.io/nvidia/clara/clara-parabricks:4.1.1-1`

```auto
Sat Jul 29 00:36:16 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:1B.0 Off | 0 |
| N/A 41C P0 27W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:1C.0 Off | 0 |
| N/A 39C P0 26W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 Tesla T4 Off | 00000000:00:1D.0 Off | 0 |
| N/A 39C P0 25W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 Tesla T4 Off | 00000000:00:1E.0 Off | 0 |
| N/A 40C P0 26W / 70W | 0MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

```

As you can see from the below `pb_fq2bam` finishes the BWA-MEM portion of it, but errs out during `MarkDuplicates, BQSR` with some CUDA errors

```auto
[PB Info 2023-Jul-29 01:08:55] 577766824 85240000000
[PB Info 2023-Jul-29 01:09:12] 582788250 85990000000
[PB Info 2023-Jul-29 01:09:28] 587809088 86740000000
[PB Info 2023-Jul-29 01:09:44] 592828882 87450000000
[PB Info 2023-Jul-29 01:09:54]
GPU-BWA Mem time: 1990.227598 seconds
[PB Info 2023-Jul-29 01:09:54] GPU-BWA Mem is finished.
[main] CMD: /usr/local/parabricks/binaries//bin/bwa mem -Z ./pbOpts.txt -M -l -F 0 . <other_options>

[main] Real time: 2008.670 sec; CPU: 86784.898 sec
[PB Info 2023-Jul-29 01:09:54] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:09:54] || Program: GPU-BWA mem, Sorting Phase-I ||
[PB Info 2023-Jul-29 01:09:54] || Version: 4.1.1-1 ||
[PB Info 2023-Jul-29 01:09:54] || Start Time: Sat Jul 29 00:36:25 2023 ||
[PB Info 2023-Jul-29 01:09:54] || End Time: Sat Jul 29 01:09:54 2023 ||
[PB Info 2023-Jul-29 01:09:54] || Total Time: 33 minutes 29 seconds ||
[PB Info 2023-Jul-29 01:09:54] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:09:56] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:09:56] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jul-29 01:09:56] || Version 4.1.1-1 ||
[PB Info 2023-Jul-29 01:09:56] || Sorting Phase-II ||
[PB Info 2023-Jul-29 01:09:56] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:09:56] progressMeter - Percentage
[PB Info 2023-Jul-29 01:09:56] 0.0 0.00 GB
[PB Info 2023-Jul-29 01:10:01] 10.1 2.00 GB
[PB Info 2023-Jul-29 01:10:06] 37.7 2.00 GB
[PB Info 2023-Jul-29 01:10:11] 64.3 1.00 GB
[PB Info 2023-Jul-29 01:10:16] 94.0 1.00 GB
[PB Info 2023-Jul-29 01:10:21] Sorting and Marking: 25.001 seconds
[PB Info 2023-Jul-29 01:10:21] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:10:21] || Program: Sorting Phase-II ||
[PB Info 2023-Jul-29 01:10:21] || Version: 4.1.1-1 ||
[PB Info 2023-Jul-29 01:10:21] || Start Time: Sat Jul 29 01:09:56 2023 ||
[PB Info 2023-Jul-29 01:10:21] || End Time: Sat Jul 29 01:10:21 2023 ||
[PB Info 2023-Jul-29 01:10:21] || Total Time: 25 seconds ||
[PB Info 2023-Jul-29 01:10:21] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:10:22] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:10:22] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2023-Jul-29 01:10:22] || Version 4.1.1-1 ||
[PB Info 2023-Jul-29 01:10:22] || Marking Duplicates, BQSR ||
[PB Info 2023-Jul-29 01:10:22] ------------------------------------------------------------------------------
[PB Info 2023-Jul-29 01:10:22] CuBamWriter using CUDA device 3
[PB Info 2023-Jul-29 01:10:22] BQSR using CUDA device(s): { 0 1 2 }
[PB Info 2023-Jul-29 01:10:24] Using PBCuBamWriter for BAM writing
[PB Error 2023-Jul-29 01:10:25][src/PBCuBGZFWriter.cu:590] CUDA_CHECK() failed with out of memory (2), exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.1.0/Help.html
Exiting...

```

Thanks.

---

<div class="post-metadata">

**Author:** ![tongz](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tongz](https://forums.developer.nvidia.com/u/tongz)\
**Post date:** [November 1, 2024, 10:07pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/15 "2024-11-01T22:07:46Z")

</div>

Hi @huyen.nguyen

No --low-memory doesn’t affcet the output.

---

<div class="post-metadata">

**Author:** ![tongz](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tongz](https://forums.developer.nvidia.com/u/tongz)\
**Post date:** [November 1, 2024, 10:10pm UTC](https://forums.developer.nvidia.com/t/problem-with-gpu/256825/16 "2024-11-01T22:10:54Z")

</div>

Hi @avenkatraman

–gpusort and --gpuwrite give you more speedup but requires more GPU memory. Since you are running on T4 GPUs which only has 16GB GPU memory and that is not enough to enable these 2 flags. So please get rid of both and try again.