Hi,
I’ve been trying to perform an alignment using pbrun minimap2 but it seems to crash after a few minutes. Below is my command and log output:
#!/bin/bash
#SBATCH -p lrz-hgx-h100-92x4
#SBATCH --gres=gpu:3
#SBATCH --cpus-per-task=40
#SBATCH --mem-per-cpu=6G
#SBATCH --time=05:00:00
echo "Job Started"
# Define the path to the Enroot container file
CONTAINER_NAME="NVIDIA_Parabricks_2"
CONTAINER_FILE="NVIDIA_Parabricks.sqsh"
if ! enroot list | grep -q "^$CONTAINER_NAME$"; then
echo "Container '$CONTAINER_NAME' does not exist. Creating it..."
enroot create -n "$CONTAINER_NAME" "$CONTAINER_FILE"
if [ $? -eq 0 ]; then
echo "Container '$CONTAINER_NAME' created successfully."
else
echo "Failed to create container '$CONTAINER_NAME'."
exit 1
fi
else
echo "Container '$CONTAINER_NAME' already exists."
fi
time enroot start --rw --mount $ref_dir:/workdir/ref_dir --mount $reads_dir:/workdir/input_reads --mount $output_dir:/workdir/output_dir NVIDIA_Parabricks_2 \
pbrun minimap2 --preset map-ont --ref /workdir/ref_dir/$ref --in-fq /workdir/input_reads/$read_1 --out-bam /workdir/output_dir/out.bam --num-threads 40
Job Started
Container ‘NVIDIA_Parabricks_2’ does not exist. Creating it…
[INFO] Extracting squashfs filesystem…
Parallel unsquashfs: Using 40 processors
29669 inodes (52588 blocks) to write
created 28832 files
created 3167 directories
created 718 symlinks
created 0 devices
created 0 fifos
created 0 sockets
Container ‘NVIDIA_Parabricks_2’ created successfully.
[Parabricks Options Mesg]: Automatically generating ID prefix
[Parabricks Options Mesg]: Read group created for /workdir/input_reads/23122024_ONT_SUP_UL-
methylation_6mA_5mC-5hmC.fastq.gz
[Parabricks Options Mesg]:
@RG\tID:1530f911-2bd3-47ed-815c-4c015b\tLB:lib1\tPL:bar\tSM:sample\tPU:1530f911-2bd3-47ed-815c-4c015b
[PB Info 2025-Jan-17 09:23:45] ------------------------------------------------------------------------------
[PB Info 2025-Jan-17 09:23:45] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2025-Jan-17 09:23:45] || Version 4.4.0-1 ||
[PB Info 2025-Jan-17 09:23:45] || minimap2 ||
[PB Info 2025-Jan-17 09:23:45] ------------------------------------------------------------------------------
[PB Info 2025-Jan-17 09:23:49] Reading reference file.
[PB Info 2025-Jan-17 09:24:54] -------------------------------------
[PB Info 2025-Jan-17 09:24:54] Elapsed-Minutes Processed-Reads
[PB Info 2025-Jan-17 09:24:54] -------------------------------------
[PB Info 2025-Jan-17 09:24:54] 0.0 0
[PB Info 2025-Jan-17 09:25:00] 0.1 0
[PB Info 2025-Jan-17 09:25:06] 0.2 0
[PB Info 2025-Jan-17 09:25:12] 0.3 0
[PB Info 2025-Jan-17 09:25:18] 0.4 0
[PB Info 2025-Jan-17 09:25:24] 0.5 0
[PB Info 2025-Jan-17 09:25:30] 0.6 0
[PB Info 2025-Jan-17 09:25:36] 0.7 0
[PB Info 2025-Jan-17 09:25:42] 0.8 0
[PB Info 2025-Jan-17 09:25:48] 0.9 0
[PB Info 2025-Jan-17 09:25:54] 1.0 0
[PB Info 2025-Jan-17 09:26:00] 1.1 0
[PB Info 2025-Jan-17 09:26:06] 1.2 0
[PB Info 2025-Jan-17 09:26:12] 1.3 0
[PB Info 2025-Jan-17 09:26:18] 1.4 0
[PB Info 2025-Jan-17 09:26:24] 1.5 0
[PB Info 2025-Jan-17 09:26:30] 1.6 0
[PB Info 2025-Jan-17 09:26:36] 1.7 0
[PB Info 2025-Jan-17 09:26:42] 1.8 0
[PB Info 2025-Jan-17 09:26:48] 1.9 0
[PB Info 2025-Jan-17 09:26:54] 2.0 0
[PB Info 2025-Jan-17 09:27:00] 2.1 0
[PB Info 2025-Jan-17 09:27:06] 2.2 0
[PB Info 2025-Jan-17 09:27:12] 2.3 0
[PB Info 2025-Jan-17 09:27:18] 2.4 0
[PB Info 2025-Jan-17 09:27:24] 2.5 0
[PB Info 2025-Jan-17 09:27:30] 2.6 0
[PB Info 2025-Jan-17 09:27:36] 2.7 0
[PB Info 2025-Jan-17 09:27:42] 2.8 0
[PB Info 2025-Jan-17 09:27:48] 2.9 0
[PB Info 2025-Jan-17 09:27:54] 3.0 0
[PB Info 2025-Jan-17 09:28:00] 3.1 0
[PB Info 2025-Jan-17 09:28:06] 3.2 0
[PB Info 2025-Jan-17 09:28:12] 3.3 0
[PB Info 2025-Jan-17 09:28:18] 3.4 0
[PB Info 2025-Jan-17 09:28:24] 3.5 0
[PB Info 2025-Jan-17 09:28:30] 3.6 0
[PB Info 2025-Jan-17 09:28:36] 3.7 0
[PB Info 2025-Jan-17 09:28:42] 3.8 0
[PB Info 2025-Jan-17 09:28:48] 3.9 0
[PB Info 2025-Jan-17 09:28:54] 4.0 0
[PB Info 2025-Jan-17 09:29:00] 4.1 0
[PB Info 2025-Jan-17 09:29:06] 4.2 0
[PB Info 2025-Jan-17 09:29:12] 4.3 0
[PB Info 2025-Jan-17 09:29:18] 4.4 0
[PB Info 2025-Jan-17 09:29:24] 4.5 0
[PB Info 2025-Jan-17 09:29:30] 4.6 0
[PB Info 2025-Jan-17 09:29:36] 4.7 0
[PB Info 2025-Jan-17 09:29:42] 4.8 0
[PB Info 2025-Jan-17 09:29:48] 4.9 0
[PB Info 2025-Jan-17 09:29:54] 5.0 0
[PB Info 2025-Jan-17 09:30:00] 5.1 0
[PB Info 2025-Jan-17 09:30:06] 5.2 0
[PB Info 2025-Jan-17 09:30:12] 5.3 0
[PB Info 2025-Jan-17 09:30:18] 5.4 10000
[PB Info 2025-Jan-17 09:30:24] 5.5 10000
[PB Info 2025-Jan-17 09:30:30] 5.6 10000
[PB Info 2025-Jan-17 09:30:36] 5.7 10000
[PB Info 2025-Jan-17 09:30:42] 5.8 10000
[PB Info 2025-Jan-17 09:30:48] 5.9 10000
[PB Info 2025-Jan-17 09:30:54] 6.0 10000
[PB Info 2025-Jan-17 09:31:00] 6.1 10000
[PB Info 2025-Jan-17 09:31:06] 6.2 10000
[PB Info 2025-Jan-17 09:31:12] 6.3 10000
[PB Info 2025-Jan-17 09:31:18] 6.4 10000
[PB Info 2025-Jan-17 09:31:24] 6.5 10000
[PB Info 2025-Jan-17 09:31:30] 6.6 10000
[PB Info 2025-Jan-17 09:31:36] 6.7 10000
[PB Info 2025-Jan-17 09:31:42] 6.8 10000
[PB Info 2025-Jan-17 09:31:48] 6.9 10000
[PB Info 2025-Jan-17 09:31:54] 7.0 10000
[PB Info 2025-Jan-17 09:32:00] 7.1 10000
[PB Info 2025-Jan-17 09:32:06] 7.2 10000
[PB Info 2025-Jan-17 09:32:12] 7.3 10000
[PB Info 2025-Jan-17 09:32:18] 7.4 10000
[PB Info 2025-Jan-17 09:32:24] 7.5 10000
[PB Info 2025-Jan-17 09:32:30] 7.6 10000
[PB Info 2025-Jan-17 09:32:36] 7.7 10000
[PB Info 2025-Jan-17 09:32:42] 7.8 10000
[PB Info 2025-Jan-17 09:32:48] 7.9 10000
[PB Info 2025-Jan-17 09:32:54] 8.0 10000
[PB Info 2025-Jan-17 09:33:00] 8.1 10000
[PB Info 2025-Jan-17 09:33:06] 8.2 10000
[PB Info 2025-Jan-17 09:33:12] 8.3 10000
[PB Info 2025-Jan-17 09:33:18] 8.4 10000
[PB Info 2025-Jan-17 09:33:24] 8.5 10000
[PB Info 2025-Jan-17 09:33:30] 8.6 10000
[PB Info 2025-Jan-17 09:33:36] 8.7 10000
[PB Info 2025-Jan-17 09:33:42] 8.8 10000
[PB Info 2025-Jan-17 09:33:48] 8.9 10000
[PB Info 2025-Jan-17 09:33:54] 9.0 10000
[PB Info 2025-Jan-17 09:34:00] 9.1 10000
[PB Info 2025-Jan-17 09:34:06] 9.2 10000
[PB Info 2025-Jan-17 09:34:12] 9.3 10000
[PB Info 2025-Jan-17 09:34:18] 9.4 10000
[PB Info 2025-Jan-17 09:34:24] 9.5 10000
[PB Info 2025-Jan-17 09:34:30] 9.6 10000
[PB Info 2025-Jan-17 09:34:36] 9.7 30000
[PB Info 2025-Jan-17 09:34:42] 9.8 30000
[PB Info 2025-Jan-17 09:34:48] 9.9 30000
[PB Info 2025-Jan-17 09:34:54] 10.0 30000
[PB Info 2025-Jan-17 09:35:00] 10.1 30000
[PB Info 2025-Jan-17 09:35:06] 10.2 30000
[PB Info 2025-Jan-17 09:35:12] 10.3 30000
[PB Info 2025-Jan-17 09:35:18] 10.4 30000
[PB Info 2025-Jan-17 09:35:24] 10.5 30000
[PB Info 2025-Jan-17 09:35:30] 10.6 30000
[PB Info 2025-Jan-17 09:35:36] 10.7 30000
[PB Info 2025-Jan-17 09:35:42] 10.8 30000
[PB Info 2025-Jan-17 09:35:48] 10.9 30000
[PB Info 2025-Jan-17 09:35:54] 11.0 30000
[PB Info 2025-Jan-17 09:36:00] 11.1 30000
[PB Info 2025-Jan-17 09:36:06] 11.2 30000
[PB Info 2025-Jan-17 09:36:12] 11.3 30000
[PB Info 2025-Jan-17 09:36:18] 11.4 30000
[PB Info 2025-Jan-17 09:36:24] 11.5 30000
[PB Info 2025-Jan-17 09:36:30] 11.6 30000
[PB Info 2025-Jan-17 09:36:36] 11.7 30000
[PB Info 2025-Jan-17 09:36:42] 11.8 30000
[PB Info 2025-Jan-17 09:36:48] 11.9 30000
[PB Info 2025-Jan-17 09:36:54] 12.0 30000
[PB Info 2025-Jan-17 09:37:00] 12.1 30000
[PB Info 2025-Jan-17 09:37:06] 12.2 30000
[PB Info 2025-Jan-17 09:37:12] 12.3 30000
[PB Info 2025-Jan-17 09:37:18] 12.4 30000
[PB Info 2025-Jan-17 09:37:24] 12.5 30000
[PB Info 2025-Jan-17 09:37:30] 12.6 30000
[PB Info 2025-Jan-17 09:37:36] 12.7 30000
[PB Info 2025-Jan-17 09:37:42] 12.8 30000
[PB Info 2025-Jan-17 09:37:48] 12.9 30000
For technical support visit NVIDIA Clara - NVIDIA Docs
Exiting…
Please visit NVIDIA Clara - NVIDIA Docs for detailed documentation
Could not run minimap2
Exiting pbrun …
real 14m22.541s
user 462m3.819s
sys 11m42.314s
slurmstepd: error: Detected 1 oom_kill event in StepId=5028649.batch. Some of the step tasks have been OOM Killed.
I have tried to gradually increase the RAM memory up to 250 Gb and the number of gpus to 3. It runs for a little bit longer each time but still crashes. Maybe it’s relevant to mention that the size of my fq file is ~100Gb and the reference genome is 3.4Gb. Could you please recommend how to approach this issue? Thanks for your help!