Collectmutlplemetrics error at pdf generation step

[ranjit58]

I am trying to run collectmutlplemetrics in aws omics environment (using wdl file). Its almost completes but seems stuck at one of the qc score pdf generation. For 10X WGS it might takes 30 min and finishes but on 100X it remains stuck even after 12 hrs.
I tried to run it directly on the cluster (bypassing aws omics ) and it seems to work fine, which points to aws omics issue, but still thought to report here for any suggestion.

command
pbrun collectmultiplemetrics
–ref ~{ref}
–bam ~{input_bam}
–gen-all-metrics
–out-qc-metrics-dir ~{out_QC_dir}

output
[PB Info 2023-Sep-27 00:20:48] ------------------------------------------------------------------------------
[PB Info 2023-Sep-27 00:20:48] || Program: collectmultiplemetrics ||
[PB Info 2023-Sep-27 00:20:48] || Version: 4.0.0-1 ||
[PB Info 2023-Sep-27 00:20:48] || Start Time: Wed Sep 27 00:18:41 2023 ||
[PB Info 2023-Sep-27 00:20:48] || End Time: Wed Sep 27 00:20:48 2023 ||
[PB Info 2023-Sep-27 00:20:48] || Total Time: 2 minutes 7 seconds ||
[PB Info 2023-Sep-27 00:20:48] ------------------------------------------------------------------------------
/tmp/PC612AXC_run.sh
Generating qualityscore pdf…

[gburnett]

Hello @ranjit58,

When you ran this on Omics, you used a private workflow or one of the pre-configured Ready2Run workflows?

[ranjit58]

This is on private workflow.

below are some addition details of wdl file

task generate_QC_metrics {
input {
File input_bam
File RefTar_file
String out_QC_dir = “output_qc”
String docker_parabricks
String sample_name
}

String ref = basename(RefTar_file, “.tar”)

command <<<
set -e
set -x
set -o pipefail
mkdir ~{out_QC_dir}
tar xf ~{RefTar_file}
nvidia-smi
pbrun collectmultiplemetrics
–ref ~{ref}
–bam ~{input_bam}
–gen-all-metrics
–out-qc-metrics-dir ~{out_QC_dir}

runtime {
docker: docker_parabricks
acceleratorCount: 4
cpu : 32
memory : “180 GB”
disks: “local-disk 500 SSD”
acceleratorType: “nvidia-tesla-t4”
}

output {
File qc_mean_quality_by_cycle_pdf = “~{out_QC_dir}/mean_quality_by_cycle.pdf”
File qc_mean_quality_by_cycle_png = “~{out_QC_dir}/mean_quality_by_cycle.png”
File qc_mean_quality_by_cycle_txt = “~{out_QC_dir}/mean_quality_by_cycle.txt”
}
}

[ranjit58]

I was able to replicate the issue outside aws omics environment, directly on a g4dn gpu from ec2.

I capture the screenshots below (the program was stuck for hours at this stage)

[ranjit58]

another image from command top