"Could not run collectmultiplemetrics" : Record contains library that is missing from header:

container version: clara-parabricks_4.2.0-1
I followed the documentation to run collectmultiplemetrics in singularity but thats generating error due to (probably) missing library: Record contains library that is missing from header
I also noticed a warning:
WARNING: Skipping mount /var/run/nvidia-persistenced/socket [files]: /var/run/nvidia-persistenced/socket doesn’t exist in container

My understanding is the container should be self contained. How to fix this?
singularity_sif=“./clara-parabricks_4.2.0-1.sif”
singularity exec --nv ${singularity_sif} /bin/bash -c “nvidia-smi;
pbrun collectmultiplemetrics
–ref ./Homo_sapiens.GRCh38.110.dna_sm.primary_assembly.fa
–bam ./sample_two_pass_R.bam
–out-qc-metrics-dir ./QC/all-metrics/sample_two_pass_R/
–gen-all-metrics”

[PB Info 2024-Feb-28 06:32:52] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 06:32:52] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2024-Feb-28 06:32:52] || Version 4.2.0-1 ||
[PB Info 2024-Feb-28 06:32:52] || collectmultiplemetrics ||
[PB Info 2024-Feb-28 06:32:52] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 06:32:52] Using 4 GPUs, 3 BAM decompressor threads, 8 GC calculator threads
Chunk size 1000 (# of BAM records/chunk)
Recycle bin size 100 (# of reusable chunks)
Processor SPSC queue size 100 (# of chunks per processor thread)

[PB Info 2024-Feb-28 06:32:52] Loading Reference /panfs/accrepfs.vampire/nobackup/h_vmac/user/janveva1/Project/RNAseq/A4/QC/genome_index/Homo_sapiens.GRCh38.110.dna_sm.primary_assembly.fa
[PB Info 2024-Feb-28 06:33:04] Reference loaded
[PB Info 2024-Feb-28 06:33:11] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 06:33:11] Time-Stamp Elapsed-Minutes Current-Locus
[PB Info 2024-Feb-28 06:33:11] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 06:33:12] 06:33:12 0.0
[PB Info 2024-Feb-28 06:33:13] 06:33:13 0.0
terminate called after throwing an instance of ‘std::runtime_error’
what(): Record contains library that is missing from header:
[PB ESC[31mErrorESC[0m 2024-Feb-28 06:33:14][-unknown-:0] Received signal: 6
For technical support visit Clara Parabricks v4.2.0 - NVIDIA Docs, exiting.
For technical support visit Clara Parabricks v4.2.0 - NVIDIA Docs
Exiting…
Please visit NVIDIA Clara - NVIDIA Docs for detailed documentation

Could not run collectmultiplemetrics
Exiting pbrun …


detailed log:

WARNING: Skipping mount /var/run/nvidia-persistenced/socket [files]: /var/run/nvidia-persistenced/socket doesn't exist in container
Wed Feb 28 06:32:50 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A6000               Off | 00000000:31:00.0 Off |                  Off |
| 30%   26C    P8              20W / 300W |      3MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000               Off | 00000000:4B:00.0 Off |                  Off |
| 30%   26C    P8              20W / 300W |      3MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA RTX A6000               Off | 00000000:98:00.0 Off |                  Off |
| 30%   30C    P8              31W / 300W |      3MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA RTX A6000               Off | 00000000:CA:00.0 Off |                  Off |
| 30%   27C    P8              20W / 300W |      3MiB / 49140MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
[PB Info 2024-Feb-28 06:32:52] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 06:32:52] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2024-Feb-28 06:32:52] ||                              Version 4.2.0-1                             ||
[PB Info 2024-Feb-28 06:32:52] ||                          collectmultiplemetrics                          ||
[PB Info 2024-Feb-28 06:32:52] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 06:32:52] Using 4 GPUs, 3 BAM decompressor threads, 8 GC calculator threads
Chunk size 1000 (# of BAM records/chunk)
Recycle bin size 100 (# of reusable chunks)
Processor SPSC queue size 100 (# of chunks per processor thread)

[PB Info 2024-Feb-28 06:32:52] Loading Reference /panfs/accrepfs.vampire/nobackup/h_vmac/user/janveva1/Project/RNAseq/A4/QC/genome_index/Homo_sapiens.GRCh38.110.dna_sm.primary_assembly.fa
[PB Info 2024-Feb-28 06:33:04] Reference loaded
[PB Info 2024-Feb-28 06:33:11] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 06:33:11] Time-Stamp        Elapsed-Minutes        Current-Locus
[PB Info 2024-Feb-28 06:33:11] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 06:33:12] 06:33:12              0.0
[PB Info 2024-Feb-28 06:33:13] 06:33:13              0.0
terminate called after throwing an instance of 'std::runtime_error'
  what():  Record contains library that is missing from header:
[PB ESC[31mErrorESC[0m 2024-Feb-28 06:33:14][-unknown-:0] Received signal: 6
For technical support visit https://docs.nvidia.com/clara/parabricks/4.2.0/index.html, exiting.
For technical support visit https://docs.nvidia.com/clara/parabricks/4.2.0/index.html
Exiting...
Please visit https://docs.nvidia.com/clara/#parabricks for detailed documentation


Could not run collectmultiplemetrics
Exiting pbrun ...

running on different GPU node, the socket file indicated in log WARNING is present :
[janveva1@gpu0078 scripts]$ ls /var/run/nvidia-persistenced/socket
/var/run/nvidia-persistenced/socket

However, still getting the same error:
[PB Info 2024-Feb-28 07:17:52] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 07:17:52] || Parabricks accelerated Genomics Pipeline ||
[PB Info 2024-Feb-28 07:17:52] || Version 4.2.0-1 ||
[PB Info 2024-Feb-28 07:17:52] || collectmultiplemetrics ||
[PB Info 2024-Feb-28 07:17:52] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 07:17:52] Using 4 GPUs, 3 BAM decompressor threads, 8 GC calculator threads
Chunk size 1000 (# of BAM records/chunk)
Recycle bin size 100 (# of reusable chunks)
Processor SPSC queue size 100 (# of chunks per processor thread)

[PB Info 2024-Feb-28 07:17:52] Loading Reference /panfs/accrepfs.vampire/nobackup/h_vmac/user/janveva1/Project/RNAseq/A4/QC/genome_index/Homo_sapiens.GRCh38.110.dna_sm.primary_assembly.fa
[PB Info 2024-Feb-28 07:18:16] Reference loaded
[PB Info 2024-Feb-28 07:18:23] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 07:18:23] Time-Stamp Elapsed-Minutes Current-Locus
[PB Info 2024-Feb-28 07:18:23] ------------------------------------------------------------------------------
[PB Info 2024-Feb-28 07:18:24] 07:18:24 0.0
terminate called after throwing an instance of ‘std::runtime_error’
what(): Record contains library that is missing from header:
[PB ESC[31mErrorESC[0m 2024-Feb-28 07:18:25][-unknown-:0] Received signal: 6
For technical support visit Clara Parabricks v4.2.0 - NVIDIA Docs, exiting.
[PB ESC[31mErrorESC[0m 2024-Feb-28 07:18:25][-unknown-:0] Received signal: 11
For technical support visit Clara Parabricks v4.2.0 - NVIDIA Docs, exiting.
For technical support visit Clara Parabricks v4.2.0 - NVIDIA Docs
Exiting…
Please visit NVIDIA Clara - NVIDIA Docs for detailed documentation

Could not run collectmultiplemetrics
Exiting pbrun …

Tried different container versions and QC metrics but same error as above:
clara-parabricks_4.0.1-1
clara-parabricks_4.1.2-1
clara-parabricks_4.2.0-1
clara-parabricks_3.8.0-1 ( /bin/bash: line 1: pbrun: command not found )

Hi @vaibhav.a.janve,

We are not supporting Singularity very much other than having the option to convert to .sif. Are you able to run this in a Docker environment?

Yes, Its the same docker container. I have just converted the docker container to .sif as docker is not compatible to shared HPC environment due to root access requirement. None of the HPC facilities allow docker on clusters.

No, can’t run docker. I don’t have access to machine with NVIDIA GPU that’s not on cluster. In cluster environment docker use is not approved.