Apptainer pull failed when Copying blob 91b70b555b09 of Morpheus 23.07 docker image from NGC

I could pull the image from my local laptops successfully: either an Unbutu 22.04 Linux or a Windows 10 WSL2 Ubuntu 22.04 using Docker -
$ docker pull nvcr.io/nvidia/morpheus/morpheus:23.07-runtime

I cannot succeed on pulling morpheus:v23.07.03-runtime Docker image from NGC to build Apptainer morpheus2307_03-runtime.sif image.
The blob 91b70b555b09 could not be copied/downloaded; but, all other blobs did not have that error.
See the attached screenshot below:

$ apptainer pull morpheus2307_03-runtime.sif docker://nvcr.io/nvidia/morpheus/morpheus:v23.07.03-runtime
INFO: Converting OCI blobs to SIF format
INFO: Starting build…
Getting image source signatures
Copying blob d963a42bc712 skipped: already exists
Copying blob 8a8079358f46 skipped: already exists
Copying blob afa3f70b397f skipped: already exists
Copying blob 4586c00479c6 skipped: already exists
Copying blob 4304fa233a80 skipped: already exists
Copying blob 91b70b555b09 [=================>--------------------] 31.2MiB / 64.7MiB
Copying blob 3f94e4e483ea skipped: already exists
Copying blob a720e434644a skipped: already exists
Copying blob 31704e1c9acb skipped: already exists
Copying blob 09e7660654fc skipped: already exists
Copying blob 06b116b3de2d skipped: already exists
Copying blob e648e292bae4 skipped: already exists
Copying blob aa49ad844e63 skipped: already exists
Copying blob eace2402c7ab skipped: already exists
Copying blob fcf2b98145ab skipped: already exists
Copying blob e4feec9d56e4 skipped: already exists
Copying blob 666477b07156 skipped: already exists
Copying blob 081c1ca15f13 skipped: already exists
Copying blob c8b08c95327d skipped: already exists
Copying blob 1c3c4d61fc5e skipped: already exists
Copying blob 4c6d1068d4bd skipped: already exists
Copying blob b671e7b6e9ce skipped: already exists
Copying blob 4f706e8fc2e5 skipped: already exists
Copying blob 342746cebecb skipped: already exists
FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: initializing source oci:/home/cjliu/.apptainer/cache/blob:12da97f91fe45d6ce01789ec0bea6a98101332b54d41b8981e4e4e82d27a55c5: writing blob: happened during read: read tcp 10.12.14.247:53092->18.67.76.24:443: read: connection reset by peer
$

How can this issue be resolved?
Thanks.

Dr. CJ Liu

I am able to pull it using apptainer 1.2.5

There is a blob cache file created by default in $HOME/.local/share/containers/cache/blob-info-cache-v1.boltdb

Perhaps trying removing that and retrying. It has possibly become corrupted.

\Pete

Hi, Peter,
Thanks.
Have just tried that. But the problem remains.
In addition to deleting the blob cache file you specified, I also tried removing the whole cache subfolder and then pulling again. It failed the same way.
V/R
CJ

I’m sorry, but I don’t have any other suggestions.

tcp 10.12.14.247:53092->18.67.76.24:443: read: connection reset by peer

Seems like a network issue there.

Hi, Peter,
Problem resolved. I have successfully pulled the Morpheus 23.07 Docker image and converted it into its apptainer counterpart.
It failed when I was pulling the image on a login node without GPU.
If I ssh’ed into a compute node with GPUs and tried again, then I could pull the image successfully.
Please feel free to try, test, and verify.
Thanks for your assistance looking into this issue.
V/R
CJ Liu, D.Sc.

1 Like