Error trying to build Triton Inference Server container using Singularity and Docker bootstrap setting

I am using apptainer (a.k.a singularity) containerization and docker bootstrap setting to pull the 22.08 version of the Triton Inference server. We currently have the 18.04 version. It starts the build and gets to the last blob and I receive the following message.

FATAL: While performing build: conveyor failed to get: initializing source oci:/root/.apptainer/cache/blob:6daea0f76f1f452ff40e58d15d49236300cda2281bafa4e48712a14649518d9b: copying system image from manifest list: writing blob: happened during read: unexpected EOF.

I downloaded the latest Rapids container and tried to build it. It got past the blobs, but errored out here:

INFO: Creating SIF file…
FATAL: While performing build: while creating squashfs: create command failed: exit status 1:
Write failed because No space left on device

We point to a another directory using symbolic links for both the previous install of Singularity and the latest for the Apptainer.

lrwxrwxrwx 1 root root 21 Aug 4 07:36 .apptainer → /data/root/.apptainer
lrwxrwxrwx 1 root root 23 Aug 4 07:38 .singularity → /data/root/.singularity

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGdata-LGdata 26T 1.3T 25T 6% /data

I have also tried to pull the Triton Inference Server container to another server with the same result.

FATAL: While performing build: conveyor failed to get: initializing source oci:/root/.apptainer/cache/blob:6daea0f76f1f452ff40e58d15d49236300cda2281bafa4e48712a14649518d9b: copying system image from manifest list: writing blob: happened during read: unexpected EOF

I was able to get build the Rapids container after setting the Apptainer $APPTAINER_TMPDIR directory to drive with plenty of space. Also tried setting the $APPTAINER_CACHEDIR, but that didn’t help the Triton build.