Cannot pull deepstream image with rootless podman

• Hardware Platform: dGPU
• DeepStream Version: 6.2-triton
• Issue Type: bugs
• How to reproduce the issue:

Using a default setup of rootless podman (user namespace size 65535, as specified by /etc/subuid), execute the following command as a non-root user:

podman pull nvcr.io/nvidia/deepstream:6.2-triton

It will eventually fail with the following error:

Error: writing blob: adding layer with blob “sha256:1f38867ae61659d2394ce2253fde6b112fd97f41ea5737405d1a1d1587767271”: processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 71873:0 for /opt/dev_ucx/MLNX_OFED_LINUX-5.7-1.0.2.0-ubuntu20.04-x86_64/DEBS/neohost-backend_1.5.0-102_amd64.deb): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /opt/dev_ucx/MLNX_OFED_LINUX-5.7-1.0.2.0-ubuntu20.04-x86_64/DEBS/neohost-backend_1.5.0-102_amd64.deb: invalid argument): exit status 1

In the layer with ID f1313b11cc, a tgz file MLNX_OFED_LINUX-5.7-1.0.2.0-ubuntu20.04-x86_64.tgz is downloaded and extracted. This tar file contains two files with owner=71873; neohost-backend_1.5.0-102_amd64.deb and neohost-sdk_1.5.0-102_amd64.deb. Even though both files are removed in a later layer, 8bb8243025, rootless podman does not allow combining the layers because the owner uid of the two files is outside the maximum size of the user namespace.

This problem can be amended by increasing the maximum allowed user namespace in /etc/subuid. However, in my case this is not a viable option as I do not have root privileges on the server.

• Proposed resolution:
When unpacking MLNX_OFED_LINUX-5.7-1.0.2.0-ubuntu20.04-x86_64.tgz, add the flag --no-same-owner to the tar command, to extract all files as the root user.

Are you able to run this docker image after pull it in this way? Sorry but we didn’t verify podman in this way, assume you can manage all the dependencies.

In case you need compare the behavior in docker, below are the links for reference:

No, the pull fails and I cannot run the image. The same thing happens if I try to run the image without pulling first.

If I run podman as root or docker, it works as expected. Unfortunately, I am forced to use rootless podman in build and production environments.

Can you check whether this document will help you? Installation Guide — NVIDIA Cloud Native Technologies documentation

Unfortunately, I did not find anything in the document you linked to, that helps me with this particular problem.

The problem occurs only with that particular image: deepstream:6.2-triton. I do not have a problem with pulling or running other NVIDIA containers, and can utilize the GPU when doing so. For example, I can run the following command without issues:

podman run --rm nvcr.io/nvidia/deepstream:6.2-base nvidia-smi

As I described in my original post, the problem with deepstream:6.2-triton is that it contains layers where a few files have abnormal owner IDs. This could be easily fixed by rebuilding the image in a way such that the files in question get normal ownership. I proposed one way of doing that in the original post. However, this has to be done when building the original image on your side.

Sorry for late response. The dockerfiles are open source. You can modify them to remove unecessary layers and generate the suitable docker for you.

Please get the dockerfiles from NVIDIA-AI-IOT/deepstream_dockers at dev/ds/ds6.2 (github.com), the branch for DeepStream 6.2 GA containers is “dev/ds/ds6.2”.

That is great, I did not know that. Thanks!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.