Docker Buildkit and nvidia-docker

[unphased]

I am trying to set up accelerated builds using ccache. This is done using the cache mount feature provided by docker buildkit.

However, buildkit does not support the nvidia runtime.

I have found through searching that it is possible to use stub libraries e.g. found here: /usr/local/cuda/lib64/stubs to allow software to build against cuda and other dependencies in a runc runtime docker build (as buildkit requires this) and then make it work somehow at runtime.

buildkit loose the binding of the nvidia driver libraries available on the host

Is this possible? Can someone point out for me to how to install the stub libraries and how to approach this?

So far this seems potentially possible and it is tantalizing, but I think that I will need to resort to this solution in the meantime:

Incremental Docker builds using ccache

[AastaLLL]

Hi,

Have you tried the container built for JetPack 5?

We have included all the libraries into the container from JetPack.
So you don’t need to mount it from the native.

Thanks.

[unphased]

Thanks, that should prove to be helpful. i think we are not able to use that right now though since we’re stuck on 4.4.1 due to kernel device tree compatibility issues. But I will be able to test it at least to evaluate.

Are you saying that the nvidia docker runtime (since this is incompatible with docker buildkit) is no longer required to run docker builds against cuda libs with the latest jetpack 5?

[AastaLLL]

Hi,

For JetPack 5, we directly include the libraries in the containers rather than mounting.
Thanks.