After flashing my device with Jetpack 4.3, I am able to set the default docker runtime to nvidia via /etc/docker/daemon.json. This is so I have access to nvidia tooling such as cicc when running docker build. Initially, this works perfectly fine. However, at some point this appears to stop working.
A simple test is the following Dockerfile:
FROM nvcr.io/nvidia/l4t-base:r32.3.1 # Make sure cuda compiler is present RUN test -e /usr/local/cuda/nvvm/bin/cicc
Initially this runs and builds an image. When the issue starts presenting, building the image fails (also the image is rebuilt instead of using the cached image, which is another indicator that the runtime has changed).
Notably, docker run still works using the correct default runtime setting.