You can set the default docker runtime to nvidia, and then CUDA/cuDNN/VisionWorks/ect will be available to you during docker build operations. See here: GitHub - dusty-nv/jetson-containers: Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
This should enable OpenCV to find the cuDNN library it is looking for in your Dockerfile. You shouldn’t need to copy CUDA into your container or modify the CSV files, just change the docker daemon’s default-runtime. Users of your container won’t need to change their default runtime, they will just need to run the container with --runtime nvidia. The daemon.json change is needed because you can’t specify the --runtime argument to docker build.