Thanks for your reply.
I found the answer here: Getting linux_for_tegra into a container on balena-os - #20 by acostach - balenaOS - balenaForums
Basically nvidia-l4t-core is meant to be installed on a physical device - hence the complaint when doing it in a docker build. But there was a workaround.