In the end it is as easy as not using nvidia-docker2 and just exposing these devices to the container:
Ansible:
- docker_container:
devices:
- /dev/nvhost-ctrl
- /dev/nvhost-ctrl-gpu
- /dev/nvhost-prof-gpu
- /dev/nvmap
- /dev/nvhost-gpu
- /dev/nvhost-as-gpu
- /dev/nvhost-vic
- /dev/tegra_dc_ctrl