Cross compiling zed ros2 wrapper for xavier?

I have been trying to cross compile zed ros2(foxy wrapper).

For Cross compiling I have been using the following docker image on a x86 ubuntu Host machine.
dustynv/ros:foxy-pytorch-l4t-r32.6.1
How ever there are no cuda libs in /usr/local/cuda/lib64 and this is causing zed_components to fail compiling.
But on the other hand on xavier when I run this docker image there are cuda libs in /usr/local/cuda/lib64.
How do I set up cross compiling for xavier ?