L4T-tensorrt buildx missings libs

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Hi I wish to use buildx to create a custom image based off nvcr.io/nvidia/l4t-tensorrt:r8.5.2.2-devel. Since buildx does not support nvidia runtime, when I compile my application I get linker errors. What is the point of docker if we are relying on libraries to be mounted from the host? How can I compile my application while using buildx and not using nvidia runtime?

333.6 /usr/bin/ld: warning: libnvdla_compiler.so, needed by /usr/lib/aarch64-linux-gnu/libnvinfer.so, not found (try using -rpath or -rpath-link) 333.6 /usr/bin/ld: warning: libnvdla_runtime.so, needed by /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1, not found (try using -rpath or -rpath-link) 333.6 /usr/bin/ld: warning: libcuda.so.1, needed by /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1, not found (try using -rpath or -rpath-link) 333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::IProfile::setUseSoftMaxOptz(bool)’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to NvRmMemMap' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to nvdla::createSyncStrideSemaphore(NvDlaSemaphoreRec const*, unsigned int)’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuGetExportTable' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to nvdla::IRuntime::bindSubmitEvent(int, NvDlaSyncEventType, nvdla::ISync*, int*)’
333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::INetwork::addReshape(nvdla::ITensor*, nvdla::Dims4)' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuDeviceGet’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuMemHostGetDevicePointer_v2' 333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::INetwork::addSlice(nvdla::ITensor*, nvdla::Weights, nvdla::Weights, nvdla::Weights, nvdla::Weights, nvdla::SliceLayerMode)’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuMemFreeHost' 333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::destroyNetwork(nvdla::INetwork*)’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to nvdla::IRuntime::submit(bool, bool, unsigned int, unsigned int, nvdla::ISync**)' 333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::createWisdom()’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to NvRmMemUnmap' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to NvRmMemHandleFree’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to nvdla::createSyncSemaphore(NvDlaSemaphoreRec const*)' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuStreamIsCapturing’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuStreamWriteValue32' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuDevicePrimaryCtxRetain’
333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::IProfile::setCanCompressStructuredSparseWeights(bool)' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to nvdla::destroySync(nvdla::ISync*)’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuCtxPushCurrent_v2' 333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::IPoolingLayer::setPoolingPaddingInclusionType(nvdla::PoolingPaddingInclusionType)’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to nvdla::destroyRuntime(nvdla::IRuntime*)' 333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::destroyWisdom(nvdla::IWisdom*)’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to nvdla::createRuntime()' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuCtxPopCurrent_v2’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuDeviceGetAttribute' 333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::INetwork::addTranspose(nvdla::ITensor*, nvdla::Dims4)’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to nvdla::createSyncSyncpoint(NvDlaFenceRec const*)' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuStreamWaitValue32’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuCtxGetDevice' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuGetErrorName’
333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::createNetwork()' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to NvRmMemHandleAllocAttr’
333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuMemHostAlloc' 333.6 /usr/bin/ld: /usr/local/cuda-11.4/targets/aarch64-linux/lib/libcudla.so.1: undefined reference to cuDevicePrimaryCtxRelease_v2’
333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::ISoftMaxLayer::setAxis(int)' 333.6 /usr/bin/ld: /usr/lib/aarch64-linux-gnu/libnvinfer.so: undefined reference to nvdla::INetwork::addResize(nvdla::ITensor*, nvdla::ResizeMode, nvdla::Weights)’
333.6 collect2: error: ld returned 1 exit status`

2

Hi @rbelshaw, on the newer JetPack 6.0 DP release with TensorRT 8.6, there are fewer mounted libraries from the host (actually this has been a longer-term process we have been migrating from since JetPack 4), however there are still some lower-level device drivers that get mounted in (you can find these listed in the CSV files under /etc/nvidia-container-runtime/host-files-for-container.d).

Agree with your sentiments, but believe that CUDA device drivers get dynamically mounted in even on x86/dGPU (although there is not that DLA dependency on those systems). Actually you can find another topic about this issue on x86 too (unrelated to Jetson):

For the aforementioned reasons (and also because I typically use the GPU or invoke the CUDA runtime in some way during docker builds), I build with DOCKER_BUILDKIT=0

4

Thanks for the quick response @dusty_nv, I thought this might be the case. Unfortunately i am currently tied to ros noetic which means I am stuck with Ubuntu 20.04 and hence I cannot use JP 6. I noticed some cross compilation images when I trying to solve this problem. Admittedly we were hoping to cross compile the l4t images on an x86 machine with a multistage build. These images are very large and not suited to our application however would they solve the issue we’re having above?

If so do you think it would be possible to copy binaries built in this cross compilation image to a runtime image?

5

I’m not personally experience with cross-compiling, as I believe it tends to require support of each project/dependency in the build chain (and I have lots of those), however there is a cross-compilation container on NGC for JetPack that you might want to try: