Nvidia l4t-base missing cublas_v2.h

Robotics & Edge Computing Jetson Systems Jetson Xavier NX
1

Just pulled a fresh nvcr.io/nvidia/l4t-base r32.4.4 image and it of course comes with cuda 10.2 but cublas_v2.h is no where to be found amongst all of the other cuda lbraries:

find /usr/local/ -name *cub*    
/usr/local/cuda-10.2/targets/aarch64-linux/include/thrust/system/cuda/detail/cub
/usr/local/cuda-10.2/targets/aarch64-linux/include/thrust/system/cuda/detail/cub/cub.cuh

/usr/include# find . -name "*cub*"
<no results>

…all return nothing even though I have every other cuda header

I tried with r32.5.0 as well and same results:

find /usr/local/ -name *cub* 
/usr/local/cuda-10.2/targets/aarch64-linux/include/thrust/system/cuda/detail/cub
/usr/local/cuda-10.2/targets/aarch64-linux/include/thrust/system/cuda/detail/cub/cub.cuh
find /usr/include -name "*cub*"

That library is needed for many things, why is it missing and what am I supposed to do? Reinstall cuda? Seems to defeat the purpose of the L4T image already doing that for me.

This is for running an L4T container using docker nvidia on my Jetson Xavier NX. The Jetson Xavier NX has these headers in /usr/include. I can copy them from the host into my container but that is hacky so I was hoping to know why L4T doesn’t include them.

1 Like
3

Hi,

We test the l4t-base:r32.5.0 container, and find the cublas header in the /usr/include/.

# find / -iname *cublas*
/usr/include/cublasXt.h
/usr/include/cublas_api.h
/usr/include/cublas.h
/usr/include/cublas_v2.h
/usr/include/cublasLt.h
/usr/lib/aarch64-linux-gnu/libcublas.so.10.2.2.89
/usr/lib/aarch64-linux-gnu/libcublasLt.so
/usr/lib/aarch64-linux-gnu/libcublasLt.so.10.2.2.89
/usr/lib/aarch64-linux-gnu/libcublasLt.so.10
/usr/lib/aarch64-linux-gnu/libcublas.so.10
/usr/lib/aarch64-linux-gnu/libcublas.so
...

Could you check it again?
Thanks.

4

@AastaLLL On my Jetson Xavier NX here are the containers:

nx@nx-fusion:$ docker image ls
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
nvcr.io/nvidia/l4t-base      r32.5.0             d9141f00a2c3        4 months ago        635MB
nvcr.io/nvidia/l4t-base      r32.4.4             10faffedd5fa        5 months ago        634MB

I then start l4t-base 32.5 and there are no cublas headers:

docker run -it d9141f00a2c3 /bin/bash
root@b2ccbe9429ac:/# find /usr -name "*cub*"
/usr/share/mime/application/x-gamecube-rom.xml
/usr/local/cuda-10.2/targets/aarch64-linux/include/thrust/system/cuda/detail/cub
/usr/local/cuda-10.2/targets/aarch64-linux/include/thrust/system/cuda/detail/cub/cub.cuh

The only way to get the headers is by running the container with nvidia runtime only after a build

My problem is I need those headers during a build in order to make darknet, which requires them to build.

Example running a docker build with runtime: nvidia to find that cublas is missing:

Step 14/14 : RUN echo "$(find /usr -name *cub*)"
 ---> Running in 330a3155e721
/usr/share/mime/application/x-gamecube-rom.xml
/usr/share/sip/PyQt5/QtQml/qqmlincubator.sip
/usr/share/cmake-3.10/Modules/FindCUDA/parse_cubin.cmake
/usr/include/aarch64-linux-gnu/qt5/QtQml/QQmlIncubator
/usr/include/aarch64-linux-gnu/qt5/QtQml/qqmlincubator.h
/usr/include/aarch64-linux-gnu/qt5/QtQml/QQmlIncubationController
/usr/include/armadillo_bits/operator_cube_plus.hpp
/usr/include/armadillo_bits/subview_cube_bones.hpp
/usr/include/armadillo_bits/subview_cube_meat.hpp
/usr/include/armadillo_bits/operator_cube_div.hpp
/usr/include/armadillo_bits/operator_cube_times.hpp
/usr/include/armadillo_bits/subview_cube_each_meat.hpp
/usr/include/armadillo_bits/operator_cube_minus.hpp
/usr/include/armadillo_bits/operator_cube_relational.hpp
/usr/include/armadillo_bits/unwrap_cube.hpp
/usr/include/armadillo_bits/subview_cube_each_bones.hpp
/usr/include/armadillo_bits/operator_cube_schur.hpp
/usr/include/simbody/simmath/internal/Geo_BicubicBezierPatch.h
/usr/include/simbody/simmath/internal/Geo_BicubicHermitePatch.h
/usr/include/simbody/simmath/internal/BicubicSurface.h
/usr/include/boost/math/interpolators/detail/cubic_b_spline_detail.hpp
/usr/include/boost/math/interpolators/cubic_b_spline.hpp
/usr/include/pcl-1.8/pcl/surface/marching_cubes.h
/usr/include/pcl-1.8/pcl/surface/marching_cubes_rbf.h
/usr/include/pcl-1.8/pcl/surface/impl/marching_cubes.hpp
/usr/include/pcl-1.8/pcl/surface/impl/marching_cubes_rbf.hpp
/usr/include/pcl-1.8/pcl/surface/impl/marching_cubes_hoppe.hpp
/usr/include/pcl-1.8/pcl/surface/3rdparty/poisson4/marching_cubes_poisson.h
/usr/include/pcl-1.8/pcl/surface/marching_cubes_hoppe.h
/usr/local/cuda-10.2/targets/aarch64-linux/include/thrust/system/cuda/detail/cub
/usr/local/cuda-10.2/targets/aarch64-linux/include/thrust/system/cuda/detail/cub/cub.cuh
5

Hi @ec2020, try setting your default docker-runtime to nvidia and reboot: https://github.com/dusty-nv/jetson-containers#docker-default-runtime

You need to run the container with --runtime nvidia to get the CUDA headers/libraries mounted into the container, except that docker build doesn’t support the runtime option. So set it as the default in your docker daemon.json and it will be used during build time.

I also verified that cublas headers are under /usr/include when I run l4t-base:r32.5.0 with --runtime nvidia. If you still don’t see the headers, please confirm that they are on your device (outside of container)

6

@dusty_nv good catch, and I believe you can close this issue too, as with the solution you proposed I am able to build darknet just fine with no additional modifications: