Building Kaldi in Docker on Jetson Nano

Hi,

I’m curious if there’s a way to build a custom Kaldi version in Docker with GPU support on Jetson Nano?
I saw this image: nvcr.io/nvidia/kaldi:20.08-py3, but it’s for linux/amd64 only.
When I tried to build from nvcr.io/nvidia/l4t-base:r32.4.3, the build has failed with the following log:

make[2]: Entering directory '/opt/kaldi/src/tree'
rm -f .depend.mk
g++ -M -std=c++11 -I.. -isystem /opt/kaldi/tools/openfst-1.7.7/include -O3 -march=armv8-a+crypto  -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_OPENBLAS -I/opt/kaldi/tools/OpenBLAS/install/include -ftree-vectorize -pthread -g -fPIC -DHAVE_CUDA -I/usr/local/cuda/include -fPIC -pthread -isystem /opt/kaldi/tools/openfst-1.7.7/include cluster-utils.cc build-tree.cc build-tree-utils-test.cc cluster-utils-test.cc build-tree-questions.cc build-tree-utils.cc event-map-test.cc context-dep-test.cc clusterable-classes.cc build-tree-test.cc tree-renderer.cc event-map.cc context-dep.cc >> .depend.mk
In file included from ../cudamatrix/cu-matrix.h:35:0,
                 from ../cudafeat/feature-online-cmvn-cuda.h:21,
                 from ../cudafeat/feature-online-batched-cmvn-cuda.h:20,
                 from feature-online-batched-cmvn-cuda.cc:18:
../cudamatrix/cu-common.h:33:10: fatal error: cublas_v2.h: No such file or directory
 #include <cublas_v2.h>
          ^~~~~~~~~~~~~
compilation terminated.
make[2]: Leaving directory '/opt/kaldi/src/util'
make -C gmm/ depend
make[2]: Entering directory '/opt/kaldi/src/gmm'
rm -f .depend.mk
g++ -M -std=c++11 -I.. -isystem /opt/kaldi/tools/openfst-1.7.7/include -O3 -march=armv8-a+crypto  -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_OPENBLAS -I/opt/kaldi/tools/OpenBLAS/install/include -ftree-vectorize -pthread -g -fPIC -DHAVE_CUDA -I/usr/local/cuda/include -fPIC -pthread -isystem /opt/kaldi/tools/openfst-1.7.7/include mle-diag-gmm.cc mle-full-gmm.cc model-test-common.cc mle-am-diag-gmm.cc diag-gmm-test.cc full-gmm-test.cc diag-gmm-normal.cc decodable-am-diag-gmm.cc ebw-diag-gmm-test.cc model-common.cc full-gmm-normal.cc diag-gmm.cc indirect-diff-diag-gmm.cc am-diag-gmm-test.cc full-gmm.cc mle-diag-gmm-test.cc ebw-diag-gmm.cc mle-am-diag-gmm-test.cc mle-full-gmm-test.cc am-diag-gmm.cc >> .depend.mk
In file included from ../cudamatrix/cu-matrix.h:35:0,
                 from ../cudafeat/feature-window-cuda.h:21,
                 from ../cudafeat/feature-spectral-cuda.h:25,
                 from ../cudafeat/online-cuda-feature-pipeline.h:28,
                 from online-cuda-feature-pipeline.cc:20:
../cudamatrix/cu-common.h:33:10: fatal error: cublas_v2.h: No such file or directory
 #include <cublas_v2.h>
          ^~~~~~~~~~~~~
compilation terminated.
In file included from ../cudamatrix/cu-matrix.h:35:0,
                 from ../cudafeat/feature-online-cmvn-cuda.h:21,
                 from ../cudafeat/feature-online-batched-cmvn-cuda.h:20,
                 from ../cudafeat/feature-online-batched-ivector-cuda.h:21,
                 from feature-online-batched-ivector-cuda.cc:18:
../cudamatrix/cu-common.h:33:10: fatal error: cublas_v2.h: No such file or directory
 #include <cublas_v2.h>
          ^~~~~~~~~~~~~
compilation terminated.
In file included from ../cudamatrix/cu-matrix.h:35:0,
                 from ../cudafeat/feature-window-cuda.h:21,
                 from ../cudafeat/feature-spectral-cuda.h:25,
                 from ../cudafeat/feature-online-batched-spectral-cuda.h:25,
                 from feature-online-batched-spectral-cuda.cc:18:
../cudamatrix/cu-common.h:33:10: fatal error: cublas_v2.h: No such file or directory
 #include <cublas_v2.h>
          ^~~~~~~~~~~~~
compilation terminated.
make[2]: Leaving directory '/opt/kaldi/src/feat'
make -C transform/ depend
make[2]: Entering directory '/opt/kaldi/src/transform'
rm -f .depend.mk
g++ -M -std=c++11 -I.. -isystem /opt/kaldi/tools/openfst-1.7.7/include -O3 -march=armv8-a+crypto  -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_OPENBLAS -I/opt/kaldi/tools/OpenBLAS/install/include -ftree-vectorize -pthread -g -fPIC -DHAVE_CUDA -I/usr/local/cuda/include -fPIC -pthread -isystem /opt/kaldi/tools/openfst-1.7.7/include lda-estimate-test.cc transform-common.cc regression-tree-test.cc compressed-transform-stats.cc mllt.cc fmllr-diag-gmm-test.cc decodable-am-diag-gmm-regtree.cc regtree-fmllr-diag-gmm.cc fmpe.cc lda-estimate.cc lvtln.cc regression-tree.cc regtree-mllr-diag-gmm-test.cc fmllr-diag-gmm.cc fmllr-raw.cc cmvn.cc fmllr-raw-test.cc regtree-mllr-diag-gmm.cc basis-fmllr-diag-gmm.cc regtree-fmllr-diag-gmm-test.cc fmpe-test.cc >> .depend.mk
In file included from ../cudamatrix/cu-matrix.h:35:0,
                 from ../cudafeat/feature-online-cmvn-cuda.h:21,
                 from ../cudafeat/feature-online-batched-cmvn-cuda.h:20,
                 from ../cudafeat/online-batched-feature-pipeline-cuda.h:33,
                 from online-batched-feature-pipeline-cuda.cc:19:
../cudamatrix/cu-common.h:33:10: fatal error: cublas_v2.h: No such file or directory
 #include <cublas_v2.h>
          ^~~~~~~~~~~~~
compilation terminated.
In file included from ../cudamatrix/cu-matrix.h:35:0,
                 from ../cudafeat/feature-online-cmvn-cuda.h:21,
                 from online-ivector-feature-cuda.cc:26:
../cudamatrix/cu-common.h:33:10: fatal error: cublas_v2.h: No such file or directory
 #include <cublas_v2.h>
          ^~~~~~~~~~~~~
compilation terminated.
../makefiles/default_rules.mk:149: recipe for target 'depend' failed
make[2]: *** [depend] Error 1
make[2]: Leaving directory '/opt/kaldi/src/cudafeat'
Makefile:127: recipe for target 'cudafeat/depend' failed
make[1]: *** [cudafeat/depend] Error 2
make[1]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/opt/kaldi/src/tree'
make[2]: Leaving directory '/opt/kaldi/src/gmm'
make[2]: Leaving directory '/opt/kaldi/src/transform'
make[1]: Leaving directory '/opt/kaldi/src'
Makefile:122: recipe for target 'base/.depend.mk' failed
make: *** [base/.depend.mk] Error 2

Note that I was able to build the same source code w/o Docker on Jetson Nano with GPU support. But it’s a bit painful process. That’s why I’m looking into Docker option to make it simpler for everyone. Any thoughts or suggestions would be greatly appreciated.

1 Like

Got it working by adding a default nvidia runtime, as described here.