Hi Cliff,
So what I’ve done so far is pretty interesting, haha!
I’ve uninstalled CUDA 10.0, installed CUDA 9.0, pointed cuda folder → cuda-9.0. I’ve also uninstalled tensorflow1.13, installed tensorflow 1.12.
Thus far, tensorflow can run “hello world”, however, it can’t run the convolution functions (complains that probably cudNN not initialized).
So I’ve gone and downloaded cudNN7.5.0, stored them into the cuda-9.0 folder. But it seems that tensorflow still complains of the cudNN.h file.
As for the environment variables, I’m going to post what I see now, from a CUDA9.0, tensorflow1.12 perspective.
{‘BASH_ENV’: ‘/etc/bash.bashrc’,
‘BAZELRC’: ‘/root/.bazelrc’,
‘CLICOLOR’: ‘1’,
‘CUBLAS_VERSION’: ‘10.0.130’,
‘CUDA_CACHE_DISABLE’: ‘1’,
‘CUDA_DRIVER_VERSION’: ‘410.48’,
‘CUDA_TOOLKIT_PATH’: ‘/usr/local/cuda’,
‘CUDA_VERSION’: ‘10.0.130’,
‘CUDNN_INSTALL_PATH’: ‘/usr/lib/x86_64-linux-gnu’,
‘CUDNN_VERSION’: ‘7.4.2.24’,
‘ENV’: ‘/etc/shinit’,
‘GIT_PAGER’: ‘cat’,
‘HOME’: ‘/root’,
‘HOROVOD_GPU_ALLREDUCE’: ‘NCCL’,
‘HOROVOD_NCCL_INCLUDE’: ‘/usr/include’,
‘HOROVOD_NCCL_LIB’: ‘/usr/lib/x86_64-linux-gnu’,
‘HOROVOD_NCCL_LINK’: ‘SHARED’,
‘HOROVOD_WITHOUT_PYTORCH’: ‘1’,
‘HOSTNAME’: ‘50105e43ccca’,
‘JPY_PARENT_PID’: ‘1222’,
‘LC_ALL’: ‘C.UTF-8’,
‘LESSCLOSE’: ‘/usr/bin/lesspipe %s %s’,
‘LESSOPEN’: ‘| /usr/bin/lesspipe %s’,
‘LIBRARY_PATH’: ‘/usr/local/cuda/lib64/stubs:’,
‘LS_COLORS’: ‘rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:.xspf=00;36:’,
‘MOFED_VERSION’: ‘4.4-1.0.0’,
‘MPLBACKEND’: ‘module://ipykernel.pylab.backend_inline’,
‘NCCL_HDR_PATH’: ‘/usr/include’,
‘NCCL_INSTALL_PATH’: ‘/usr/lib/x86_64-linux-gnu’,
‘NCCL_VERSION’: ‘2.3.7’,
‘NVIDIA_BUILD_ID’: ‘5618942’,
‘NVIDIA_DRIVER_CAPABILITIES’: ‘compute,utility,video’,
‘NVIDIA_REQUIRE_CUDA’: ‘cuda>=9.0’,
‘NVIDIA_TENSORFLOW_VERSION’: ‘19.02’,
‘NVIDIA_VISIBLE_DEVICES’: ‘all’,
‘OLDPWD’: ‘/home/workspace/cs6216_project’,
‘OMPI_MCA_btl_vader_single_copy_mechanism’: ‘none’,
‘OPENMPI_VERSION’: ‘3.1.3’,
‘PAGER’: ‘cat’,
‘PATH’: ‘/usr/local/bin:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin’,
‘PWD’: ‘/home/workspace’,
‘PYTHONIOENCODING’: ‘utf-8’,
‘SHELL’: ‘bash’,
‘SHLVL’: ‘2’,
‘STY’: ‘81.pts-1.50105e43ccca’,
‘TENSORFLOW_VERSION’: ‘v1.13.0-rc0’,
‘TERM’: ‘xterm-color’,
‘TERMCAP’: ‘SC|screen|VT 100/ANSI X3.64 virtual terminal:\\n\t:DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\\n\t:cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\\n\t:do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\\n\t:le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\\n\t:li#24:co#80:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\\n\t:cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\\n\t:im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:\\n\t:ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\\n\t:ti=\E[?1049h:te=\E[?1049l:us=\E[4m:ue=\E[24m:so=\E[3m:\\n\t:se=\E[23m:mb=\E[5m:md=\E[1m:mh=\E[2m:mr=\E[7m:\\n\t:me=\E[m:ms:\\n\t:Co#8:pa#64:AF=\E[3%dm:AB=\E[4%dm:op=\E[39;49m:AX:\\n\t:vb=\Eg:G0:as=\E(0:ae=\E(B:\\n\t:ac=\140\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~…–++,hhII00:\\n\t:po=\E[5i:pf=\E[4i:Km=\E[M:k0=\E[10~:k1=\EOP:k2=\EOQ:\\n\t:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:k7=\E[18~:\\n\t:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:\\n\t:F3=\E[1;2P:F4=\E[1;2Q:F5=\E[1;2R:F6=\E[1;2S:\\n\t:F7=\E[15;2~:F8=\E[17;2~:F9=\E[18;2~:FA=\E[19;2~:kb=\x7f:\\n\t:K2=\EOE:kB=\E[Z:kF=\E[1;2B:kR=\E[1;2A:*4=\E[3;2~:\\n\t:*7=\E[1;2F:#2=\E[1;2H:#3=\E[2;2~:#4=\E[1;2D:%c=\E[6;2~:\\n\t:%e=\E[5;2~:%i=\E[1;2C:kh=\E[1~:@1=\E[1~:kH=\E[4~:\\n\t:@7=\E[4~:kN=\E[6~:kP=\E[5~:kI=\E[2~:kD=\E[3~:ku=\EOA:\\n\t:kd=\EOB:kr=\EOC:kl=\EOD:km:’,
‘TF_ADJUST_HUE_FUSED’: ‘1’,
‘TF_ADJUST_SATURATION_FUSED’: ‘1’,
‘TF_AUTOTUNE_THRESHOLD’: ‘2’,
‘TF_ENABLE_WINOGRAD_NONFUSED’: ‘1’,
‘TRT_VERSION’: ‘5.0.2.6’,
‘WINDOW’: ‘0’,
‘_’: ‘/usr/local/bin/jupyter’,
‘_CUDA_COMPAT_PATH’: ‘/usr/local/cuda/compat’,
‘_CUDA_COMPAT_STATUS’: ‘CUDA Driver OK’}
Funny thing is, despite all the uninstall/reinstalling, jupyter’s environment variables still point to cuda 10.0. Though I am quite sure that since I removed cuda10.0 and put cuda9.0, as well as established the symlink from cuda → cuda9.0, I should be on cuda9.0 right now.