nvidia-smi: command not found but GPU works fine

Accelerated Computing CUDA CUDA Setup and Installation
1

I recently need to install some package but end up having problem with

nvidia-smi: command not found

utility. I’m working on ubuntu 16.04, CUDA toolkit 9.1 version. But I’m not sure which which driver am I using so maybe that’s part of problem too:

$ dpkg -l | grep -E "nvidia-[0-9]{3}"
rc  nvidia-390                         390.59-0ubuntu0~gpu16. amd64                  NVIDIA binary driver - version 390.59
rc  nvidia-396                         396.54-0ubuntu0~gpu16. amd64                  NVIDIA binary driver - version 396.54
iU  nvidia-440                         440.33.01-0ubuntu1     amd64                  NVIDIA binary driver - version 440.33.01
iU  nvidia-440-dev                     440.33.01-0ubuntu1     amd64                  NVIDIA binary Xorg driver development files

$ grep "X Driver" /var/log/Xorg.0.log
[  4612.223] (II) NVIDIA dlloader X Driver  396.54  Tue Aug 14 22:15:03 PDT 2018

I don’t know if i’m using 396 or those four? Any help would be appreciated!

2

what is the output of

dmesg |grep NVRM

?

3 
[214461.087061] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[214461.100174] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[214461.151824] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[214461.361610] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[214554.281119] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[214554.423063] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[214554.441038] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[214554.588352] NVRM: Xid (PCI:0000:04:00): 43, Ch 00000008, engmask 00000101
[214554.657509] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[214692.278218] NVRM: Xid (PCI:0000:04:00): 43, Ch 00000008, engmask 00000101
[214695.611351] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[214923.777076] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[214923.818995] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[214923.884493] NVRM: Xid (PCI:0000:04:00): 43, Ch 00000008, engmask 00000101
[214924.118478] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[214924.417907] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[607830.077462] NVRM: Xid (PCI:0000:04:00): 43, Ch 00000008, engmask 00000101
[1902037.168111] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[1902037.345671] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[1902037.645326] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[1902037.690881] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[1902224.658521] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[1902225.332003] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[1902225.346832] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[1902252.103294] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[1902252.132314] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[1902252.178938] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[1902252.264398] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[1902320.578752] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[1902320.794330] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[1902320.841009] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[1902320.918827] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[1902819.594987] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[1902819.629456] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[1902819.693592] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[1902819.743622] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[1902857.972950] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[1902858.038968] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[1902858.131103] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[1902858.173831] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[2256275.656649] NVRM: GPU at PCI:0000:89:00: GPU-a23e2a14-1fb7-ca4f-87a6-03830a61bdfd
[2256275.656659] NVRM: GPU Board Serial Number: 
[2256275.656665] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000018, engmask 00000101
[2275152.089784] NVRM: GPU at PCI:0000:88:00: GPU-4f5d54c6-902b-adbb-f5c7-a98c647fd718
[2275152.089793] NVRM: GPU Board Serial Number: 
[2275152.089799] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3312806.454342] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3312806.557331] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3312806.612715] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3312806.756874] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000018, engmask 00000101
[3312877.284107] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3312877.910895] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000018, engmask 00000101
[3312877.951486] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3325287.970598] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3325288.028493] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3325288.154543] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3325288.232181] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3325288.384158] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3325339.799611] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3325339.834888] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3325339.921408] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3325339.938641] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3325340.006608] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3325482.185809] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3325482.226398] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3325482.303738] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3325482.312021] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3325482.334955] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3325534.417203] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3325534.419193] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3325534.426143] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3325534.484524] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3325534.577213] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3325576.519457] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3325576.561565] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3325576.705829] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3325576.750688] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3325576.835426] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3325606.870322] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3325606.976066] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3325606.981738] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3325606.989215] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3325607.011643] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3325970.926912] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3325970.972572] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3325971.008127] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3325971.072618] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3325971.102811] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3326207.079789] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3326207.163508] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3326207.357511] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3326207.558147] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3326207.600490] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3326431.390944] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3326431.642793] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3326431.693085] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3326431.756372] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3326431.904538] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3376915.784866] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3376915.818213] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3376916.002923] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3376916.125488] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3376916.244993] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3377058.069788] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3377058.109485] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3377058.142059] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3377058.289557] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3377058.294069] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3377849.639307] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3377849.646293] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3377849.736346] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3377849.768104] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3377849.916040] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3378821.660904] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3378821.840519] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3378821.907654] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3378821.975350] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3378822.045872] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3378912.760763] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3378912.803564] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3378912.917628] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3378912.937491] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3378913.127153] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3379120.240394] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3379120.253206] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3379120.344866] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3379120.407147] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3379120.509450] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3379409.610874] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3379409.766009] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3379409.831901] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3379410.124489] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3379410.137112] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3379526.865065] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3379527.127510] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3379527.238252] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3379527.246348] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3379527.385598] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3382890.302930] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3382890.672611] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3382890.688685] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3382890.852411] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3382890.955331] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3382998.451240] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3382998.602262] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3382998.621157] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3382998.622704] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3382998.661404] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3383783.688920] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3383783.787754] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3383783.887828] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3383783.909079] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3383783.923097] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3383886.565393] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3383886.993744] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3383887.042145] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3383887.054675] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3383887.185071] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3384024.064802] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3384024.158169] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3384024.187106] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3384024.251511] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3384024.311147] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3384149.259157] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3384149.402030] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
[3384149.528795] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3384149.546614] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3384149.731835] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3647466.562291] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[3647466.640290] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3647466.657926] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3647466.696835] NVRM: Xid (PCI:0000:04:00): 43, Ch 00000008, engmask 00000101
[3647466.850969] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[3648200.524280] NVRM: Xid (PCI:0000:04:00): 43, Ch 00000008, engmask 00000101
[3648200.946109] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[3648200.974731] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3648201.101230] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[3648201.340226] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3648398.714960] NVRM: Xid (PCI:0000:04:00): 43, Ch 00000008, engmask 00000101
[3648398.769331] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3648398.871470] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[3648398.911919] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3648398.951240] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[3648452.613095] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000008, engmask 00000101
[3648452.716625] NVRM: Xid (PCI:0000:08:00): 43, Ch 00000008, engmask 00000101
[3648452.909239] NVRM: Xid (PCI:0000:04:00): 43, Ch 00000008, engmask 00000101
[3648452.921732] NVRM: Xid (PCI:0000:05:00): 43, Ch 00000008, engmask 00000101
[3648453.004844] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000008, engmask 00000101
[3751454.852819] NVRM: Xid (PCI:0000:88:00): 43, Ch 00000008, engmask 00000101
[3751455.089814] NVRM: Xid (PCI:0000:89:00): 43, Ch 00000008, engmask 00000101
[3751455.207740] NVRM: Xid (PCI:0000:09:00): 43, Ch 00000018, engmask 00000101
[3751455.307361] NVRM: Xid (PCI:0000:84:00): 43, Ch 00000018, engmask 00000101
[3751455.348761] NVRM: Xid (PCI:0000:85:00): 43, Ch 00000008, engmask 00000101
4

It looks like some output is missing. reboot the machine and rerun the command, and post the output

It’s likely that you need to reload the NVIDIA GPU driver.

5

It didn’t show anything with

dmesg |grep NVRM

after reboot. I’m a bit puzzled now