Thanks Michael for this.
I have to say that I thought that going back to previous linux kernel sorted out my problems. In fact, nvidia-smi was working but then it was not possible for CUDA to work anyway–it had to be rebuild for the kernel again, and then the problem reappeared.
Adopting the method you suggested (i.e., using sudo add-apt-repository ppa:graphics-drivers) worked for me. That is, I had to install the driver (387) separately and then CUDA 9.1 after extracting it from the three components (driver, cuda-toolkit and samples).