How to set up two CUDA Toolkit versions simultaneously? TotalView needs CUDA Toolkit 3.0 (instead of

I have CUDA toolkit 3.2 installed on my Linux GPU system. Additionally, I want to use TotalView as CUDA debugger. However, TotalView is only compatible with CUDA Toolkit version 3.0 at the moment. Thus, how can I set up a second CUDA Toolkit on my system? My aim is to use CUDA 3.2 in all cases, only for compilations for TotalView I want to use CUDA 3.0. And TotalView must find CUDA 3.0 instead of 3.2.
Any hints would be great!


I guess that if the device drivers work fine with old versions of CUDA you can switch on/off the cuda toolkit version by hand, by changing where cuda looks for the libraries, I only have one version installed and I have the following lines in my .bashrc

export PATH=$PATH:$HOME/local/cuda/bin

export LD_LIBRARY_PATH=$PATH:$HOME/local/cuda/lib64

I guess that you could have cuda3.0 and cuda3.2 and change the .bashrc as a “working” solution. Although surely not the best.

Hope it works.

ghiret’s solution works and even lets you set the toolkit in a single Makefile to build under the toolkits.

I admit I use a cruder solution… I install each toolkit and rename them to be /usr/local/cuda3.2 /usr/local/cuda2.3 etc, then use “sudo ln -s cuda3.2 cuda” to make 3.2 the default.
The symbolic link way is easier for manual compiles, but ghiret’s custom path method is more scriptable.

thanks for your replies.

The symbolic link way is not a solution for me, as we have several users on our system and they cannot get superuser rights.
I understand that it is quite easy to select the appropiate Cuda Toolkit using a makefile. However, as we have a cluster environment (sorry, I forget to mention), I am wondering how to do it there.

  1. I can set the PATH and LD_LIBRARY_PATH variables to Toolkit 3.2 when the user logs into our GPU machine. Then he does not have to think about the right toolkit and can just use it. However, he must know that he has to switch to Toolkit 3.0 for a debug compilation (i.e. when using TotalView). Okay, I assume that is possible. But then TotalView has still the wrong path in the PATH variable (if still 3.2) :-(

  2. I do not set any PATH and LD_LIBRARY_PATH variables as default. Then the user always has to chose a certain toolkit fot every compilation. But I think, that is also not a nice solution.

Dominik’s solution is the best one. I use modules for all the libraries and toolkits I use, by the time you need to work with a couple of different MPI flavours, compilers, math libraries, toolkits and all their versions, there is really no other way to go. I current have 6 different toolkit versions and installed on my development box to support testing of code, both old and new.

avidday@cuda:~$ module avail cuda

------------------------------ /opt/env/Modules/3.2.7/modulefiles -------------------------------

cuda/2.3          cuda/3.0b         cuda/3.2(default)

cuda/3.0          cuda/3.1          cuda/3.2rc

avidday@cuda:~$ module display cuda/3.2



module-whatis	 Loads the CUDA gpu computing environment 

conflict	 cuda/2.3 cuda/3.0b cuda/3.0 cuda/3.1 cuda/3.2rc 

setenv		 CUDA_ROOT /opt/cuda-3.2 

setenv		 CUDA_INSTALL_PATH /opt/cuda-3.2 

setenv		 CUBLAS_INSTALL_PATH /opt/cuda-3.2 

setenv		 CUBLAS_LIBRARIES -L/opt/cuda-3.2/lib64 -lcublas -lcudart 

setenv		 CUBLAS_INCLUDES -I/opt/cuda-3.2/include 

append-path	 PATH /opt/cuda-3.2/bin:/opt/cuda-3.2/computeprof/bin 

append-path	 LD_LIBRARY_PATH /opt/cuda-3.2/lib64:/opt/cuda-3.2/computeprof/bin 


Using modules allows you to do this sort of thing on the fly:

avidday@cuda:~$ module load cuda/2.3

avidday@cuda:~$ nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver

Copyright (c) 2005-2009 NVIDIA Corporation

Built on Thu_Jul_30_09:24:36_PDT_2009

Cuda compilation tools, release 2.3, V0.2.1221

avidday@cuda:~$ module switch cuda/2.3 cuda/3.2

avidday@cuda:~$ nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver

Copyright (c) 2005-2010 NVIDIA Corporation

Built on Wed_Nov__3_16:16:57_PDT_2010

Cuda compilation tools, release 3.2, V0.2.1221

Can anyone provide an example of a modulefile for using cuda 3.2 and 4.0 please? I have cuda 4.0 as default at /usr/local/cuda, and cuda 3.2 at /usr/local/cuda32/cuda. This is under OpenSUSE 11.4 x64.