simpleCUFFT_callback, CUDA 7.0 and compute capability 3.7

I tried to recompile the simpleCUFFT_callback both with “-gencode arch=compute_37,code=compute_37” and “-gencode arch=compute_37,code=sm_37” (in separate times). In the first case the compilation succeeds but it fails at the first execution of the plan (CUFFT_EXEC_FAILED, cuda error appears to be cudaErrorInvalidDeviceFunction), and in the second case it fails to link:

/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a(kernel.o): In function `__sti____cudaRegisterAll_53_tmpxft_0000294c_00000000_16_kernel_compute_53_cpp1_ii_e0b5062a()':
tmpxft_0000294c_00000000-13_kernel.compute_35.cudafe1.cpp:(.text+0x2d): undefined reference to `__cudaRegisterLinkedBinary_53_tmpxft_0000294c_00000000_16_kernel_compute_53_cpp1_ii_e0b5062a'
/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a(kernel.o): In function `global constructors keyed to lut_dp_2_4':
tmpxft_0000294c_00000000-13_kernel.compute_35.cudafe1.cpp:(.text+0x1ac7d): undefined reference to `__cudaRegisterLinkedBinary_53_tmpxft_0000294c_00000000_16_kernel_compute_53_cpp1_ii_e0b5062a'
collect2: error: ld returned 1 exit status

Is this expected, and I should use 3.5?

I would also like to add that simpleCUFFT does not fail instead. So I’m believing that something weird is going on with callbacks and/or statically linking cufft.

Try using CUDA 7.5. And I believe you may have errors in your compile command sequence, which you haven’t shown. (My guess is you are not invoking separate compilation with device linking correctly, which is required for this sample code and for cufft callbacks in general, since the callback function must be linked across compilation units).

On CUDA 7.5, I did the following:

  1. Copy all the files from /usr/local/cuda/samples/7_CUDALibraries/simpleCUFFT_callback to a working directory. (I did the remaining steps in the working directory.)
  2. Modify the following line in the Makefile from:
INCLUDES := -I../../common/inc

to:

INCLUDES := -I$(CUDA_PATH)/samples/common/inc
  1. Modify the following line in the Makefile from:
SMS ?= 20 30 35 50 52

to:

SMS ?= 20 30 35 37 50 52
  1. I then issued

make

and got the following output:

$ make
"/usr/local/cuda-7.5"/bin/nvcc -ccbin g++ -I"/usr/local/cuda-7.5"/samples/common/inc  -m64    -dc -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o simpleCUFFT_callback.o -c simpleCUFFT_callback.cu
"/usr/local/cuda-7.5"/bin/nvcc -ccbin g++   -m64      -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_52,code=compute_52 -o simpleCUFFT_callback simpleCUFFT_callback.o  -lcufft_static -lculibos
mkdir -p ../../bin/x86_64/linux/release
cp simpleCUFFT_callback ../../bin/x86_64/linux/release
$

There were no compile errors or warnings.

On CUDA 7.0, the situation was a little different. The compile successfully completed, but I had a variety of warnings. It appears that the static cufft libraries in CUDA 7.0 may have been built without cc 3.7 binary support. If you must use CUDA 7.0, you may have better luck just running the binaries built using the standard Makefile with cc 3.5 support. cc 3.5 code should be binary-compatible with a cc 3.7 device.

For example, change the first line of the Makefile to point to CUDA 7.0 instead of CUDA 7.5, and change the SMS line to:

SMS ?= 35

Then, when I issue make, I get the following:

$ make
"/usr/local/cuda-7.0"/bin/nvcc -ccbin g++ -I"/usr/local/cuda-7.0"/samples/common/inc  -m64    -dc -gencode arch=compute_35,code=sm_35 -gencode arch=compute_35,code=compute_35 -o simpleCUFFT_callback.o -c simpleCUFFT_callback.cu
"/usr/local/cuda-7.0"/bin/nvcc -ccbin g++   -m64      -gencode arch=compute_35,code=sm_35 -gencode arch=compute_35,code=compute_35 -o simpleCUFFT_callback simpleCUFFT_callback.o  -lcufft_static -lculibos
mkdir -p ../../bin/x86_64/linux/release
cp simpleCUFFT_callback ../../bin/x86_64/linux/release
$

Again, no errors or warnings.

Yes I’m doing that, and no I cannot complete linking when compiling with sm_37. The only thing that I omitted in the first post is a long streak of warnings, which according to the cuFFT docs can be ignored:

nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:memory_helpers.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:planner_size_one.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:size_one_kernel_body.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:bluestein.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:radix_launcher.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:vector_launcher.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:radixM.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0002A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0004A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0008A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0016A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0016B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0032A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0032B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0064B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0128B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0128C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0256B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0256C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0512C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix1024C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0003A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0009A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0027B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0081B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0243C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0729C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0005A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0025B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0125C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0007A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0049B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRadix0343C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0002A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0004A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0008A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0016A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0016B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0032B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0064B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0128B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0128C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0256B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0256C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0512C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0003A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0009A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0027B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0081B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0243C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0005A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0025B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0125C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0007A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0049B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRadix0343C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0004A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0008A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0016A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0016B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0032B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0064C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0128C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0256B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0256C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector0512C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector1024C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector1024D.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector2048C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector2048D.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector4096C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector4096D.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector8192D.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spVector8192E.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector0004A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector0008A.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector0016B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector0032B.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector0064C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector0128C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector0256C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector0512C.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector1024D.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpVector2048D.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:spRealComplex.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:dpRealComplex.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:radix_sp.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:vector_sp.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:radix_dp.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:vector_dp.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:tmpdb_radix_m_callback.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:radix_m_sp_callback.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:radix_m_dp_callback.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:tmpdb_radix_callbacks.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:tmpdb_vector_callbacks.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:vector_callback_sp.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:vector_callback_dp.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:radix_callback_sp.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:radix_callback_dp.o'
nvlink warning : SM Arch ('sm_37') not found in '/cineca/prod/compilers/cuda/7.0.28/none/bin/..//lib64/libcufft_static.a:kernel.o'

Mind that I’m using the unpatched libcufft (7.0.28, not 7.0.35).

Since compilation and execution fails, I would question the docs on the fact that warnings of this kind should be ignored.

I guess my last post was not clear.

The recommendation is to use CUDA 7.5 if you want to compile/link against sm_37.

Otherwise, if you must use CUDA 7.0 in this fashion, then compile/link against sm_35.