cufftXtLibDesc fields and cuFFT_INTERNAL_ERROR from struct

Accelerated Computing GPU-Accelerated Libraries
1

I am attempting to write several cufftXt transforms in a loop and am only transforming certain variables forward and certain variables back using in place C2C transforms distributed on multiple GPUs. After transforming the variables, I place them in natural order using cufftXtMemcpy(CUDA_DEVICE_TO_DEVICE). Then after transforming, the variables are manipulated in k-space and transformed back into real space.

The real space variables are then stored in the previously transformed variables and the complex part of the float2 type variable are set equal to 0. However, because of this, when attempting to use the same variable for returning to natural order, I get an error. I am assuming this is because I am attempting to transform a “complex” variable, but was made real by me setting the “complex, .y” part of the float2 = 0, rather than needlessly transforming back.

The work flow is below:

  1. forward in-place transform three different variables →
  2. return k-space output of all variables to natural order →
  3. manipulate in k-space by multiplying with a kernel (which is why the k-space variables are placed in natural order) →
  4. store the output of the manipulation in one of the three transformed variables →
  5. inverse in-place transform of the one variable →
  6. return real-space output to natural order →
  7. manipulations in real space saved to the other two variables and the “complex” part set to zero →
  8. forward in-place transform three different variables →
  9. return k-space output of all variables to natural order (this time I get an error cuFFT_INTERNAL_ERROR)

I attribute the error to not transforming the two variables using CUFFT_INVERSE. However, is there some field in the cufftXtLibDesc struct that can be changed without needlessly transforming back into real-space? Additionally, is there someway to “reset” the struct?

Additionally, what are the fields in the cufftXtLibDesc structs? I can find very little information.

Alternatively, if I know how cufft permutes the data, all of these steps could be avoided by permuting the k-space kernel in the appropriate pattern.

2

There is a whole section in cuFFT documentation dedicated to multi-GPU data organization, see link.

The fields of cudaLibXtDesc are described here. When you allocate the descriptor using cufftXtMalloc, an object of the cudaLibXtDesc type is allocated. It contains some information about the decriptor version etc.
However, the fields you should be interested about the most are:

  1. cudaXtDesc *descriptor - this is a pointer to cudaXtDesc object that holds the data about the allocated memory on GPUs such as the device memory pointers (data) and allocated size (size). Those members are arrays, so there is one entry per gpu in the order you specified when calling cufftXtSetGPUs.

  2. int subFormat - this is an integer value representing the current data distribution of the descriptor - one of the cufftXtSubFormat (see docs). There are some constraints on which subformats are supported for a given transformation, see docs for more information. You can change this value manually, however, you are responsible for the data being in the specified subformat.

I hope this will help you.

David

3

Thank you for your helpful response. The subFormat does indeed need to be manually adjusted if reusing variables that were not inverse transformed. The code now works!

1 Like