cufft: cufftPlan1d() returns same handle

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,

I’m having a strange phenomenon while using cufft for multiple FFT calculations.
in order to save on plan calculation time i have cufftHandle (plan) member variables in each “image class” object.
these plans are created only if there is a need to perform fft on this image and then only if the fft size has changed from the previous time an FFT was performed on this image

i noticed that at some point during program the “cufftPlan1d()” function started returning the same 4 handles even though these handles have already been created for different size fft in other image objects (and not destroyed yet).

Of course i’m getting bad fft results in addition to a crash at the end due to failure in cufftDestroy (returns INVALID_PLAN)

does anybody have any idea why this is happening and how to avoid it?

thanks!!!

The code for my plan creating and destruction functions:

[i]void CudaImage32fBatch::MakeFFTPlans(int width,int height,int numFrames)
{
if(m_DCTPlanWidth != width || m_DCTPlanHeight != height || m_DCTPlanNumFrames != numFrames)
{
KillFFTPlans();
ASSERT(m_DCTPlanA == NULL && m_DCTPlanB == NULL && m_DCTPlanAODD == NULL && m_DCTPlanBODD == NULL);
m_DCTPlanWidth = width;
m_DCTPlanHeight = height;
m_DCTPlanNumFrames = numFrames;
cufftOK(cufftPlan1d(&m_DCTPlanA, height, CUFFT_C2C, widthnumFrames)); //perform 1d fft along columns
cufftOK(cufftPlan1d(&m_DCTPlanB, width, CUFFT_C2C, heightnumFrames)); //perform 1d fft along columns
cufftOK(cufftPlan1d(&m_DCTPlanAODD, 2* height, CUFFT_C2C,widthnumFrames)); //perform 1d fft along columns
cufftOK(cufftPlan1d(&m_DCTPlanBODD, 2width, CUFFT_C2C, height*numFrames)); //perform 1d fft along columns
}
}

void CudaImage32fBatch::KillFFTPlans()
{
if(m_DCTPlanA != NULL)
{
cufftOK(cufftDestroy(m_DCTPlanA));
}
if(m_DCTPlanB != NULL)
{
cufftOK(cufftDestroy(m_DCTPlanB));
}
if(m_DCTPlanAODD != NULL)
{
cufftOK(cufftDestroy(m_DCTPlanAODD));
}
if(m_DCTPlanBODD != NULL)
{
cufftOK(cufftDestroy(m_DCTPlanBODD));
}
m_DCTPlanA = NULL;
m_DCTPlanB = NULL;
m_DCTPlanAODD = NULL;
m_DCTPlanBODD = NULL;
m_DCTPlanWidth = 0;
m_DCTPlanHeight = 0;
m_DCTPlanNumFrames = 0;
}[/i]

my config:

  • cufft 3.1
  • toolkit 3.1
  • coresponding drivers
  • tesla c1060
  • windows XP
  • visual studio 2008
2

Hi eldadk,

Yes, this is a bug in CUFFT 3.1, unfortunately. We just discovered it a few days ago.

Basically what happened was that a change was made to cufftDestroy() in CUFFT 3.1 to facilitate development of a new feature, but it had an unintended impact on some assumptions the planner makes in the way it assigns cufftHandles.

We will definitely fix this issue in CUFFT 3.2. Meanwhile, there are two known workarounds from which to choose if you need to destroy plans and then create new plans:

  1. Always destroy your plans in LIFO order (last allocated, first destroyed).

-or-

  1. Whenever you need to destroy some plans, destroy all plans and re-create the ones you still need.

Hope this helps,

Cliff

3

Hi eldadk,

Yes, this is a bug in CUFFT 3.1, unfortunately. We just discovered it a few days ago.

Basically what happened was that a change was made to cufftDestroy() in CUFFT 3.1 to facilitate development of a new feature, but it had an unintended impact on some assumptions the planner makes in the way it assigns cufftHandles.

We will definitely fix this issue in CUFFT 3.2. Meanwhile, there are two known workarounds from which to choose if you need to destroy plans and then create new plans:

  1. Always destroy your plans in LIFO order (last allocated, first destroyed).

-or-

  1. Whenever you need to destroy some plans, destroy all plans and re-create the ones you still need.

Hope this helps,

Cliff

4

Hi Cliff,

Thank you very much for your prompt reply.

  • Can you distribute a patch earlier than 3.2?

  • Can I work with CUFFT 2.3/3.0 side by side with the 3.1 toolkit and drivers? will this solve the problem?

have a nice week,

eldad.

5

Unfortunately there is no patch release planned for this issue, no. Sorry about the inconvenience.

I guess the reason that the workarounds don’t help you is that the cufftHandles are members of objects that can be instantiated and destroyed in any order, right? If so, you can still work around this issue by creating an FFT class (and a single instance of that class to which all your other objects refer) that manages all of the cufftHandles on behalf of your other objects. This is a bit more effort to implement, but only a little bit. Actually, it could be better for perf in the long run anyway, since then your objects can share FFT plans if they use the same FFT dimensions. This would save on device memory as well as on plan creation time.

If you’re on Linux or MacOS, you might be able to get CUFFT 3.0 to interoperate with version 3.1 of the CUDA Runtime (libcudart). Of course this is completely untested.

You can absolutely use the CUDA Toolkit 3.0 in its entirety with a CUDA 3.1 driver, though.

Hope this helps,

Cliff

6

Hi Cliff,

I’m actually using cufft v2.3 with the v3.1 driver and v3.1 cudart.lib on windows XP 32bit - and it all seems to be working well.

I understand this is untested - but it allows me to keep developing seamlessly as if i had upgraded cufft to v3.1 until the next version of cufft is released.

I just replaced cufft.lib cufft.h and cufft.dll in my project with these files from v2.3 - from which i had upgraded to v3.1.

Should i be looking for specific problems in the interoperability of the two versions?

FYI -

The system I’m working on is a medical system which has FDA and CE clearance and is currently deployed in a operating room and helping to treat human patients. The deployed system is using CUDA v2.3 and the CUDA algorithms play a key role in the system’s performance.

In conversations I have had with colleagues from my company and from other companies in conferences, there is a feeling that Nvidia treats this product as if it was still used exclusively for academic purposes. The fact is that CUDA is already a part of the industry, and that it plays a major role in real world applications.

We love CUDA, it is essential for our work, and we want to keep getting new exciting features. We would be happier if CUDA releases were more stable and well tested.

I still hope you might be able to patch this up somehow before the next version … ;-)

Thanks for all your help,

eldad.

7

We are aware that CUDA is now used for real work ( not just to produce academic papers…) and we are making changes in the release cycle to get stabler releases.

8

Thank you.