I am having an intermittent problem with CUFFT_INVALID_PLAN return codes from my code. Below is a snippet with error handling omitted:
cufftResult created = cufftCreate(&plan);
//created == CUFFT_SUCCESS
cufftResult configured = cufftSetAutoAllocation(plan, false);
//configured == CUFFT_INVALID_PLAN
or
cufftResult created = cufftCreate(&plan);
//created == CUFFT_SUCCESS
cufftResult configured = cufftSetAutoAllocation(plan, false);
//configured == CUFFT_SUCCESS
cufftResult planned = cufftPlanX(...);
//planned == CUFFT_INVALID_PLAN
I have scenarios where the cufftCreate() returns success, then a subsequent calls complain the plan is invalid. I don’t understand what, if any, circumstances could result in the above sequence of events?
Any clarification anyone could provide would be most appreciated.
Edit: After printing out the plan numbers as they are created, I find the first incidence of this issue is with plan 1023 (which immediately looked suspicious). This ultimately lead me to a comment at the following URL: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/CuFFTPlanCache.h which notes:
// NB: cuFFT allocates a starting plan array of size 1024. It should grow the
// array as more plans are created. However, a bug in cuFFT (at least
// present in CUDA 9.1) causes the cufftSetAutoAllocation call on the
// 1024-th plan to fail with CUFFT_INVALID_PLAN. Therefore, we check that
// cache size is leq 1023. The initial plan array size is 1024 for
// CUDA 8.0 ~ 9.2 so setting this as a CUDA-version-agnostic constant should
// be fine for now.
// TODO: When CUDA 10 comes out, check if the bug is fixed or if we need another
// number for CUDA 10.
constexpr int64_t CUFFT_MAX_PLAN_NUM = 1023;
static_assert(CUFFT_MAX_PLAN_NUM >= 0 && CUFFT_MAX_PLAN_NUM <= std::numeric_limits<size_t>::max(),
"CUFFT_MAX_PLAN_NUM not in size_t range");
Is this actually a known cufft problem, and if so, is there a suggested workaround?