[CUFFT] Intermittent CUFFT_INVALID_PLAN return codes.

Accelerated Computing GPU-Accelerated Libraries
1

I am having an intermittent problem with CUFFT_INVALID_PLAN return codes from my code. Below is a snippet with error handling omitted:

cufftResult created = cufftCreate(&plan);
//created == CUFFT_SUCCESS

cufftResult configured = cufftSetAutoAllocation(plan, false);
//configured == CUFFT_INVALID_PLAN

or

cufftResult created = cufftCreate(&plan);
//created == CUFFT_SUCCESS

cufftResult configured = cufftSetAutoAllocation(plan, false);
//configured == CUFFT_SUCCESS

cufftResult planned = cufftPlanX(...);
//planned == CUFFT_INVALID_PLAN

I have scenarios where the cufftCreate() returns success, then a subsequent calls complain the plan is invalid. I don’t understand what, if any, circumstances could result in the above sequence of events?

Any clarification anyone could provide would be most appreciated.

Edit: After printing out the plan numbers as they are created, I find the first incidence of this issue is with plan 1023 (which immediately looked suspicious). This ultimately lead me to a comment at the following URL: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/CuFFTPlanCache.h which notes:

// NB: cuFFT allocates a starting plan array of size 1024. It should grow the
//     array as more plans are created. However, a bug in cuFFT (at least
//     present in CUDA 9.1) causes the cufftSetAutoAllocation call on the
//     1024-th plan to fail with CUFFT_INVALID_PLAN. Therefore, we check that
//     cache size is leq 1023. The initial plan array size is 1024 for
//     CUDA 8.0 ~ 9.2 so setting this as a CUDA-version-agnostic constant should
//     be fine for now.
// TODO: When CUDA 10 comes out, check if the bug is fixed or if we need another
//       number for CUDA 10.
constexpr int64_t CUFFT_MAX_PLAN_NUM = 1023;
static_assert(CUFFT_MAX_PLAN_NUM >= 0 && CUFFT_MAX_PLAN_NUM <= std::numeric_limits<size_t>::max(),
"CUFFT_MAX_PLAN_NUM not in size_t range");

Is this actually a known cufft problem, and if so, is there a suggested workaround?