I am having an intermittent problem with CUFFT_INVALID_PLAN return codes from my code. Below is a snippet with error handling omitted:
cufftResult created = cufftCreate(&plan); //created == CUFFT_SUCCESS cufftResult configured = cufftSetAutoAllocation(plan, false); //configured == CUFFT_INVALID_PLAN
cufftResult created = cufftCreate(&plan); //created == CUFFT_SUCCESS cufftResult configured = cufftSetAutoAllocation(plan, false); //configured == CUFFT_SUCCESS cufftResult planned = cufftPlanX(...); //planned == CUFFT_INVALID_PLAN
I have scenarios where the cufftCreate() returns success, then a subsequent calls complain the plan is invalid. I don’t understand what, if any, circumstances could result in the above sequence of events?
Any clarification anyone could provide would be most appreciated.
Edit: After printing out the plan numbers as they are created, I find the first incidence of this issue is with plan 1023 (which immediately looked suspicious). This ultimately lead me to a comment at the following URL: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/CuFFTPlanCache.h which notes:
// NB: cuFFT allocates a starting plan array of size 1024. It should grow the // array as more plans are created. However, a bug in cuFFT (at least // present in CUDA 9.1) causes the cufftSetAutoAllocation call on the // 1024-th plan to fail with CUFFT_INVALID_PLAN. Therefore, we check that // cache size is leq 1023. The initial plan array size is 1024 for // CUDA 8.0 ~ 9.2 so setting this as a CUDA-version-agnostic constant should // be fine for now. // TODO: When CUDA 10 comes out, check if the bug is fixed or if we need another // number for CUDA 10. constexpr int64_t CUFFT_MAX_PLAN_NUM = 1023; static_assert(CUFFT_MAX_PLAN_NUM >= 0 && CUFFT_MAX_PLAN_NUM <= std::numeric_limits<size_t>::max(), "CUFFT_MAX_PLAN_NUM not in size_t range");
Is this actually a known cufft problem, and if so, is there a suggested workaround?