cuvidDestroyDecoder Taking a Long Time To Complete

I have an application that decodes an H264 stream using FFMPEG 4.1 and the h264_cuvid decoder. Whenever the user seeks to a new time, a call to avcodec_flush_buffers must be made to reset the internal state of the decoder before decoding the new sequence of packets. When using h264_cuvid, this call eventually leads to a call to cuvidDestroyDecoder(), and I’ve timed this function taking around 500 ms to complete, making the application feel unresponsive. Is this sort of execution time expected for a cuvid decoder? When using libx264, avcodec_flush_buffers returns in less than 1 ms. I am running on RHEL 7.5, with a Quadro P4000 and driver version 470.63.01