In my experience, this error is occasionally the result of having an OpenGL context established on a GPU that is different than the one where the CUDA context is established. This could only be possible if you have more than one GPU in your system (including non-NVIDIA GPUs).
If that is the situation, then there are various articles discussing how to fix this. One typical approach is to use optimus profiles to force the behavior you desire (everything on the same GPU).
This recent thread also covers a similar idea.
Another possible reason is that you ahve exceeded the number of encode or decode streams that your GPU supports.