NVENC performance issues when creating multiple encoders

I have an application that creates multiple encode instances using FFMPEG avlib. The time to open each encode instance increases the more that I open. It seems to take even longer the larger the frame size is. Below is an excerpt from my log illustrating this (times in ms):

[2016/09/01 15:12:34.812] Enc::Create=> AVCodecOpen: 953, Core:1
[2016/09/01 15:12:35.906] Enc::Create=> AVCodecOpen: 1094, Core:0
[2016/09/01 15:12:37.124] Enc::Create=> AVCodecOpen: 1172, Core:1
[2016/09/01 15:12:38.780] Enc::Create=> AVCodecOpen: 1656, Core:0

[2016/09/08 15:19:56.394] Enc::Create=> AVCodecOpen: 1734, Core: 0
[2016/09/08 15:19:59.160] Enc::Create=> AVCodecOpen: 2766, Core: 1
[2016/09/08 15:20:01.816] Enc::Create=> AVCodecOpen: 2656, Core: 0
[2016/09/08 15:20:04.222] Enc::Create=> AVCodecOpen: 2406, Core: 1
[2016/09/08 15:20:07.394] Enc::Create=> AVCodecOpen: 3172, Core: 0

Timings are captured around this function:
int iResult = avcodec_open2(m_pCodecContext,pEncoder,ppDict);

where the m_pEncoder object is NVENC.

Using plain libx264 rather than NVENC results in times of 0-5 ms. We are using a Tesla K80 on a server machine.

Any ideas?