Running GPU heavy tasks significantly decrease NvENC performance

NvENC encodes 200+ frames per second with <10ms encoding latency when nothing runs on GPU. However, when I run GPU heavy tasks like GPU benchmark test or GPU heavy games, NvENC encoding latency significantly increase to 30ms and can only encode 60 frames.

I think it is because I pass RGBA to NvENC and GPU heavy tasks may affect performance of color converting from RGBA to YUV. Is this correct? Or it may be other issue?

Also, can GPU heavy tasks decrease NvENC performance in terms of latency and frame rate?

By the way I am using GTX 1080