decode video on multiple graphics cards

Hi there,
I am currently trying to squeeze a couple of 4k video streams out of one box. I can get a single 4K stream out of one card using cuvidDecodePicture but I try more than one, the frame rate drops below 30fps. So I chucked a second card in my machine and even though it seems to work fine, the performance is the same. So I believe it is still using only one card to decode both videos. I am creating 2 contexts per card and pushing and popping the context when I decode each video.

Please Help