Hi,
For further optimization, we would suggest shift the loading from CPU to GPU. You may refer to this patch:
How to create opencv gpumat from nvstream? - #18 by DaneLLL
And check if there is other APIs which can achieve the same function. If not, may need to implement CUDA code.