Create Tensorflow cnn layer out of memeory when decode large video

Gaming and Visualization Technologies Video Processing & Optical Flow
1

I use GTX1050 decode a large video(2050*1040) from ipcamera,then pass the decoded video frame to Tensorflow humen detect
layer (include CNN),report out of memory in
session->run(...)
anyone happened this problem?

2

Since you are running out of memory we need to look in to scope of video memory optimization that can be plausible from the upper layers.

Please let us know the following:

  1. Are you intending to do training or inferencing?
  2. Are you running the decoding and tensorflow in parallel? If Yes, then can you do the decoding offline and store it and run the network. This is not a solution but it may help in some isolation.
  3. What is the number of ulNumDecodeSurfaces and ulNumOutputSurfaces you are allocating in CUVIDDECODECREATEINFO? Any reduction in these values would result to reduction in video memory consumption.
  4. Are you decoding multiple bit-streams or you are decoding a single bit-stream in your process of training/inference?
  5. By any chance is the video an I/IDR frame only video?
  6. Do you have an option of doing the training/inference at a lesser resolution? Is it possible that you downscale the original video to a lesser resolution and let your network run on that. Also, is reducing the resolution of your camera feed a valid option in the scenario you are operating?

Your card is GTX1050, it has got lesser video memory and the chances that you will run out of video memory is higher. Hence we need to ensure that we utilize the memory precisely - only where it is needed.

Thanks,
Ryan Park