CUDA_ERROR_OUT_OF_MEMORY in creating optical flow images using AppOFCuda

[edit_or]

• Hardware (A30 Gpu)
• action action_recognition_net
• nvcr.io/nvidia/tao/tao-toolkit:5.0.0-pyt

I am creating optical flow images using the script below.
My server has 4 A30 Gpus.

When I run the following shell script,

#!/bin/bash

dirs=/workspace/Nyan/cv_samples_v1.3.0/action_recognition_net/data/selfinf
declare -a RGB_PATH_LIST
i=0
for entry in "$dirs"/*
do
  RGB_PATH_LIST[i++]="$entry"
done

#for((i=1;i<=${#RGB_PATH_LIST[@]};i++))
#do
#    echo "${RGB_PATH_LIST[i]}"
#done

for((i=0;i<${#RGB_PATH_LIST[@]};i++)); do
   if [ ${RGB_PATH_LIST[i]} != "NULL" ]; then
      #echo "${RGB_PATH_LIST[i]}g"
      mkdir -p "${RGB_PATH_LIST[i]}/flow"
      mkdir -p "${RGB_PATH_LIST[i]}/of"
      ./AppOFCuda --input=${RGB_PATH_LIST[i]}/"rgb"/"*.png" --output=${RGB_PATH_LIST[i]}/"flow" --preset=fast --gridSize=1 \
      &&
      python ./convert_of.py --input_flow_folder ${RGB_PATH_LIST[i]}/"flow" --output_folder ${RGB_PATH_LIST[i]}/"of" &
   fi
done
wait
   for((i=0;i<${#RGB_PATH_LIST[@]};i++)); do
      if [ ${RGB_PATH_LIST[i]} != "NULL" ]; then
        rm -r ${RGB_PATH_LIST[i]}/"flow"
        rm -r ${RGB_PATH_LIST[i]}/"of"
      fi
   done

I have errors as

GPU in use: NVIDIA A30
GPU in use: NVIDIA A30
GPU in use: NVIDIA A30
GPU in use: NVIDIA A30
GPU in use: NVIDIA A30
main : CUDA driver API error CUDA_ERROR_OUT_OF_MEMORY at /opt/tylerz/tlt_dev/action_recognition/convert_dataset/Optical_Flow_SDK_2.0.23/NvOFBasicSamples/AppOFCuda/AppOFCuda.cpp;431

main : CUDA driver API error CUDA_ERROR_OUT_OF_MEMORY at /opt/tylerz/tlt_dev/action_recognition/convert_dataset/Optical_Flow_SDK_2.0.23/NvOFBasicSamples/AppOFCuda/AppOFCuda.cpp;431

main : CUDA driver API error CUDA_ERROR_OUT_OF_MEMORY at /opt/tylerz/tlt_dev/action_recognition/convert_dataset/Optical_Flow_SDK_2.0.23/NvOFBasicSamples/AppOFCuda/AppOFCuda.cpp;431

main : CUDA driver API error CUDA_ERROR_OUT_OF_MEMORY at /opt/tylerz/tlt_dev/action_recognition/convert_dataset/Optical_Flow_SDK_2.0.23/NvOFBasicSamples/AppOFCuda/AppOFCuda.cpp;431

[Morganh]

@edit_or

When I run the following shell script

Can you share the link of this script?

[edit_or]

The original script is here.
I modify to suit my use.

NVIDIA-AI-IOT/tao_toolkit_recipes/blob/main/tao_action_recognition/data_generation/preprocess_HMDB.sh

[Morganh]

You run the script after you “docker run” nvcr.io/nvidia/tao/tao-toolkit:5.0.0-pyt , right? Can you share the command? And what is the result of nvidia-smi now?

[edit_or]

I go inside docker first.

docker run --runtime=nvidia -it --rm --entrypoint "" -v $PWD:/workspace -p 8888:8888 nvcr.io/nvidia/tao/tao-toolkit:5.0.0-pyt-actionnet /bin/bash

Then run the script.

[Morganh]

How about $nvidia-smi inside the docker?

[edit_or]

From outside docker, Nvidia-smi still worked.
Inside docker, didn’t check. I’ll check and update you. Thanks

[edit_or]

nvidia-smi output looks ok.
Is GPU in use: NVIDIA A30 message normal?

Also found the following error message.

DoInit : m_NvOFAPI->GetAPI()->nvOFInit(m_NvOFAPI->GetHandle(), &initParams)returned error 8 at /opt/tylerz/tlt_dev/action_recognition/convert_dataset/Optical_Flow_SDK_2.0.23/Common/NvOFBase/NvOFCuda.cpp;121

[Morganh]

Please download latest 5.0 notebook.

wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/tao/tao-getting-started/versions/5.0.0/zip -O getting_started_v5.0.0.zip
unzip -u getting_started_v5.0.0.zip  -d ./getting_started_v5.0.0 && rm -rf getting_started_v5.0.0.zip && cd ./getting_started_v5.0.0

The latest AppOFCuda is inside it.

Then login 5.0-pyt docker
$ docker run --runtime=nvidia -it -v /home/xxx:/home/xxx nvcr.io/nvidia/tao/tao-toolkit:5.0.0-pyt /bin/bash

and follow https://github.com/NVIDIA/tao_tutorials/blob/main/notebooks/tao_launcher_starter_kit/action_recognition_net/actionrecognitionnet.ipynb to run the commands.

apt update
apt-get install unrar
wget http://serre-lab.clps.brown.edu/wp-content/uploads/2013/10/hmdb51_org.rar
mkdir videos
unrar x -o+ hmdb51_org.rar  videos/
unrar x -o+ videos/fall_floor.rar raw_data/
unrar x -o+ videos/ride_bike.rar raw_data/
git clone https://github.com/NVIDIA-AI-IOT/tao_toolkit_recipes
pip3 install xmltodict opencv-python
cd tao_toolkit_recipes/tao_action_recognition/data_generation/
cp path_to_AppOFCuda .
apt install -y libfreeimage-dev
./preprocess_HMDB.sh raw_data output_result

[edit_or]

It works for hmdb51 dataset. I’ll try to make it work for my own dataset.