When I run a tensorflow pipeline with the coco model alongside an nvenc decoding pipeline. I observe freeze image blockages and alternating gpu consumption at zero with nvidia-smi . How do the two pipelines cohabit?
how to finely debug ?
GPU RTX 3070/ docker image file nvidia/cuda:11.2.2-cudnn8-devel-ubuntu18.04 / tensorflow_gpu==2.5
Can you share more information on the Nvidia SDK that you are using? Please also provide some detailed information of your pipeline, such how encoding/decoding/resolution/FPS and number of streams etc.
Video_Codec_SDK_11.0.10.zip
I have fork this project GitHub - jackersson/deep-learning-in-scale-workshop for my use case.
you must be add this option in command line -v /dev/shm:/dev/shm for start you docker and add export TF_ENABLE_GPU_GARBAGE_COLLECTION=false
pipeline 1 :
gst-launch-1.0 udpsrc port=9012 ! typefind ! queue ! tsdemux ! h264parse ! tee name=tee_nal tee_nal. ! queue ! nvh264dec ! capsfilter caps=video/x-raw(memory:GLMemory),format=NV12 ! glcolorconvert ! capsfilter caps=video/x-raw(memory:GLMemory),format=RGB
! gst_tf_detection name=detect config=“data/tf_object_api_cfg.yml” ! gldownload ! videoconvert ! capsfilter caps=“video/x-raw,format=NV12” ! shmsink socket-path=/dev/shm/input shm-size=159252480 wait-for-connection=false sync=true tee_nal. ! queue ! shmsink socket-path=/dev/shm/input_dec sync=true wait-for-connection=false
gst_tf_detection have been patched for accept this caps video/x-raw(memory:GLMemory)
start same docker with another pipeline :
gst-launch-1.0 shmsrc socket-path=/dev/shm/input_dec ! capssetter caps=video/x-h264 ! queue ! h264parse ! nvh264dec ! videoconvert ! autovideosink
when the tensorflow model is running, I observe freezes on the outputs ( /dev/shm/input_dec and /dev/shm/input). When it is not running, everything is functional
after investigation, the problem seems to come from the limitation of the vram fixed for tensorflow and also from the speed of writing in /dev/shm ( bench with dd bin )Compatibility nvenc and tensorflow - #3 by draden
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.