March 9, 2023, 8:23am
When I run a tensorflow pipeline with the coco model alongside an nvenc decoding pipeline. I observe freeze image blockages and alternating gpu consumption at zero with nvidia-smi . How do the two pipelines cohabit?
how to finely debug ?
GPU RTX 3070/ docker image file nvidia/cuda:11.2.2-cudnn8-devel-ubuntu18.04 / tensorflow_gpu==2.5
Can you share more information on the Nvidia SDK that you are using? Please also provide some detailed information of your pipeline, such how encoding/decoding/resolution/FPS and number of streams etc.
March 10, 2023, 8:20am
I have fork this project
GitHub - jackersson/deep-learning-in-scale-workshop for my use case.
you must be add this option in command line -v /dev/shm:/dev/shm for start you docker and add export TF_ENABLE_GPU_GARBAGE_COLLECTION=false
pipeline 1 :
gst-launch-1.0 udpsrc port=9012 ! typefind ! queue ! tsdemux ! h264parse ! tee name=tee_nal tee_nal. ! queue ! nvh264dec ! capsfilter caps=video/x-raw(memory:GLMemory),format=NV12 ! glcolorconvert ! capsfilter caps=video/x-raw(memory:GLMemory),format=RGB
! gst_tf_detection name=detect config=“data/tf_object_api_cfg.yml” ! gldownload ! videoconvert ! capsfilter caps=“video/x-raw,format=NV12” ! shmsink socket-path=/dev/shm/input shm-size=159252480 wait-for-connection=false sync=true tee_nal. ! queue ! shmsink socket-path=/dev/shm/input_dec sync=true wait-for-connection=false
gst_tf_detection have been patched for accept this caps video/x-raw(memory:GLMemory)
start same docker with another pipeline :
gst-launch-1.0 shmsrc socket-path=/dev/shm/input_dec ! capssetter caps=video/x-h264 ! queue ! h264parse ! nvh264dec ! videoconvert ! autovideosink
when the tensorflow model is running, I observe freezes on the outputs ( /dev/shm/input_dec and /dev/shm/input). When it is not running, everything is functional
March 15, 2023, 12:23pm
after investigation, the problem seems to come from the limitation of the vram fixed for tensorflow and also from the speed of writing in /dev/shm ( bench with dd bin )
Compatibility nvenc and tensorflow - #3 by draden
March 29, 2023, 12:23pm
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.