Multi-CPU + 2 GPU on computational expensive process


I have a hardware with 24 AMD CPU core and 2 Nvidia GPU. Each CPU cores will start a independent process which will call functions A/B/C. Function A has a lot of if/else, so I would like to run it on CPU. Function B is image processing so I would like to run it on GPU. Function C will run on CPU too.

The machine works in a very timing-critical environment where these CPU/GPUs receive jobs continuously and run A/B/C for each job. The time needed to complete A/B/C is data dependent and different for each job.

What is the best way to implement the system? I can implement function B in CUDA and call it. But do I need to do anything more? How can I coordinate the 24 CPU cores and GPU? What would happen if a process wants to use GPU but the GPU is occupied? Is there any existing software to handle this?