FFMPEG cannot find CUDA after GPU Sharing / Time Slicing on AWS EKS (Tesla T4)


I have a Kubernetes workload that uses FFMPEG that works just fine on a single g4dn.xlarge workload (T4 Tesla) with the standard 1 GPU setup.

After applying time slicing as described in the docs(I tried both manual via a custom node group, Ubuntu NVidia AMI, and manually installing GPU Operator, as well as using the AWS Bottlerocket GPU enabled AMIs), I am able to spin up multiple pods for one physical GPU, but the same workload fails to find CUDA after time slicing.

Any clue how to tackle this?