Multi-Process Service : MPS client limit issue


I’m using 1 server with 1 GPU, Ubuntu 14.04 and K20c (cc 3.5 Kepler), CUDA8.0)

When I run MPS server and send device query (make process on gpu), it makes MPS clients and works very well during the number of MPS client spawned is lower than 256. But MPS server seems to have a limitation. When it makes MPS client(gpu process) over 255 times, it doesn’t works well. New MPS client spawns and connect to MPS server, but it can’t use any GPU query and disconnect immediately. I restart the MPS server, then it works well again, 255 times.

This problem happens on K40c and K20c(cc 3.5). And It doesn’t happen on Geforce GTX 980(cc 5.2) and GTX 1080(cc 6.1). I think this happens on cuda capability 3.5 device only.

Is it just limitation on cc3.5 device? Or is there an any solution?

I solved this problem. I just reinstall Nvidia graphic driver latest version.