It seems that setting the CUDA_MPS_ACTIVE_THREAD_PERCENTAGE variable while the application is running does not immediately change the active thread percentage, it only takes effect after the application is restarted. Is there any way to apply the change of allocated SMs to an application while it is running? For example, when I have multiple Tensorflow applications sharing an A100 GPU using MPS, when I change the CUDA_MPS_ACTIVE_THREAD_PERCENTAGE variable, I have to restart all the applications, which brings in extra overhead.
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Multi-Process Service Active Thread Percentage | 0 | 548 | May 5, 2022 | |
| Can I dynamically change CUDA_MPS_ACTIVE_THREAD_PERCENTAGE to a running MPS process? | 3 | 668 | April 11, 2026 | |
| MPS set_default_active_thread_percentage not working as expected | 2 | 2284 | November 9, 2021 | |
| General question on MPS set_active_thread_percentage | 0 | 656 | December 14, 2020 | |
| Mutli Process Service crashes on setting up the `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` when launching a huge number of processes (say around 40~48 ) | 0 | 805 | August 11, 2023 | |
| Misunderstand about MPS non-uniform partitioning | 1 | 118 | April 11, 2026 | |
| MPS: Limiting threads to different thresholds for multi-GPU processes | 1 | 793 | October 27, 2021 | |
| Set_default_active_thread_percentage mps server limits memory too | 1 | 554 | February 15, 2023 | |
| Improving MPS performance using Volta MPS Execution Resource Provisioning | 5 | 1530 | July 4, 2019 | |
| MPS with multiGPUs | 0 | 1047 | May 9, 2020 |