Hi Team,
I have created a cluster with two workstations, and installed BCM on the head node. The regular node is also a workstation with two GPUs. My goal is to assign a specific gpu to a job in slurm. For example, I want to start the first job with the second GPU in my regular node and the second job with the first GPU. When I check the slurm documentation, I see that this is doable:
It is also implied here:
" Requests for typed vs non-typed generic resources must be consistent within a job. For example, if you request –gres=gpu:2 with sbatch , you would not be able to request –gres=gpu:tesla:2 with srun to create a job step. The same holds true in reverse, if you request a typed GPU to create a job allocation, you should request a GPU of the same type to create a job step."
First, I created the two GPU without type entries to verify that it works, and in deed it worked. Here is the second gpu without type field is filled:
And the node is able to see them:
Here are how the config files look like for slurm:
I can start a job and use both gpus:
Then, I modified the GPUs by specifying the types:
The config files are as described in the documentation:
But when I do that, the node is unable to get to the idle state:
And the journal output is:
I managed to assign the gpu I choose by using the CUDA_VISIBLE_DEVICES environment, but it only worked when I change it in the prolog script. Can I do this by using the Type field or by any other way.
Thank you.