Is your Mellanox switch managed ? Does it run OpenSM ?
In any case, starting up OpenSM on a Linux server, should not cause the server to reboot (regardless of whether or not there are other SMs running in the subnet). What is the OpenSM version ? How is it started ?