Highly variable token/sec for minimax m3

Sometimes I get ~70 tok/sec other times I get ~3 tok/sec, is there any way to make this more stable?