Here are a few resources:
- question on SO
- question on these forums, I imagine there is a straightforward extension of the 256-bit ninja stuff there to 512 bits (here is 128 bit version)
- xmp/cgbn
If it were me, I would start with xmp/cgbn, and then look at the ninja stuff by njuffa only if needed.
I wouldn’t start out on this road by worrying about register usage. You don’t have direct control over that anyway. Assuming that registers limit your options is not supportable. And I don’t know how you assess efficiency without building test cases. Unless you are a compiler genius.