Low performance of cuSOLVER compared to CSparse?

I need to invert a matrix of >10k by 10k elements. I was using CSparse, which took around 20 ms for each solve. I wanted to go to higher matrix sizes, so I needed higher performance.
I have ported my code to use cuSOLVER, cusolverSpDcsrlsvqr to be specific. Now each solve takes around ~200 ms. Is this expected? My GPU (GTX1070) reports ~70% usage during usage.