Hi Bradley,
This scenario is a little challenging due to the scale. I wonder, are you using the gripper visual mesh for collision detection? It is quite complex which will lead to a lot of redundant contacts making the problem harder. If possible you should use a simplified mesh for the rigid body collision shape.
In this case linear (inner) iterations may be more important than substeps. This is because we use triangle mesh-based contact, and once interpenetration occurs it cannot be recovered from. This means each substep needs to be solved to convergence. I would try some solver params in this range:
sim_params.flex.substeps = 4
sim_params.flex.num_outer_iterations = 6
sim_params.flex.num_inner_iterations = 40
sim_params.flex.relaxation = 0.7
Also, ensure you have enough thickness on the shapes to avoid interpenetration.
Hope that helps.
Cheers,
Miles