How are you modelling your hand? Is it a dynamic actor or kinematic actor (infinite mass but moveable)? How are you moving it? By setting the global pose or by setting a kinematic target? If it’s kinematic and you are not setting kinematic targets but instead setting global poses (effectively teleporting the hand), this will behavioural issues because the hand is teleported inside another object rather than being moved into the object with velocity.
PhysX’s joint constraints are provided in extensions (see ExtD6Joint, ExtRevoluteJoint etc.). In UE4, all constraints are implemented using D6Joints. There is also a specialised revolute joint which implements a hinge but both the D6 and revolute should produce equivalent joints.
The problem you may be seeing is that, if the hand passes into the door, the door needs to move to escape from this penetration. If the door cannot move in a legal way to overcome the penetration (say if it would have to move past its limit or become embedded in some static/immveable geometry), then you are in a situation where the state cannot be solved in such a way that all error is eliminated - either the door is going to penetrate the hand or the door is going to violate joint limits/pass into another object.
You may be able to reduce the jitter by setting maxDepentrationVelocity to be something smaller than the default PX_MAX_F32 on your door. Something like 2m/s (in UE4, that would be 200units) should limit the amount of energy introduced by error between the door and the hand.