Please try turning on requires_grad=True in your PointVTKInferencer. By default this is off to save on computation, but this means the any gradients cannot be computed. Turning this on will enable gradients during inference at the sacrifice of increased memory usage, so monitor your batch size.
The same is true if you are using a validator or monitor: