The network including deformable convolution plugin was builded to trt-engine many times, but only one time the build result is right and not NAN. The different build process logs are given below. It seems like the layer fusion approaches are different, and the right approach happens by chance.
TensorRT Version: 126.96.36.199
GPU Type: 2080
Nvidia Driver Version: 440.33.01
CUDA Version: 10.2
CUDNN Version: 8.0.2
Operating System + Version: ubuntu 16.04
PyTorch Version (if applicable): 1.3