Nvenc failed after a few frames

[pes1202203489]

I am working with gstreamer based nvv4l2h264 for h264 encoding of frames
however the pipeline fails after sending 6 frames to the receiver. (Using RTP264pay to send frames over using UDP)

this is my pipeline

It works when I remove the hw encoder
Is this a driver related issue