Issues with nvcompositor on TX2 running Jetpack 4.5.1

After running nvcompositor for 10-20 mins, it will crash, no bus messages or signals are emitted, only these messages in the stdout/stderr:

SYNC_IOC_FENCE_INFO ioctl failed with 9
NvMMLiteVideoEncDoWork:NvMSEncConvertSurfaceFormat failed
SYNC_IOC_FENCE_INFO ioctl failed with 9
NvMMLiteVideoEncDoWork:NvMSEncConvertSurfaceFormat failed
SYNC_IOC_FENCE_INFO ioctl failed with 9
NvDdkVicExecute Failed
nvbuffer_composite Failed

I am able to reproduce this almost 100% of the time, on different TX2s, with different live input sources ranging from test to live cameras.
This makes the VIC nvcompositor element essentially unusable, as via a GStreamer application I can’t even track when this message happens, there is no indication except for the stderr messages and the stream stopping.

Anyone encountered this?
Right now I’m using something along these lines:

nvcompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=640 sink_0::height=360 \
sink_1::xpos=640 sink_1::ypos=0 sink_1::width=640 sink_1::height=360 \
sink_2::xpos=0 sink_2::ypos=360 sink_2::width=640 sink_2::height=360 \
sink_3::xpos=640 sink_3::ypos=360 sink_3::width=640 sink_3::height=360 \
! "video/x-raw(memory:NVMM),framerate=15/1" ! nvvidconv ! <Rest of pipeline...> \
udpsrc port=27000 ! application/x-rtp ! rtph264depay ! nvv4l2decoder enable-max-performance=true ! nvvidconv ! comp.sink_0 \
udpsrc port=37000 ! application/x-rtp ! rtph264depay ! nvv4l2decoder enable-max-performance=true ! nvvidconv ! comp.sink_1 \
udpsrc port=47000 ! application/x-rtp ! rtph264depay ! nvv4l2decoder enable-max-performance=true ! nvvidconv ! comp.sink_2 \
udpsrc port=57000 ! application/x-rtp ! rtph264depay ! nvv4l2decoder enable-max-performance=true ! nvvidconv ! comp.sink_3

Hi,
Please share a complete pipeline and steps. We can set up and run to reproduce the issue, and do investigation.