Oh, turns out the addition of nvbuf-memory-type=3 to nvstreammux and the two nvvideoconverts was helpful :)
nvbuf-memory-type=3
nvstreammux
nvvideoconverts