Update:
The issue has been gone after I saw this thread Very small Bounding Boxes with custom sgie model - #9 by mchi and increased the muxer width and height.
Now all the objects detected in the pgie are infered with sgie.
However, this shouldn’t be the case because I didn’t specify any minimum sizes for the sgie model. If you can explain why this happens, that would be great. Because I used to set the muxer shape exactly like the pgie model input shape to prevent multiple resizing of the frame inside the pipeline plugins, but now I would have to set it to some high value just in case any small bbox is detected.