Environment
• Hardware Platform (Jetson / GPU) AGX orin
• DeepStream Version 7.1
• JetPack Version (valid for Jetson only) 6.2 (L4T 36.4.3)
Issue Description
I’ve setup a pipeline of 3 cameras using multiurisrcbin and it works as intended initially. However, after testing REST API remove/add functionality on one of the already started cameras, I’m experiencing the following issues:
- Initially, camera removal and addition works properly without FPS loss in most cases
- Sometimes the FPS drops from 10 FPS to around 4.7 FPS (possibly due to muxing issues)
- After one or maximum two more remove/add operations on the same camera, my pipeline receives an EOS message and shuts down
API Calls Used
For reference, here are the API calls I’m using to remove and add the camera streams:
Remove Camera API Call:
curl -XPOST 'http://localhost:9000/api/v1/stream/remove' -d '{
"key": "sensor",
"value": {
"camera_id": "CAMERA_NAME",
"camera_name": "CAMERA_NAME",
"camera_url": "rtsp://x.x.x.x,
"change": "camera_remove",
"metadata": {
"resolution": "1280 x720",
"codec": "h264",
"framerate": 10
}
},
"headers": {
"source": "vst",
"created_at": "2021-06-01T14:34:13.417Z"
}
}'
Add Camera API Call:
curl -XPOST 'http://localhost:9000/api/v1/stream/add' -d '{
"key": "sensor",
"value": {
"camera_id": "CAMERA_NAME",
"camera_name": "CAMERA_NAME",
"camera_url": "rtsp://x.x.x.x,
"change": "camera_add",
"metadata": {
"resolution": "1280 x720",
"codec": "h264",
"framerate": 10
}
},
"headers": {
"source": "vst",
"created_at": "2021-06-01T14:34:13.417Z"
}
}'
Debug Logs
When removing a camera source with debug enabled, I get these errors:
uri:/api/v1/stream/remove
method:POST
0:00:41.213847908 1570 0xffff50004270 WARN rtspsrc gstrtspsrc.c:6526:gst_rtsp_src_receive_response:<src> receive interrupted
0:00:41.213904389 1570 0xffff50004270 WARN rtspsrc gstrtspsrc.c:6624:gst_rtspsrc_try_send:<src> receive interrupted
0:00:41.213925157 1570 0xffff50004270 WARN rtspsrc gstrtspsrc.c:9037:gst_rtspsrc_pause:<src> PAUSE interrupted
0:00:41.327538335 1570 0xffff24006640 ERROR v4l2allocator gstv4l2allocator.c:1398:gst_v4l2_allocator_qbuf:<nvv4l2decoder3:pool:src:allocator> failed queueing buffer 4: Bad file descriptor
0:00:41.327603551 1570 0xffff24006640 ERROR v4l2bufferpool gstv4l2bufferpool.c:1498:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder3:pool:src> could not queue a buffer 4
0:00:41.327647744 1570 0xffff24006640 ERROR v4l2allocator gstv4l2allocator.c:1398:gst_v4l2_allocator_qbuf:<nvv4l2decoder3:pool:src:allocator> failed queueing buffer 1: Bad file descriptor
0:00:41.327660576 1570 0xffff24006640 ERROR v4l2bufferpool gstv4l2bufferpool.c:1498:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder3:pool:src> could not queue a buffer 1
0:00:41.418268697 1570 0xffff50008fe0 ERROR v4l2allocator gstv4l2allocator.c:1398:gst_v4l2_allocator_qbuf:<nvv4l2decoder3:pool:src:allocator> failed queueing buffer 3: Bad file descriptor
0:00:41.418329049 1570 0xffff50008fe0 ERROR v4l2bufferpool gstv4l2bufferpool.c:1498:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder3:pool:src> could not queue a buffer 3
0:00:41.419896548 1570 0xffff50008fe0 WARN v4l2allocator gstv4l2allocator.c:840:gst_v4l2_allocator_stop:<nvv4l2decoder3:pool:src:allocator> error releasing buffers buffers: Bad file descriptor
And right before EOS I get:
uri:/api/v1/stream/remove
method:POST
0:00:56.789463455 1570 0xffff1c006640 ERROR v4l2allocator gstv4l2allocator.c:1398:gst_v4l2_allocator_qbuf:<nvv4l2decoder4:pool:src:allocator> failed queueing buffer 3: Bad file descriptor
0:00:56.789530394 1570 0xffff1c006640 ERROR v4l2bufferpool gstv4l2bufferpool.c:1498:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder4:pool:src> could not queue a buffer 3
0:00:56.789574359 1570 0xffff1c006640 ERROR v4l2allocator gstv4l2allocator.c:1398:gst_v4l2_allocator_qbuf:<nvv4l2decoder4:pool:src:allocator> failed queueing buffer 4: Bad file descriptor
0:00:56.789592533 1570 0xffff1c006640 ERROR v4l2bufferpool gstv4l2bufferpool.c:1498:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder4:pool:src> could not queue a buffer 4
End-Of-Stream reached
Setting pipeline to NULL state...
0:00:56.884911997 1570 0xaaaaedecc520 ERROR v4l2allocator gstv4l2allocator.c:1398:gst_v4l2_allocator_qbuf:<nvv4l2decoder4:pool:src:allocator> failed queueing buffer 2: Bad file descriptor
0:00:56.884965914 1570 0xaaaaedecc520 ERROR v4l2bufferpool gstv4l2bufferpool.c:1498:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder4:pool:src> could not queue a buffer 2
0:00:56.886487435 1570 0xaaaaedecc520 WARN v4l2allocator gstv4l2allocator.c:840:gst_v4l2_allocator_stop:<nvv4l2decoder4:pool:src:allocator> error releasing buffers buffers: Bad file descriptor
0:00:56.887670614 1570 0xffff50008fe0 ERROR default gstnvstreammux_pads.cpp:338:push:<src_bin_muxer> push failed [-3]
0:00:56.936155070 1570 0xffff5000a0d0 WARN videodecoder gstvideodecoder.c:3158:gst_video_decoder_prepare_finish_frame:<nvv4l2decoder0> decreasing timestamp (0:00:51.416019804 < 0:00:52.913709970)
Stopping the server..!!
Stopped the server..!!
Pipeline stopped
Additional Information
- The EOS only happens after restarting a camera 2 or 3 times, not on the initial remove/add cycle
- For testing simplicity, my current pipeline is just
multiurisrcbinwith 3 cameras linked tofakesinkwith a probe connected to retrieve batched meta for FPS counting - Changing to another camera source shows same behaviour
Is this usual behavior, or is something wrong with my setup? Also, any suggestions on how to solve the occasional FPS drop issue would be appreciated.