Argus daemon & multiple pipelines = issues

  • ubuntu 18.04 rootfs
  • JP 4.6.2 (32.7.2)
  • TX2 on a CTI spacely using CTI BSP on top of NVIDIA BSP (as instructed)
  • Deepstream 6.0.1 installed

Most examples and question on this forum apply to following typical flow:

  • Single process
  • Creates a single pipeline
  • Runs the pipeline forever until it stops for whatever reason
  • process terminates
  • restart process if required …

This is mostly applicable for people using gst-launch and such.

In my case I’m doing the following:

  • Single process (C app)
  • Up to 20 pipelines (1 thread per pipeline)
    → 8x analog inputs (Intersil TW6869 capture card)
    → 8x network inputs (RTSP, youtube … doesn’t matter)
    → 4x usb camera input (webcams and such)

Each thread is responsible for creating & destroying it’s pipeline based on user config.

The Analog and USBcam inputs use v4l2src source element.

So this approach only works sometimes …
The main issues here is Argus daemon.

Starting the threads mostly result in Argus errors all over the place:
Snippet below was when simply starting all analog inputs 1 by 1 serialized with a 30 sec gap between each start to properly investigate:

root@000C8B920897:/home/ncentric# ./ncss2/ncss2 -S
Using default vserver config: /etc/ncentric/ncss/vserver.conf
Using default sserver config: /etc/ncentric/ncss/sserver.conf
Using default sclient config: /etc/ncentric/ncss/sclient.conf
Detected GStreamer version 1.14.5
RTSP: Attached mount: rtsp://0.0.0.0:554/analog1
Starting stream analog1
RTSP: Attached mount: rtsp://0.0.0.0:554/analog2
Starting stream analog2
RTSP: Attached mount: rtsp://0.0.0.0:554/analog3
Starting stream analog3
RTSP: Attached mount: rtsp://0.0.0.0:554/analog4
Starting stream analog4
RTSP: Attached mount: rtsp://0.0.0.0:554/analog5
Starting stream analog5
RTSP: Attached mount: rtsp://0.0.0.0:554/analog6
Starting stream analog6
RTSP: Attached mount: rtsp://0.0.0.0:554/analog7
RTSP: Attached mount: rtsp://0.0.0.0:554/analog8
Starting stream analog7
Starting stream analog8
RTSP server started on port 554
Config: Watch Started
apiv2: Thread Started
apiv2: Starting HTTP server on port 8000
analog1: using device /dev/video0 "tw6869"
analog1: local: Using encoder: nvv4l2h264enc
analog1: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 100, Level = 0 
NVMEDIA: H265 : Profile : 1 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
analog2: using device /dev/video1 "tw6869"
analog2: local: Using encoder: nvv4l2h264enc
analog2: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: Connection refused (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
ArgusV4L2_Open failed: Connection refused
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 8 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvMMLiteBlockCreate : Block : BlockType = 8 
NVMEDIA: H265 : Profile : 1 
H264: Profile = 100, Level = 0 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
analog3: using device /dev/video2 "tw6869"
analog3: local: Using encoder: nvv4l2h264enc
analog3: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: Connection refused (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
ArgusV4L2_Open failed: Connection refused
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NVMEDIA: H265 : Profile : 1 
H264: Profile = 100, Level = 0 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
analog4: using device /dev/video3 "tw6869"
analog4: local: Using encoder: nvv4l2h264enc
analog4: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: Connection refused (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
ArgusV4L2_Open failed: Connection refused
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvMMLiteBlockCreate : Block : BlockType = 8 
H264: Profile = 100, Level = 0 
NVMEDIA: H265 : Profile : 1 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
analog5: using device /dev/video4 "tw6869"
analog5: local: Using encoder: nvv4l2h264enc
analog5: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: Connection refused (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
ArgusV4L2_Open failed: Connection refused
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteOpen : Block : BlockType = 4 
NvMMLiteBlockCreate : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NVMEDIA: H265 : Profile : 1 
H264: Profile = 100, Level = 0 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE

Can Argus Daemon handle multiple connection properly?
Is this usecase tested within nvidia or is the focus solely on the avg user case of a single process with a single pipeline using gst-launch?

It’s pretty easy to simulate …

Suppose unable to reentry for the same sensor with nvarguscamerasrc.

No, it’s not possible to use this src type.
The TW6869 is a CVBS input source, and it’s required to provide the “norm” property to set the input standard (PAL, NTSC, …) as well as providing optional “interleave” caps as the video input is interlaced and this one is required to have nvvideoconvert work properly later on.

Also, even if it could be used, the same issue appears for USB camera’s.

How can we proceed to get this fixed?
As the whole argus thing is closed source … I can’t run the debugger on it.

Why is the argus daemon even required as I’m using v4l2src?

Restarting nvargus-daemon and then restarting my app:

root@000C8B920897:/home/ncentric# ./ncss2/ncss2 -S
Using default vserver config: /etc/ncentric/ncss/vserver.conf
Using default sserver config: /etc/ncentric/ncss/sserver.conf
Using default sclient config: /etc/ncentric/ncss/sclient.conf
Detected GStreamer version 1.14.5
RTSP: Attached mount: rtsp://0.0.0.0:554/analog1
Starting stream analog1
RTSP: Attached mount: rtsp://0.0.0.0:554/analog2
Starting stream analog2
RTSP: Attached mount: rtsp://0.0.0.0:554/analog3
Starting stream analog3
RTSP: Attached mount: rtsp://0.0.0.0:554/analog4
RTSP: Attached mount: rtsp://0.0.0.0:554/analog5
RTSP: Attached mount: rtsp://0.0.0.0:554/analog6
Starting stream analog4
Starting stream analog5
Starting stream analog6
RTSP: Attached mount: rtsp://0.0.0.0:554/analog7
Starting stream analog7
RTSP: Attached mount: rtsp://0.0.0.0:554/analog8
Starting stream analog8
RTSP server started on port 554
Config: Watch Started
apiv2: Thread Started
apiv2: Starting HTTP server on port 8000
analog1: using device /dev/video0 "tw6869"
analog1: local: Using encoder: nvv4l2h264enc
analog1: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 100, Level = 0 
NVMEDIA: H265 : Profile : 1 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
analog2: using device /dev/video1 "tw6869"
analog2: local: Using encoder: nvv4l2h264enc
analog2: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error EndOfFile: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error EndOfFile: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error EndOfFile:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
ArgusV4L2_Open failed: No such file or directory
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 100, Level = 0 
NVMEDIA: H265 : Profile : 1 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
analog3: using device /dev/video2 "tw6869"
analog3: local: Using encoder: nvv4l2h264enc
analog3: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error EndOfFile: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error EndOfFile: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error EndOfFile:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
ArgusV4L2_Open failed: No such file or directory
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4

If you don’t use nvargus you can stop the nvargus-daemon

sudo service nvargus-daemon stop

HI @ShaneCCC

When stopping nvargus:

root@000C8B920897:/home/ncentric# systemctl stop nvargus-daemon
root@000C8B920897:/home/ncentric# ./ncss2/ncss2 -S
Using default vserver config: /etc/ncentric/ncss/vserver.conf
Using default sserver config: /etc/ncentric/ncss/sserver.conf
Using default sclient config: /etc/ncentric/ncss/sclient.conf
Detected GStreamer version 1.14.5
RTSP: Attached mount: rtsp://0.0.0.0:554/analog1
Starting stream analog1
RTSP: Attached mount: rtsp://0.0.0.0:554/analog2
Starting stream analog2
RTSP: Attached mount: rtsp://0.0.0.0:554/analog3
Starting stream analog3
RTSP: Attached mount: rtsp://0.0.0.0:554/analog4
Starting stream analog4
RTSP: Attached mount: rtsp://0.0.0.0:554/analog5
Starting stream analog5
RTSP: Attached mount: rtsp://0.0.0.0:554/analog6
Starting stream analog6
RTSP: Attached mount: rtsp://0.0.0.0:554/analog7
Starting stream analog7
RTSP: Attached mount: rtsp://0.0.0.0:554/analog8
Starting stream analog8
RTSP server started on port 554
Config: Watch Started
apiv2: Thread Started
apiv2: Starting HTTP server on port 8000
analog1: using device /dev/video0 "tw6869"
analog1: local: Using encoder: nvv4l2h264enc
analog1: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
H264: Profile = 100, Level = 0 
NVMEDIA: H265 : Profile : 1 
NVMEDIA_ENC: bBlitMode is set to TRUE 
NVMEDIA_ENC: bBlitMode is set to TRUE 
analog2: using device /dev/video1 "tw6869"
analog2: local: Using encoder: nvv4l2h264enc
analog2: shore: Using encoder: nvv4l2h265enc
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: Connection refused (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
ArgusV4L2_Open failed: Connection refused
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 8 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
NvMMLiteBlockCreate : Block : BlockType = 4 
NVMEDIA: H265 : Profile : 1 
H264: Profile = 100, Level = 0

Please find a visual of the pipeline generated here: (I cannot seem to upload files?)

pipeline drawing

  • I don’t even know which element triggers nvargus in the 1st place.

Could you elaborate on that?

Thanks

Suppose only argus APP trigger it.
Please narrow down to figure which process trigger it.