Nvargus throws SCF: Error InvalidState when deepstream-app is launched as system service

jazal · February 14, 2023, 11:55pm

I am trying to run deepstream-transfer-learning-app as a system service, using a single CSI camera as a source and fakesink as the only sink. Although the app runs properly when run from terminal, attempts to run it as a system service result in nvargus crashing.

When I run the program from terminal, I find it runs smoothly for at least 1 hour. However, when I try to run the same program as a system service it stops after some ~35 seconds, displaying the following information in the associated journalctl log:

SYNC_IOC_FENCE_INFO ioctl failed with 9
ERROR from src_elem: DISCONNECTED
Debug info: Argus Error Status
CONSUMER: ERROR OCCURRED
Quitting
SYNC_IOC_FENCE_INFO ioctl failed with 9
GST_ARGUS: Cleaning up
App run failed

My service definition file looks like this (replacing by full paths):

[Unit]
Description=Deepstream App Service
StartLimitIntervalSec=0

[Service]
ExecStart=/opt/nvidia/deepstream/<PATH_TO_PROGRAM>/deepstream-transfer-learning-app -c <PATH_TO_CONFIG_FILE>
Type=simple
Restart=no

[Install]
WantedBy=multi-user.target

I tried to check nvargus-daemon by initializing it from terminal using sudo /usr/sbin/nvargus-daemon, and I found the following information:

=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections…=== deepstream-transfer-learning-app[31405]: Connection established (7F7D9C91D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
(NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
(NvCamV4l2) Error ModuleNotPresent: (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
(NvOdmDevice) Error ModuleNotPresent: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 107)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
NvPclOpen: PCL Open Failed. Error: 0xf
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 593)
SCF: Error BadParameter: (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 437)
SCF: Error BadParameter: (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 305)
SCF: Error BadParameter: (propagating from src/api/CameraDriver.cpp, function getSource(), line 471)
Acquiring SCF Camera device source via index 1 has failed. === deepstream-transfer-learning-app[31405]: CameraProvider initialized (0x7f789e4d80)LSC: LSC surface is not based on full res!
CRASH STARTS HERE
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 542)
=== deepstream-transfer-learning-app[31405]: CameraProvider destroyed (0x7f789e4d80)=== deepstream-transfer-learning-app[31405]: Connection closed (7F7D9C91D0)=== deepstream-transfer-learning-app[31405]: Connection cleaned up (7F7D9C91D0)

I have tried setting the user in the system definition file and calling the application from a script that I later called from the service definition file. Both situations resulted in the same error. Do you have any idea of what may be causing this situation?

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Nvidia Jetson Nano Developer Kit 4GB
• DeepStream Version 6.0.1
• JetPack Version (valid for Jetson only) 4.6.2
• TensorRT Version 8.2.1

fanzh · February 15, 2023, 2:22pm

please refer to this topic Run deepstream-app on reboot.

jazal · February 15, 2023, 7:34pm

For what I understand, the previous reference refers to a case in which deepstream fails on boot because some of the required services have not finished loading yet. In my case, the system is already running when I start the service. More specifically, I would see my error when doing the following sequence:

Boot system
Run deepstream app from terminal → works OK, stop and quit after some minutes
Initialize deepstream app from service → works for ~35 seconds, then nvargus fails
Restart nvargus-daemon service
Run deepstream app from terminal → works OK, stop and quit after some minutes
Restart nvargus-daemon service
Initialize deepstream app from service → works for ~35 seconds, then nvargus fails
Restart nvargus-daemon service
Run deepstream app from terminal → works OK

Do you think waiting for services to start, which should already have happened by the time I start the service, could be related to the issue?

JerryChang · February 16, 2023, 6:30am

hello jazal,

there’s camera app dependency, you shall wait till /etc/systemd/system/nvargus-daemon.service is enabled.

jazal · February 16, 2023, 8:27am

Hi JerryChang,

I tried adding the following to the [Unit] section of the service definition file:

Wants=nvargus-daemon.service
After=nvargus-daemon.service

However, behavior is still the same. Service starts, captures images for some 35 seconds and then fails. I think as the system has already booted by the time I run the service the After clause should have no effect.

JerryChang · February 16, 2023, 8:54am

hello jazal,

could you please check the basic camera functionality,
you may enable gst pipeline with nvarguscamerasrc plugin,
for example, $ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format= NV12' ! nvoverlaysink -ev

jazal · February 16, 2023, 9:13am

Hi again JerryChang,

I have checked camera functionality both from terminal and as a service by replacing my service file definition ExecStart with the following:

/usr/bin/gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM),width=1280,height=720,framerate=60/1’ ! nvjpegenc ! multifilesink location=/tmp/test1.jpg

Interestingly enough, I found the pipeline works perfectly in both cases, what discards the problem from being associated to the camera itself…

jazal · February 16, 2023, 10:59am

I have also checked disabling img-save and found the problem persists.

jazal · February 17, 2023, 1:07pm

Hi again, JerryChan

Do you have any idea of further tests I could perform?

fanzh · February 17, 2023, 2:29pm

there is no deepstream logs, can you redirect deepstream log to a file, such as deeptream-app xx >1.log 2>1.log
can you use a local file or rtsp camera as input source? need to check if it is related to source.

jazal · February 27, 2023, 8:38pm

We actually performed further testing and found out that removing the PAUSE, RESUME event thread function solves the issue. As we were not using that functionality, it can easily be removed from the app. The obvious question would be why would this function cause such a behavior only when the app is run as a service.

fanzh · February 28, 2023, 2:47am

thanks for your update, do you mean it works after removing event_thread_func thread?

jazal · February 28, 2023, 9:10am

Hi fanzh!
Yes, we removed the event_thread_func and suddenly everything started working properly. We can now run the application for at least 10 minutes (haven’t tried longer yet).

fanzh · March 6, 2023, 3:04am

did you tried longer time?

jazal · March 6, 2023, 1:39pm

Hi Fanzh,

Yeah, it has currently been running for several days non-stop. Seems to be as stable as expected.

system · March 23, 2023, 1:44am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.