Acquisition at 120fps using libargus not working

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX1
1

Hello,

I’ve been trying to encode a video from the CSI camera (or from a webcam) in H264.
I’m also trying to reach the Jetson TX1 announced performance (encoding 1080p@120fps or 4K@30fps)

I’ve tried to use several libraries to acquire frames but the main are:

VisionWorks: I use a FrameSource and it works with both CSI camera and my webcam.
(Bad performance as I need to convert vx_image to NvBuffer)
Argus: Works only with the CSI camera.

So I just tried to run a L4T samples: ~/tegra_multimedia_api/samples/10_camera_recording
I changed in the main.cpp the “DEFAULT_FPS” to 120fps and tried to encode videos of size: 1280x720
Then I get the following:

ubuntu@tegra-ubuntu:~/tegra_multimedia_api/samples/10_camera_recording$ ./camera_recording
Set governor to performance before enabling profiler
NvPclHwGetModuleData: Misc Driver v4l2_focuser_stub already exists. Avoiding duplicate drivers
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
(Argus) Error InvalidState: Scf SensorModeType 0 not found (in src/common/ScfTranslate.cpp, function fromNvMMSensorModeType(), line 318)
(Argus) Error InvalidState: Scf SensorModeType 0 not found (in src/common/ScfTranslate.cpp, function fromNvMMSensorModeType(), line 318)
(Argus) Error InvalidState: Scf SensorModeType 0 not found (in src/common/ScfTranslate.cpp, function fromNvMMSensorModeType(), line 318)
(Argus) Error InvalidState: Scf SensorModeType 0 not found (in src/common/ScfTranslate.cpp, function fromNvMMSensorModeType(), line 318)
(Argus) Error InvalidState: Scf SensorModeType 0 not found (in src/common/ScfTranslate.cpp, function fromNvMMSensorModeType(), line 318)
(Argus) Error InvalidState: Scf SensorModeType 0 not found (in src/common/ScfTranslate.cpp, function fromNvMMSensorModeType(), line 318)
(Argus) Error InvalidState: Scf SensorModeType 0 not found (in src/common/ScfTranslate.cpp, function fromNvMMSensorModeType(), line 318)
PRODUCER: Creating output stream
PRODUCER: Launching consumer thread
Failed to query video capabilities: Bad address
NvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
create vidoe encoder return true
CONSUMER: Waiting until producer is connected...
PRODUCER: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Sensor_GetV4LPixelType: pixel type 0x101 invalid
CONSUMER: Got EOS, exiting...
Number of frames: 298
CONSUMER: Done.
PRODUCER: Done -- exiting.
************************************
Total Profiling Time = 0 sec
************************************

  1. Why does it always fails to load SensorMode ?

  2. I can’t achieve to capture frames at 120fps.

Thanks for your help :)

2

Hi Yassine
The message are not the key point. I can get the 120fps by below command. For argus the problem should be the wrong sensor table be selected. Could you try to remove all of the sensor table form the DT and ov5693_mode_table.h to try.

gst-launch-1.0 nvcamerasrc num-buffers=200 sensor-id=0 ! 'video/x-raw(memory:NVMM), width=1280, height=720, framerate=(fraction)120/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=(string)I420, framerate=(fraction)120/1' ! nvtee ! omxh264enc quality-level=0 bitrate=20000000 ! qtmux ! filesink location=cm.mp4
3

Hi Shane and thank you for your answer,

Indeed the Gstreamer pipelines work well but I’m trying to use the Argus library.

I didn’t understand what you meant by:

  1. Can you tell me what is the “DT” ?

  2. And also, I didn’t find any “ov5693_mode_table.h”. How can I remove the sensor table ?
    Do I need to recompile the ov5693 driver ?

I’m using a L4T sample from the 24.2.1 Release: ~/tegra_multimedia_api/samples/10_camera_recording.

Thanks.

4

YassineH,

OV5693 Mode Tables

The ‘ov5693_mode_table.h’ is located in the kernel source code specifically located here:

/drivers/media/i2c/ov5693_mode_tables.h

Inside are sets of instructions that the TX1 feeds into the OV5693 camera to configure it to behave in one way or another such as configuring it to send images at 120FPS at a resolution of 1280x720.

DT or Device Tree

The ‘DT’, or the device tree is basically a configuration file that is fed into the kernel at boot. This is useful because it means that we don’t need to recompile the kernel if we want to enable or disable features of the kernel, just modify the config file. If you want to see the camera modes that are exposed you can look in the kernel source at

/arch/arm64/boot/dts/tegra210-platform/tegra210-camera-e3326-a00dtsi

Near the top of the file you’ll see

i2c@546c0000 {
      status = "okay";
      #address-cells = <1>;
      #size-cells = <0>;
      ov5693_c@36 {
        compatible = "nvidia,ov5693";
        /* I2C device address */
        reg = <0x36>;
...

below this you’ll find all the modes. The modes listed there are:

mode0{ // OV5693_MODE_2592X1944
...
          max_framerate = "30";
...
},
mode1{ // OV5693_MODE_2592X1458
...
          max_framerate = "30";
...
}
mode2{ // OV5693_MODE_1280X720
...
          max_framerate = "120";
...
}
..

Another way to explore the device tree is by going to the directory: /proc/device-tree on your TX1

My system is a little different because I have three cameras on and I’m not using the OV5693 but if I remember correctly the modes for the OV5693 are located at:

/proc/device-tree/host1x/i2c@546c0000/ov5693_c@36

There should be directories for each mode, you can navigate to a something like:

/proc/device-tree/host1x/i2c@546c0000/ov5693_c@36/mode2

and ‘cat’ a value like:

cd /proc/device-tree/host1x/i2c@546c0000/ov5693_c@36/mode2
cat max_framerate

will output ‘120’

Running the 10_camera_recording demo

If I recall correctly when I ran the ‘10_camera_recording’ demo it generated something like an ‘output.h264’. Does your application do that?

I just ran mine and got this output:

Set governor to performance before enabling profiler
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
Sensor_LoadModeModeType: mode 0: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 0: Failed to load pixeltype
Sensor_LoadModeModeType: mode 1: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 1: Failed to load pixeltype
Sensor_LoadModeModeType: mode 2: Failed to load pixeltype
Sensor_LoadModePixelPhase: mode 2: Failed to load pixeltype
PRODUCER: Creating output stream
PRODUCER: Launching consumer thread
Failed to query video capabilities: Bad address
NvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
create vidoe encoder return true
CONSUMER: Waiting until producer is connected...
PRODUCER: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
CONSUMER: Got EOS, exiting...
CONSUMER: Done.
PRODUCER: Done -- exiting.
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
************************************
Total Profiling Time = 0 sec
************************************

Even with all those fugly errors on the bottom it still generated an ‘output.h264’ which is correct. You can try and view the video using ‘gst-play-1.0’

gst-play-1.0 output.h264

But the video will be basically unwatchable because the H.264 video stream is not wrapped up in a container like an mp4 or mkv file. This wrapper will tell the codec how fast to play the video and without it the gst-play-1.0 will play as fast as it possibly can… entertaining but not useful.

Better Examples

There are other examples that are more sophisticated within the ~/tegra_multimedia_api/argus/samples specifically the ‘gstVideoEncode’ or the camera app in ~/tegra_multimedai_api/argus/apps/camera

The build process is not as straight forward and you will need to build these examples using the instructions provided in ~/tegra_multimedai_api/argus/README.txt

Note one thing that is strange is that the executable generated by the camera application ends up in: ~/tegra_multimedia_api/argus/build/apps/camera/ui/camera/argus_camera of all places!

Perhaps I messed up when I configured my cmake.

This was probably too much information but hopefully you may find something useful.

Dave

5

Hello Dave and thanks a lot for your answer.

The information you gave me were very interesting, but I haven’t found any of these files except : /proc/device-tree/host1x/i2c@546c0000/ov5693_c@36/mode*

Indeed when I run the sample it also generate an “output.h264” but it is only 30fps.
Every samples using libargus that I have tried are not acquiring frames at 120fps, even in 640x480.

The Gstreamer pipelines work but don’t use libargus.
Has someone already used Argus to capture frames at 120 fps ?

Yassine

6

Hi YassineH
There’s a argus_camera sample app that can select the 720p@60fps sensor mode and encode it. Below command can gen a 720P@60 video h264 file. The sample app locate tegra_multimedia_sample/argus/apps

./argus_camera --sensormode=2 --framerate=120 -v10 -x

7

Hi Shane,
Thank you for your answer I’ll try this sample.
Will this command allow me to capture frames at 120fps using Argus ?

8

Yes, it will select 720p@120fps sensor mode and encode as video.

9

Hi YassineH,

Have you made progress on trying the sample app?
Any result can be shared?

Thanks

10

Hi Kay,

Sorry I’m out of officie until June, so I can’t use the TX1 for the moment.
I will share my results as soon as I’ll be back. Sorry for the delay.

11

Hi kayccc,

I tested the argus_canera app you recommend some comments above. The app is located in:
tegra_multimedia_sample/argus/apps/camera/ui/camera

I run the following command (Because I had a camera sensor that supports 240fps):

./argus_camera --verbose --sensormode=2 --framerate=240 -v10 -x

But the app gets freezed and show some error log messages:

(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 132)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 101)
Error generated. /home/ubuntu/tegra_multimedia_api/argus/apps/camera/modules/Dispatcher.cpp, createOutputStream:1525 Failed to create OutputStream
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 132)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 101)
Error generated. /home/ubuntu/tegra_multimedia_api/argus/apps/camera/modules/EventThread.cpp, threadExecute:66 Failed to get iEventQueue
Error generated. /home/ubuntu/tegra_multimedia_api/argus/samples/utils/Thread.cpp, threadFunction:132 (propagating)

Then I have to kill the app with: kill PID

I take a look on the --help output and I figure out that it only supports a maximum framerate of 60 fps. Here is the output log:

Usage: ./argus_camera [OPTION]... [ACTION]... 
Press 'Ctrl-Up' to increase the focus position, press 'Ctrl-Down' to decrease the focus
position.
Press 'Esc' to exit.
Press 'm' to toggle between modules (still capture, video recording, multi exposure, 
multi session).
Press 'space' to execute the module action (capture an image, start and stop recording,
start and stop video playback.
The supported value range of some settings is device or sensor mode dependent.
Use the '--info' option to get a list of the supported values.
Options are set and actions are executed in the order they occur. Multiple
actions can be executed.
Mandatory arguments to long options are mandatory for short options too.

Options:
      --module=MODULE   switch to module MODULE. Valid strings are 'Capture',
                        'Video', 'Multi Exposure', 'Multi Session', 'Gallery' or
                        an index in the range [0, 4]. Default is 'Capture'.
      --verbose         enable verbose mode.  Default is '0'.
      --kpi             enable kpi mode.  Default is '0'.
  -d, --device=INDEX    select camera device with INDEX. Valid values need to be
                        in the range [0, 2]. Default is '0'.
      --exposuretimerange=RANGE
                        sets the exposure time range to RANGE, in nanoseconds.
                        Valid values need to be in the range [15000,236610000,
                        15000,236610000]. Default is '15000,236610000'.
      --gainrange=RANGE sets the gain range to RANGE. Valid values need to be in
                        the range [1,22, 1,22]. Default is '1,22'.
      --sensormode=INDEX
                        set sensor mode to INDEX. Valid strings are '1:
                        3840x2160', '2: 1920x1080', '3: 1280x540' or an index in
                        the range [0, 2]. Default is '1: 3840x2160'.
      --framerate=RATE  set the sensor frame rate to RATE. If RATE is 0 then VFR
                        (variable frame rate) is enabled. Valid values need to
                        be in the range [0.236608, 60]. Default is '60'.
      --focusposition=POSITION
                        sets the focus position to POSITION, in focuser units.
                        Valid values need to be in the range [0, 0]. Default is
                        '0'.
      --outputsize=WIDTHxHEIGHT
                        set the still and video output size to WIDTHxHEIGHT
                        (e.g. 1920x1080). If WIDTHxHEIGHT is '0x0' the output
                        size is the sensor mode size.  Default is '0x0'.
      --outputpath=PATH set the output file path. A file name, an incrementing
                        index and the file extension will be appended. E.g.
                        setting 'folder/' will result in 'folder/image0.jpg' or
                        'folder/video0.mp4'. '/dev/null' can be used to discard
                        output.  Default is '.'.
      --vstab=MODE      set the video stabilization mode. Valid strings are
                        'off', 'on' or an index in the range [0, 1]. Default is
                        'off'.
      --denoise=MODE    set the denoising mode. Valid strings are 'off', 'fast',
                        'highquality' or an index in the range [0, 2]. Default
                        is 'off'.
      --aeantibanding=MODE
                        set the auto exposure antibanding mode. Valid strings
                        are 'off', 'auto', '50hz', '60hz' or an index in the
                        range [0, 3]. Default is 'auto'.
      --aelock=LOCK     set the auto exposure lock.  Default is '0'.
      --awblock=LOCK    set the auto white balance lock.  Default is '0'.
      --awb=MODE        set the auto white balance mode. Valid strings are
                        'off', 'auto', 'incandescent', 'fluorescent',
                        'warmfluorescent', 'daylight', 'cloudydaylight',
                        'twilight', 'shade', 'manual' or an index in the range
                        [0, 9]. Default is 'auto'.
      --exposurecompensation=COMPENSATION
                        set the exposure compensation to COMPENSATION. Valid
                        values need to be in the range [-10, 10]. Default is
                        '0'.
      --defog=ENABLE    set the DeFog enable flag to ENABLE.  Default is '0'.
      --defogamount=AMOUNT
                        sets the amount of fog to be removed to AMOUNT. Valid
                        values need to be in the range [0, 1]. Default is '0.9'.
      --defogquality=QUALITY
                        sets the quality of the DeFog effect to QUALITY. Valid
                        values need to be in the range [0, 1]. Default is
                        '0.14285'.
      --videobitrate=RATE
                        set the video bit rate mode to RATE. If RATE is zero a
                        reasonable default is selected. Valid values need to be
                        in the range [0, 4294967295]. Default is '0'.
      --videoformat=FORMAT
                        set the video format. Valid strings are 'h263', 'h264',
                        'h265', 'vp8', 'mpeg4' or an index in the range [0, 4].
                        Default is 'h264'.
      --videofiletype=TYPE
                        set the video file type. For 'h265' set the file type as
                        'mkv' since 'h265' is only supported by the 'mkv'
                        container. Valid strings are 'mp4', '3gp', 'avi', 'mkv',
                        'h265' or an index in the range [0, 4]. Default is
                        'mp4'.
      --exposurerange=RANGE
                        set the exposure range to RANGE. Valid values need to be
                        in the range [-10,10, -10,10]. Default is '-2,2'.
      --exposuresteps=COUNT
                        sample the exposure range at COUNT steps. Valid values
                        need to be in the range [2, 3]. Default is '3'.

Actions:
  -h, --help            display this help and exit
  -x, --exit            exit from the program
  -i, --info            print information on devices.
      --loadconfig[=FILE]
                        load configuration from XML FILE. Default for file is
                        'argusAppConfig.xml'.
      --saveconfig[=FILE]
                        save configuration to XML FILE. Default for file is
                        'argusAppConfig.xml'.
  -s, --still[=COUNT]   do COUNT still captures and save as jpg files. If COUNT
                        is not specified do one still capture.
  -v, --video=DURATION  record video for DURATION seconds and save to a file.