12_camera_v4l2_cuda problem with mPCIe V4L2 device

anilyalchin · January 16, 2024, 1:00pm

I have a 1280x720 60fps monochrome HD-SDI camera that I am using together with a nanoGrabber HD-SDI frame grabber which creates a V4L2 (/dev/video0) interface for receiving the frames. The frame grabber is attached to an Elroy Carrier through a mPCIe port. The Jetson system in question is a TX2i.

I can successfully use this system together with Gstreamer or v4l2-ctl to capture and use the stream of raw frames. However, I am running into some problems regarding the time-tags received from Gstreamer being wrong and therefore considering migrating to the Jetson multimedia API in order to see if more low-level control over the data flow will solve my problem. I have downloaded and compiled the R32.2.0 version. With the camera attached and turned on, I can not get the 12th sample code to execute successfully. Here are some commands I have used and their results that I think may have diagnostic purposes. Any help would be appreciated!

This is the error message when trying to run the sample (I have also tried all of the different -f options, same result):

gftb@tegra-ubuntu:~/Desktop/Tegra_Multimedia_API_R32.2.0_aarch64/tegra_multimedia_api/samples/12_camera_v4l2_cuda$ ./camera_v4l2_cuda -d /dev/video0 -s 1280x720
ERROR: camera_initialize(): (line:272) Failed to set camera output format: Invalid argument (22)
ERROR: init_components(): (line:335) Failed to initialize camera device
ERROR: main(): (line:716) Failed to initialize v4l2 components
nvbuf_utils: dmabuf_fd 0 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
App run failed

This demonstrates the caps that Gstreamer uses while accessing the data stream:

gftb@tegra-ubuntu:~$ gst-launch-1.0 v4l2src ! xvimagesink -v
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)1280, height=(int)720, framerate=(fraction)120/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
Setting pipeline to PLAYING …
New clock: GstSystemClock
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)1280, height=(int)720, framerate=(fraction)120/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:14.638381536
Setting pipeline to PAUSED …
Setting pipeline to READY …
Caught SIGSEGV
^C

This shows some properties of the frame grabber and its driver:

gftb@tegra-ubuntu:~$ sudo lspci -xxxvvv
00:03.0 PCI bridge: NVIDIA Corporation Device 10e6 (rev a1) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 381
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00001000-00001fff
Memory behind bridge: 40200000-407fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: NVIDIA Corporation Device 0000
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
Mapping Address Base: 00000000fee00000
Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
LnkCtl: ASPM L0s Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd On, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet+ LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=30us PortTPowerOnTime=70us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=30us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=70us
Kernel driver in use: pcieport
00: de 10 e6 10 07 01 10 00 a1 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 11 11 00 00
20: 20 40 70 40 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 7d 01 00 00
40: 0d 48 00 00 de 10 00 00 01 50 03 f8 00 00 00 00
50: 05 60 82 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 08 80 00 a8 00 00 e0 fe 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 10 00 42 01 20 80 00 00 1f 29 00 00 12 3c 31 02
90: 41 00 11 30 00 00 00 00 c0 01 48 01 08 00 00 00
a0: 00 00 00 00 13 08 00 00 00 04 00 00 00 00 00 00
b0: 02 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

01:00.0 Multimedia video controller: Device 0907:0100 (rev 01)
Subsystem: Device 0907:0001
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 381
Region 0: Memory at 40400000 (32-bit, non-prefetchable) [size=4M]
Region 1: I/O ports at 1000 [size=256]
Region 4: I/O ports at 1100 [size=256]
Region 5: Memory at 40200000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM L0s Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Kernel driver in use: ngrabber
00: 07 09 00 01 07 00 10 00 01 00 00 04 00 00 00 00
10: 00 00 40 40 01 10 00 00 00 00 00 00 00 00 00 00
20: 01 11 00 00 00 00 20 40 00 00 00 00 07 09 01 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 7d 01 00 00
40: 01 50 c3 5b 00 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 80 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 02 00 01 87 00 00 10 20 00 00 11 3c 02 00
80: 41 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This shows that there are no directly controllable registers of the frame grabber device:

gftb@tegra-ubuntu:~$ v4l2-ctl --list-ctrls

DaneLLL · January 17, 2024, 2:41am

Hi,
Please run $ v4l2-ctl --list-formats-ext and share the prints. It looks like the format is YUV420 which is not supported in 12_camera_v4l2_cuda by default.

anilyalchin · January 17, 2024, 5:47am

The output is as follows:

gftb@tegra-ubuntu:~$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: ‘YU12’ (Planar YUV 4:2:0)

If it is not supported by default, is it possible to modify the code a bit to make it supported?

DaneLLL · January 17, 2024, 6:22am

Hi,
Please modify the sample:

Add the format to nvcolor_fmt[]
Hard code capture_dmabuf to false

Rebuild the sample and give it a try.

anilyalchin · January 17, 2024, 7:29am

Hello,

I have done the thing related to the capture_dmabuf.

However, the change I would need to make for nvcolor_fmt is not obvious, because there is already an entry related to YUV420. I’ve modified it to be like this, the part I have outlined in bold is what i have added:

static nv_color_fmt nvcolor_fmt =
{
// TODO add more pixel format mapping
{V4L2_PIX_FMT_UYVY, NvBufferColorFormat_UYVY},
{V4L2_PIX_FMT_VYUY, NvBufferColorFormat_VYUY},
{V4L2_PIX_FMT_YUYV, NvBufferColorFormat_YUYV},
{V4L2_PIX_FMT_YVYU, NvBufferColorFormat_YVYU},
{V4L2_PIX_FMT_GREY, NvBufferColorFormat_GRAY8},
{V4L2_PIX_FMT_YUV420M, NvBufferColorFormat_YUV420},
{V4L2_PIX_FMT_YUV420, NvBufferColorFormat_YUV420},
};

Of course, I’ve also modified some other parts of the code so that it now recognizes “-f YUV420” and the fact that it corresponds to V4L2_PIX_FMT_YUV420.

After these modifications, the error message changed to be like this, now a small black screen very briefly appears then disappears when I try executing the code:

gftb@tegra-ubuntu:~/Desktop/Tegra_Multimedia_API_R32.2.0_aarch64/tegra_multimedia_api/samples/12_camera_v4l2_cuda$ ./camera_v4l2_cuda -d /dev/video0 -s 1280x720 -f YUV420
[INFO] (NvEglRenderer.cpp:110) Setting Screen width 1280 height 720
ERROR: request_camera_buff_mmap(): (line:406) V4l2 buffer number is not as desired
ERROR: prepare_buffers(): (line:524) Failed to set up camera buff
ERROR: main(): (line:726) Failed to prepare v4l2 buffs
App run failed

The same error message happens regardless of if the camera is connected or not.

DaneLLL · January 17, 2024, 8:05am

Hi,
Please try the gstreamer command and see if camera preview is shown:

$ gst-launch-1.0 v4l2src ! video/x-raw, format=I420, width=1280, height=720 ! nvvidconv !  nvoverlaysink

To make sure the caps is correct in gstreamer command.

And please run $ cat /etc/nv_tegra_release to make sure you use r32.2. If you use later version, please install jetson_multimedia_api samples through SDKManager. If the version of system and samples is mismatched, it may hit segment fault or not work properly.

anilyalchin · January 17, 2024, 8:13am

Hello,

The given Gstreamer command runs correctly and displays the video stream received from the camera.

The given cat command displays this: R32 (release), REVISION: 2.3, GCID: 17644089, BOARD: t186ref, EABI: aarch64, DATE: Tue Nov 5 21:48:17 UTC 2019

DaneLLL · January 17, 2024, 8:43am

Hi,
So the versions are matched. The current error is

ERROR: request_camera_buff_mmap(): (line:406) V4l2 buffer number is not as desired

Please add prints in request_camera_buff_mmap() to debug further. Maybe the device does not support V4L2_MEMORY_MMAP, or V4L2_BUFFERS_NUM=4 is not enough.

anilyalchin · January 17, 2024, 10:58am

Hello,

The device supports both MMAP and DMABUF modes, as far as I know from Gstream.

I have tried playing around with different values of V4L2_BUFFERS_NUM, the default value is 4, only setting it to 2 allowed the program to run. However, I now notice two problems.
1- The video stream is completely green, despite the fact that it needs to be grayscale.
2- The program reports the received FPS to be 30 instead of the correct value of 60.
Edit: The 2nd problem has been solved by feeding the “-r 120” parameter, therefore only the 1st one remains.

DaneLLL · January 18, 2024, 4:31am

Hi,

This may be because the data in U/V planes are invalid, please call nvbuff_do_clearchroma() to clear the planes and see if the grayscale data shows up.

anilyalchin · January 22, 2024, 7:09am

Hello,

I have modified the code according to my use case, and can successfully retrieve the raw data from the V4L2 interface. However, I have another question I need to ask. As I have mentioned in my initial topic, the task I need to do with the frames is time-critical, and therefore I need the correct timestamps of each frame. I can just use clock_gettime() after polling for each frame, but I do not think this is the “correct/proper” way to handle this. I think the V4L2 device should be providing a timestamp that i should be able to retrieve together with the frames. However, I can not figure out how.

DaneLLL · January 22, 2024, 8:42am

Hi,
A possible solution is to get time in driver code and fill in timestamp to struct v4l2_buffer:
3.6. Buffers — The Linux Kernel documentation

So that you can get the kernel timestamp in userspace application.

system · February 14, 2024, 5:48am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Enabling camera on Jetson TX1 board Jetson TX1	73	49447	March 12, 2018
V4L2 with opencv in python Jetson TX2 opencv	8	13998	October 18, 2021
TX2 Camera convert/encode using Multimedia API issue Jetson TX2 camera , encoder	16	2025	October 18, 2021
V4L2 on Jetson TX2 Jetson TX2	44	19923	October 18, 2021
ov5693 through V4L2 interface not working in 23.2 Jetson TX1	26	11372	September 9, 2016
12_camera_v4l2_cuda works, but only after "warming up" with gstreamer Jetson Nano mmapi	10	891	October 15, 2021
NvBufferGetParams failed Jetson AGX Xavier camera , hw , gstreamer	31	1631	December 7, 2022
v4l2src is not working on tx2/? Jetson TX2	43	9744	October 18, 2021
MT9M021 Streaming Jetson TX1	45	6612	October 18, 2021
Gstreamer reports Raspberry Pi camera streaming at 120fps when in reality it is only 60fps Jetson Nano camera , gstreamer	53	3833	July 19, 2022

12_camera_v4l2_cuda problem with mPCIe V4L2 device

Related topics