Porting a Custom Camera from AGX Xavier to Orin NX

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
22

In vi5_setup_surface() width and height are set twice:

    u32 height = chan->format.height;
    u32 width = chan->format.width;

and

    if (chan->valid_ports > NVCSI_STREAM_1) {
            height = chan->gang_height;
            width = chan->gang_width;

In both cases when running the command

v4l2-ctl --set-fmt-video=width=720,height=480 --stream-mmap

I get

[ 67.014247] width = 720, height = 480 at line 433
[ 67.014248] width = 720, height = 480 at line 441

So, the frame dimensions seem right in vi5_setup_surface(). They also show up correctly in that spot for our other frame dimensions, 3840x2160, 1920x1080, and 1280x720.

I note that at that point in vi5_setup_surface() that format = 19, bpl = 1440, and data_type = 30. format of 19 looks like it’s UYVY (TEGRA_IMAGE_FORMAT_T_U8_Y8__V8_Y8). bpl of 1440 is consistent with width of 720. data_type of 30 looks like it’s TEGRA_IMAGE_DT_YUV422_8, which seems accurate.

23

Could you replace the camera_rtcpu_t234_rce,img from the JP5 to confirm.
Also confirm JP5 on Orin NX.

Thanks

24

I don’t have a way to test our input device with JP5 on Orin NX. The board is from a 3rd party and they have only put JP6 on it.

I replaced the camera-rtcpu-t234-rce.img in bootloader/ with the one from my JP5 (from when I was doing AGX Xavier work), and I get the same result. The 3840x2160, 1920x1080, and 1280x720 all work, but the 720x480 doesn’t. It still spits out on the Linux console:

[ 588.816384] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256

One interesting thing is that with the JP5 version there are some additional messages visible with dmesg. They occur before the above messages start:

[ 588.583744] tegra30_mc_handle_irq: 514 callbacks suppressed
[ 588.583753] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.583821] tegra-mc 2c00000.memory-controller: vi2falr: read @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.583841] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.583953] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256
[ 588.649501] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.649573] tegra-mc 2c00000.memory-controller: vi2falr: read @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.649595] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.649863] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256
[ 588.682830] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.682905] tegra-mc 2c00000.memory-controller: vi2falr: read @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.682927] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.683104] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256
[ 588.716158] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)

25

Sorry for the incorrect information.
Should be camera_rtcpu_t194_rce,img for Xavier.

Thanks

26

The system I tested the JP5 camera_rtcpu_txxx file on was Orin NX based, so I think the file you originally suggested was correct.

27

Can you confirm the RCE by below message.

sudo dmesg | grep -i rtcpu
28

Here’s the response. If this is related to the version of camera-rtcpu-t234-rce.img, then I believe this to be JP6.

$ sudo dmesg | grep -i rtcpu
[ 7.608199] tegra186-cam-rtcpu bc00000.rtcpu: Adding to iommu group 40
[ 7.610190] tegra186-cam-rtcpu bc00000.rtcpu: Trace buffer configured at IOVA=0xbff00000
[ 7.626380] tegra-ivc-bus bc00000.rtcpu:ivc-bus: region 0: iova=0xbfec0000-0xbfee027f size=131712
[ 7.626421] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: echo: ver=0 grp=1 RX[16x64]=0x1000-0x1480 TX[16x64]=0x1480-0x1900
[ 7.626458] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: dbg: ver=0 grp=1 RX[1x512]=0x1900-0x1b80 TX[1x512]=0x1b80-0x1e00
[ 7.626654] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: dbg: ver=0 grp=1 RX[1x8192]=0x1e00-0x3e80 TX[1x8192]=0x3e80-0x5f00
[ 7.626684] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivccontrol: ver=0 grp=1 RX[64x320]=0x5f00-0xaf80 TX[64x320]=0xaf80-0x10000
[ 7.626711] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivccapture: ver=0 grp=1 RX[512x64]=0x10000-0x18080 TX[512x64]=0x18080-0x20100
[ 7.626736] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: diag: ver=0 grp=1 RX[1x64]=0x20100-0x201c0 TX[1x64]=0x201c0-0x20280
[ 7.627408] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (217)
[ 7.627411] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful
[ 7.628039] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=e2238c99959d2df9350d393f04e1f44e5bef98cb
[ 7.628375] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: ivc channel driver missing
[ 7.628415] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: ivc channel driver missing
[ 7.628517] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: ivc channel driver missing
[ 7.628520] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivc channel driver missing
[ 7.628522] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivc channel driver missing
[ 7.628524] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: ivc channel driver missing

29

Please compare the sha1 with original to confirm updated successfully.

30

When using the JP5 version of camera-rtcpu-t234-rce.img:

[ 8.643558] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=10150ec03f8849d1537396903b70d4a48a255aba

For the whole rtcpu log:

[ 8.638218] tegra186-cam-rtcpu bc00000.rtcpu: Adding to iommu group 40
[ 8.639096] tegra186-cam-rtcpu bc00000.rtcpu: Trace buffer configured at IOVA=0xbff00000
[ 8.642350] tegra186-cam-rtcpu bc00000.rtcpu: RTCPU trace: IOVM setup error: 130
[ 8.642357] tegra186-cam-rtcpu bc00000.rtcpu: trace boot sync failed: -5
[ 8.642993] tegra-ivc-bus bc00000.rtcpu:ivc-bus: region 0: iova=0xbfec0000-0xbfee027f size=131712
[ 8.643050] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: echo: ver=0 grp=1 RX[16x64]=0x1000-0x1480 TX[16x64]=0x1480-0x1900
[ 8.643091] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: dbg: ver=0 grp=1 RX[1x512]=0x1900-0x1b80 TX[1x512]=0x1b80-0x1e00
[ 8.643116] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: dbg: ver=0 grp=1 RX[1x8192]=0x1e00-0x3e80 TX[1x8192]=0x3e80-0x5f00
[ 8.643139] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivccontrol: ver=0 grp=1 RX[64x320]=0x5f00-0xaf80 TX[64x320]=0xaf80-0x10000
[ 8.643163] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivccapture: ver=0 grp=1 RX[512x64]=0x10000-0x18080 TX[512x64]=0x18080-0x20100
[ 8.643194] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: diag: ver=0 grp=1 RX[1x64]=0x20100-0x201c0 TX[1x64]=0x201c0-0x20280
[ 8.643358] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (217)
[ 8.643360] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful
[ 8.643558] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=10150ec03f8849d1537396903b70d4a48a255aba
[ 8.643569] tegra186-cam-rtcpu bc00000.rtcpu: RTCPU trace: IOVM setup error: 130
[ 8.643570] tegra186-cam-rtcpu bc00000.rtcpu: trace boot sync failed: -5
[ 8.643801] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: ivc channel driver missing
[ 8.643802] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: ivc channel driver missing
[ 8.643803] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: ivc channel driver missing
[ 8.643804] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivc channel driver missing
[ 8.643805] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivc channel driver missing
[ 8.643806] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: ivc channel driver missing

Again with the JP5 I get errors like this in the kernel log as it starts the v4l2-ctl command to read frames, either accepted or discarded frames:

[ 492.809780] tegra30_mc_handle_irq: 413 callbacks suppressed
[ 492.809798] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 492.811152] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 492.811232] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)


and 720x480 frames aren’t accepted by the Orin NX with JP5 either.

31

Is there anything else you can suggest that will point toward a solution?

32

Have 480p and 576p video MIPI input operation been confirmed to work on Orin?

33

There’s size limited by Orin.

34

Thanks for your response, ShaneCCC.

Can you clarify? Is there a lower size limit as well as an upper size limit?
Is 720x480 @ 30fps (NTSC) not supported because it is too low a resolution?
What about 720x576 at 25fps (PAL)?

35

Sorry for the typo
I mean there’s no size limited for Orin.

Thanks

36

I instrumented the error message the driver puts out to see if I could learn anything further about the reason the driver is discarding frames. The capture status descr->status.status indicates a value of 14:

[ 1158.082732] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256, status = 14
[ 1158.182526] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256, status = 14

The status value of 14 corresponds to CAPTURE_STATUS_FALCON_ERROR, I believe. In the camrtc-capture.h file it indicates there are falcon error bits that will give details. Those appear to be in notify_bits. Correct?

Printing those too indicates a hex value of 200000000, or bit 33:

[  142.925601] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256, status = 14, notify_bits = 200000000
[  142.959134] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256, status = 14, notify_bits = 200000000

Bit 33 looks to be CAPTURE_STATUS_NOTIFY_BIT_CHANSEL_PIXEL_LONG_LINE, or the line is too long. Our NTSC-width lines should be 720 pixels. I tried bumping up the size to 721, but then the notify_bits then indicate bit 34:

[  137.459995] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 512, status = 14, notify_bits = 400000000
[  137.493249] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 512, status = 14, notify_bits = 400000000

Bit 34 indicates the line is too short. So it seems to indicate that for some reason the 720 pixel width is correct, but is getting rejected for being too long.

For another test I tweaked the driver and device tree by setting the width to 721 pixels and ran the same two tests:

v4l2-ctl  --set-fmt-video=width=721,height=480 --stream-mmap

which yielded status = 14, notify_bits = 400000000 or “Pixel line is too short” meaning to me that the lines are less than 721 pixels.

v4l2-ctl  --set-fmt-video=width=720,height=480 --stream-mmap

which yielded status = 14, notify_bits = 200000000 or “Pixel line is too long” meaning to me that the lines are more than 720 pixels.

How can 720 be too short and 721 be too long? Is there something else that can cause those bits to be set under certain circumstances?

Can you help solve this puzzle?

Thanks!

37

Does NVidia have any suggestions?

38

That’s weird!!
Maybe consider flash JP5 on Orin NX to confirm due to the VI driver is totally the same for Xavier and Orin serial.

Thanks

39

I agree, ShaneCCC. It really is weird!

Since I wrote earlier, we were able to revive our AGX Xavier system that uses JP5 and the same video input interface card. We have been unable to get it to properly receive 720x480 (NTSC) video either. It also accepts HD video at 720p, 1080p and 2160p, but not 480p, like the Orin NX does. So apparently the same issue existed there. I was sure I had tested and confirmed it operational back a year or two ago when I was working on it, but perhaps it was a “senior moment”. In any case, running it on JP5 (at least on Xavier) doesn’t appear to resolve the issue.

Are there other tests you can recommend to help isolate/illuminate the rejection mechanism?

41

How about confirm JP6 on AGX Xavier.

42

I have looked a little further at the short line and long line notify_bits cases described above, and have run some additional experiments printing more info that about the vi5 frame sizes and discarding of frames. It looks like when I got the line-too-short error previously it may have been because the drivers selected a 3840x2160 frame instead of the 720x480 frame.

Running tests confirming that the specified resolution/mode is selected, then adjusting the active width consistently in both the device-tree and driver I have gotten:

width range
720 to 735 – too long: CAPTURE_STATUS_NOTIFY_BIT_CHANSEL_PIXEL_LONG_LINE
736 – CAPTURE_STATUS_NOTIFY_BIT_CHANSEL_EMBED_MISSING_LE
737 to 800 – too short: CAPTURE_STATUS_NOTIFY_BIT_CHANSEL_PIXEL_SHORT_LINE

It’s saying there’s a missing Embedded line end, but it’s configured as embedded_metadata_height = "0";
Why would that occur?

It does seem to select the correct mode from the struct camera_common_frmfmt table and from the device tree modeX.

Another strange observation is that if I don’t include the resolution for UHD (3840x2160) in the struct camera_common_frmfmt table and in the device tree modes that the HD resolutions don’t work. They give notify_bits values of 40000000004, which is bit 2 and bit 42. Bit 42 (CAPTURE_STATUS_NOTIFY_BIT_CHANSEL_PIX_SHORT) says it means “Short frame: frame has too few pixel lines”. Bit 2 indicates CAPTURE_STATUS_NOTIFY_BIT_CSIMUX_FRAME_FS_FAULT, indicating “A FS packet was received while already in frame indicating that a FE packet was lost”. I don’t understand why these errors would only show up if a higher resolution is not present.

Are there other places I should be checking for the expected frame size or selected format information?