Porting a Custom Camera from AGX Xavier to Orin NX

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
22

In vi5_setup_surface() width and height are set twice:

    u32 height = chan->format.height;
    u32 width = chan->format.width;

and

    if (chan->valid_ports > NVCSI_STREAM_1) {
            height = chan->gang_height;
            width = chan->gang_width;

In both cases when running the command

v4l2-ctl --set-fmt-video=width=720,height=480 --stream-mmap

I get

[ 67.014247] width = 720, height = 480 at line 433
[ 67.014248] width = 720, height = 480 at line 441

So, the frame dimensions seem right in vi5_setup_surface(). They also show up correctly in that spot for our other frame dimensions, 3840x2160, 1920x1080, and 1280x720.

I note that at that point in vi5_setup_surface() that format = 19, bpl = 1440, and data_type = 30. format of 19 looks like it’s UYVY (TEGRA_IMAGE_FORMAT_T_U8_Y8__V8_Y8). bpl of 1440 is consistent with width of 720. data_type of 30 looks like it’s TEGRA_IMAGE_DT_YUV422_8, which seems accurate.

23

Could you replace the camera_rtcpu_t234_rce,img from the JP5 to confirm.
Also confirm JP5 on Orin NX.

Thanks

24

I don’t have a way to test our input device with JP5 on Orin NX. The board is from a 3rd party and they have only put JP6 on it.

I replaced the camera-rtcpu-t234-rce.img in bootloader/ with the one from my JP5 (from when I was doing AGX Xavier work), and I get the same result. The 3840x2160, 1920x1080, and 1280x720 all work, but the 720x480 doesn’t. It still spits out on the Linux console:

[ 588.816384] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256

One interesting thing is that with the JP5 version there are some additional messages visible with dmesg. They occur before the above messages start:

[ 588.583744] tegra30_mc_handle_irq: 514 callbacks suppressed
[ 588.583753] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.583821] tegra-mc 2c00000.memory-controller: vi2falr: read @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.583841] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.583953] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256
[ 588.649501] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.649573] tegra-mc 2c00000.memory-controller: vi2falr: read @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.649595] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.649863] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256
[ 588.682830] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.682905] tegra-mc 2c00000.memory-controller: vi2falr: read @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.682927] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 588.683104] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256
[ 588.716158] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)

25

Sorry for the incorrect information.
Should be camera_rtcpu_t194_rce,img for Xavier.

Thanks

26

The system I tested the JP5 camera_rtcpu_txxx file on was Orin NX based, so I think the file you originally suggested was correct.

27

Can you confirm the RCE by below message.

sudo dmesg | grep -i rtcpu
28

Here’s the response. If this is related to the version of camera-rtcpu-t234-rce.img, then I believe this to be JP6.

$ sudo dmesg | grep -i rtcpu
[ 7.608199] tegra186-cam-rtcpu bc00000.rtcpu: Adding to iommu group 40
[ 7.610190] tegra186-cam-rtcpu bc00000.rtcpu: Trace buffer configured at IOVA=0xbff00000
[ 7.626380] tegra-ivc-bus bc00000.rtcpu:ivc-bus: region 0: iova=0xbfec0000-0xbfee027f size=131712
[ 7.626421] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: echo: ver=0 grp=1 RX[16x64]=0x1000-0x1480 TX[16x64]=0x1480-0x1900
[ 7.626458] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: dbg: ver=0 grp=1 RX[1x512]=0x1900-0x1b80 TX[1x512]=0x1b80-0x1e00
[ 7.626654] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: dbg: ver=0 grp=1 RX[1x8192]=0x1e00-0x3e80 TX[1x8192]=0x3e80-0x5f00
[ 7.626684] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivccontrol: ver=0 grp=1 RX[64x320]=0x5f00-0xaf80 TX[64x320]=0xaf80-0x10000
[ 7.626711] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivccapture: ver=0 grp=1 RX[512x64]=0x10000-0x18080 TX[512x64]=0x18080-0x20100
[ 7.626736] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: diag: ver=0 grp=1 RX[1x64]=0x20100-0x201c0 TX[1x64]=0x201c0-0x20280
[ 7.627408] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (217)
[ 7.627411] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful
[ 7.628039] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=e2238c99959d2df9350d393f04e1f44e5bef98cb
[ 7.628375] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: ivc channel driver missing
[ 7.628415] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: ivc channel driver missing
[ 7.628517] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: ivc channel driver missing
[ 7.628520] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivc channel driver missing
[ 7.628522] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivc channel driver missing
[ 7.628524] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: ivc channel driver missing

29

Please compare the sha1 with original to confirm updated successfully.

30

When using the JP5 version of camera-rtcpu-t234-rce.img:

[ 8.643558] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=10150ec03f8849d1537396903b70d4a48a255aba

For the whole rtcpu log:

[ 8.638218] tegra186-cam-rtcpu bc00000.rtcpu: Adding to iommu group 40
[ 8.639096] tegra186-cam-rtcpu bc00000.rtcpu: Trace buffer configured at IOVA=0xbff00000
[ 8.642350] tegra186-cam-rtcpu bc00000.rtcpu: RTCPU trace: IOVM setup error: 130
[ 8.642357] tegra186-cam-rtcpu bc00000.rtcpu: trace boot sync failed: -5
[ 8.642993] tegra-ivc-bus bc00000.rtcpu:ivc-bus: region 0: iova=0xbfec0000-0xbfee027f size=131712
[ 8.643050] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: echo: ver=0 grp=1 RX[16x64]=0x1000-0x1480 TX[16x64]=0x1480-0x1900
[ 8.643091] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: dbg: ver=0 grp=1 RX[1x512]=0x1900-0x1b80 TX[1x512]=0x1b80-0x1e00
[ 8.643116] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: dbg: ver=0 grp=1 RX[1x8192]=0x1e00-0x3e80 TX[1x8192]=0x3e80-0x5f00
[ 8.643139] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivccontrol: ver=0 grp=1 RX[64x320]=0x5f00-0xaf80 TX[64x320]=0xaf80-0x10000
[ 8.643163] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivccapture: ver=0 grp=1 RX[512x64]=0x10000-0x18080 TX[512x64]=0x18080-0x20100
[ 8.643194] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: diag: ver=0 grp=1 RX[1x64]=0x20100-0x201c0 TX[1x64]=0x201c0-0x20280
[ 8.643358] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (217)
[ 8.643360] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful
[ 8.643558] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=10150ec03f8849d1537396903b70d4a48a255aba
[ 8.643569] tegra186-cam-rtcpu bc00000.rtcpu: RTCPU trace: IOVM setup error: 130
[ 8.643570] tegra186-cam-rtcpu bc00000.rtcpu: trace boot sync failed: -5
[ 8.643801] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: ivc channel driver missing
[ 8.643802] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: ivc channel driver missing
[ 8.643803] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: ivc channel driver missing
[ 8.643804] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivc channel driver missing
[ 8.643805] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivc channel driver missing
[ 8.643806] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: ivc channel driver missing

Again with the JP5 I get errors like this in the kernel log as it starts the v4l2-ctl command to read frames, either accepted or discarded frames:

[ 492.809780] tegra30_mc_handle_irq: 413 callbacks suppressed
[ 492.809798] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 492.811152] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)
[ 492.811232] tegra-mc 2c00000.memory-controller: vi2falw: write @0x0000007ffc230000: EMEM address decode error (EMEM decode error)


and 720x480 frames aren’t accepted by the Orin NX with JP5 either.

31

Is there anything else you can suggest that will point toward a solution?

32

Have 480p and 576p video MIPI input operation been confirmed to work on Orin?

33

There’s size limited by Orin.

34

Thanks for your response, ShaneCCC.

Can you clarify? Is there a lower size limit as well as an upper size limit?
Is 720x480 @ 30fps (NTSC) not supported because it is too low a resolution?
What about 720x576 at 25fps (PAL)?

35

Sorry for the typo
I mean there’s no size limited for Orin.

Thanks

36

I instrumented the error message the driver puts out to see if I could learn anything further about the reason the driver is discarding frames. The capture status descr->status.status indicates a value of 14:

[ 1158.082732] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256, status = 14
[ 1158.182526] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256, status = 14

The status value of 14 corresponds to CAPTURE_STATUS_FALCON_ERROR, I believe. In the camrtc-capture.h file it indicates there are falcon error bits that will give details. Those appear to be in notify_bits. Correct?

Printing those too indicates a hex value of 200000000, or bit 33:

[  142.925601] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256, status = 14, notify_bits = 200000000
[  142.959134] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 256, status = 14, notify_bits = 200000000

Bit 33 looks to be CAPTURE_STATUS_NOTIFY_BIT_CHANSEL_PIXEL_LONG_LINE, or the line is too long. Our NTSC-width lines should be 720 pixels. I tried bumping up the size to 721, but then the notify_bits then indicate bit 34:

[  137.459995] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 512, status = 14, notify_bits = 400000000
[  137.493249] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 256, flags: 0, err_data 512, status = 14, notify_bits = 400000000

Bit 34 indicates the line is too short. So it seems to indicate that for some reason the 720 pixel width is correct, but is getting rejected for being too long.

For another test I tweaked the driver and device tree by setting the width to 721 pixels and ran the same two tests:

v4l2-ctl  --set-fmt-video=width=721,height=480 --stream-mmap

which yielded status = 14, notify_bits = 400000000 or “Pixel line is too short” meaning to me that the lines are less than 721 pixels.

v4l2-ctl  --set-fmt-video=width=720,height=480 --stream-mmap

which yielded status = 14, notify_bits = 200000000 or “Pixel line is too long” meaning to me that the lines are more than 720 pixels.

How can 720 be too short and 721 be too long? Is there something else that can cause those bits to be set under certain circumstances?

Can you help solve this puzzle?

Thanks!