Nvv4l2av1enc keyFrames not working

razvanphp · March 31, 2025, 2:37pm

Related to Nvv4l2h264enc NACK / PLI / FIR event handling and AV1/RTSP Streaming Not Working - #18 by DaneLLL, can you please tell us what that test libtegrav4l2.so contains?

I’m guessing it enables keyFrames, as this is what I can see in Chrome when decoding succeeds, but I’m not sure it’s compatible with my newly added forceKeyFrame event, can you please confirm if this works for AV1 hw encoder as well?

Also, can you make this an option in nvv4l2av1enc as well, similar to insert-sps-pps=true?

Currently the AV1 encoder does not always send those keyframe, I have to restart it a few times for the stream to work, I think there is a race-condition happening.

Thank you!
R

DaneLLL · April 1, 2025, 1:01am

Hi,
The property insert-seq-hdr is added to nvv4l2av1enc in Jetpack 6.1. Do you mean force-IDR does not work in AV1 encoding?

razvanphp · April 1, 2025, 5:41am

Yes, I’m testing AV1 with JP6.0GA and there are no keyframes generated with the automatic PLI events (encoder recompiled with my patch), nor automatic ones. When using the test .so from the forum, 2 out of 4 times keyframes are generated, but not on every start.

Can you tell us what the changelog was regarding this in the latest releases and how to best configure this?

DaneLLL · April 1, 2025, 6:07am

Hi,
Are you able to try Jetpack 6.1 or 6.2? Would suggest use latest Jetpack release. And please try to set forceKeyFrame with an interval, such as > 30 frames.

razvanphp · April 1, 2025, 8:42am

Unfortunately, I’m not able to upgrade JP right now because of a driver dependancy, but I downloaded the gst-nvvideo4linux2_src source code and replaced gstv4l2av1enc.c+h in my JP and it compiles with the new option added. Is there other dependancy that I should backport? libtegrav4l2.so changes are needed for this to work?

I did many tests and most of the time the encoder just stops without any error (image freezes after 20-30 frames), but sometimes it just works and then I can see the keyframesdecoded growing.

The pipeline does not have anything special:

            nvv4l2av1enc name=encoder1 bitrate=22000000 maxperf-enable=1 insert-seq-hdr=true min-force-key-unit-interval=333333333 idrinterval=15 iframeinterval=256 !
            av1parse disable-passthrough=true ! rtpav1pay pt=96

Any clues what I could try to debug this?

DaneLLL · April 1, 2025, 9:34am

Hi,
Please share a method to replicate the issue. We will check it on Jetpack 6.1/6.2. Would still encourage to use later Jetpack release.

H264/H265 should be good on 6.0GA. We will check AV1 on later version.

razvanphp · April 3, 2025, 12:13pm

So, I was able to discover some more clues on what might happen, even with videotestsrc, so it’s not camera / drivers related.

There are 3 cases that happen randomly:

it works perfectly fine at 90 fps
It works, but sends only 6 fps out of 90 fps
It works for a few seconds (20-30 frames) and then completely crashes, not sending any byte after that

For case 2. I saw this in the kernel logs:

Apr  3 11:57:16 orin60 kernel: [185601.106499] nvidia_smmu_context_fault_bank: 11785 callbacks suppressed
Apr  3 11:57:16 orin60 kernel: [185601.106519] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.106590] tegra30_mc_handle_irq: 24188 callbacks suppressed
Apr  3 11:57:16 orin60 kernel: [185601.106593] tegra-mc 2c00000.memory-controller: nvencsrd: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
Apr  3 11:57:16 orin60 kernel: [185601.106599] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: VPR violation ((null))
Apr  3 11:57:16 orin60 kernel: [185601.106825] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.106893] tegra-mc 2c00000.memory-controller: nvencsrd: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
Apr  3 11:57:16 orin60 kernel: [185601.106897] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: VPR violation ((null))
Apr  3 11:57:16 orin60 kernel: [185601.107125] tegra-mc 2c00000.memory-controller: nvencsrd: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
Apr  3 11:57:16 orin60 kernel: [185601.107132] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: VPR violation ((null))
Apr  3 11:57:16 orin60 kernel: [185601.107226] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.107297] tegra-mc 2c00000.memory-controller: unknown: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
Apr  3 11:57:16 orin60 kernel: [185601.107302] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: VPR violation ((null))
Apr  3 11:57:16 orin60 kernel: [185601.107469] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.107539] tegra-mc 2c00000.memory-controller: nvencsrd: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
Apr  3 11:57:16 orin60 kernel: [185601.107542] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: VPR violation ((null))
Apr  3 11:57:16 orin60 kernel: [185601.107794] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.108118] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.108439] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.108764] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.109088] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2
Apr  3 11:57:16 orin60 kernel: [185601.109425] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ff3a6c000, fsynr=0x3a0003, cbfrsynra=0x435, cb=2

For case 3 I see those:

Apr  3 11:59:30 orin60 kernel: [185735.450135] ---- syncpts ----
Apr  3 11:59:30 orin60 kernel: [185735.450158] id 0 (reserved) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450165] id 1 (1-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450171] id 2 (2-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450177] id 3 (3-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450182] id 4 (4-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450188] id 5 (5-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450194] id 6 (6-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450199] id 7 (7-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450205] id 8 (8-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450211] id 9 (9-15340000.vic) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450217] id 10 (10-15480000.nvdec) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450222] id 11 (11-154c0000.nvenc) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450227] id 12 (12-15380000.nvjpg) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450230] id 13 (13-15540000.nvjpg) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450234] id 14 (14-15a50000.ofa) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450239] id 15 (15-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450243] id 16 (16-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450247] id 17 (17-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450251] id 18 (18-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450254] id 19 (19-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450258] id 20 (20-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450261] id 21 (21-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450265] id 22 (22-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450268] id 23 (23-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450272] id 24 (24-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450275] id 25 (25-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450278] id 26 (26-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450282] id 27 (27-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450285] id 28 (28-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450289] id 29 (29-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450292] id 30 (30-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450297] id 31 (31-ga10b_511) min 1160 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450300] id 32 (32-ga10b_510) min 13 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450304] id 33 (33-ga10b_509) min 14 max 10 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450308] id 34 (34-ga10b_508) min 4 max 0 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450313] id 35 (35-videotestsrc0:s) min 4527573 max 4527573 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450317] id 36 (36-ga10b_499_user) min 16885575 max 16885575 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450321] id 37 (37-ga10b_479_user) min 32004 max 32004 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450325] id 38 (38-ga10b_478_user) min 32005 max 32005 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450329] id 39 (39-ga10b_495_user) min 32006 max 32006 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450334] id 40 (40-ga10b_497_user) min 32003 max 32003 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450338] id 41 (41-ga10b_481_user) min 32006 max 32006 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450342] id 42 (42-ga10b_506_user) min 32004 max 32004 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450345] id 43 (43-ga10b_507_user) min 32004 max 32004 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450350] id 44 (44-ga10b_498_user) min 32006 max 32006 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450354] id 45 (45-ga10b_501_user) min 32002 max 32002 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450357] id 46 (46-ga10b_488_user) min 32002 max 32002 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450361] id 47 (47-ga10b_489_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450365] id 48 (48-ga10b_494_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450369] id 49 (49-ga10b_482_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450373] id 50 (50-ga10b_474_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450377] id 51 (51-ga10b_505_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450381] id 52 (52-videotestsrc14:) min 166154 max 166154 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450385] id 53 (53-videotestsrc15:) min 2935463 max 2935463 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450391] id 54 (54-V4L2_EncThread) min 787593 max 787595 (2 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450397] id 55 (55-V4L2_EncThread) min 139899 max 139902 (2 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450401] id 56 (56-V4L2_EncThread) min 197556 max 197556 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.450405] id 57 (57-V4L2_EncThread) min 3166999 max 3166999 (0 waiters)
Apr  3 11:59:30 orin60 kernel: [185735.451455]
Apr  3 11:59:30 orin60 kernel: [185735.451457] ---- channels ----
Apr  3 11:59:30 orin60 kernel: [185735.451462] 0: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.451465] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.451467] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.451478] 0-15340000.vic:
Apr  3 11:59:30 orin60 kernel: [185735.451480] inactive
Apr  3 11:59:30 orin60 kernel: [185735.451480]
Apr  3 11:59:30 orin60 kernel: [185735.451482] 1: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.451485] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.451486] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.451496] 1-15340000.vic:
Apr  3 11:59:30 orin60 kernel: [185735.451498] active class 00, offset 0100
Apr  3 11:59:30 orin60 kernel: [185735.451502] DMASTART 0x0000007fffffe000, DMAEND 0x0000000000000ffc
Apr  3 11:59:30 orin60 kernel: [185735.451505] DMAPUT 00000070 DMAGET 00000070 DMACTL 00000000
Apr  3 11:59:30 orin60 kernel: [185735.451506] CHANNELSTAT 00
Apr  3 11:59:30 orin60 kernel: [185735.451508]
Apr  3 11:59:30 orin60 kernel: [185735.451510] 2: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.451513] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.451514] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.451524] 2-15480000.nvdec:
Apr  3 11:59:30 orin60 kernel: [185735.451526] inactive
Apr  3 11:59:30 orin60 kernel: [185735.451526]
Apr  3 11:59:30 orin60 kernel: [185735.451528] 3: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.451530] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.451531] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.451541] 3-15480000.nvdec:
Apr  3 11:59:30 orin60 kernel: [185735.451542] inactive
Apr  3 11:59:30 orin60 kernel: [185735.451542]
Apr  3 11:59:30 orin60 kernel: [185735.451544] 4: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.451546] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.451547] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.451557] 4-154c0000.nvenc:
Apr  3 11:59:30 orin60 kernel: [185735.451559] inactive
Apr  3 11:59:30 orin60 kernel: [185735.451559]
Apr  3 11:59:30 orin60 kernel: [185735.451561] 5: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.451563] CMDFIFO_STAT 00004000
Apr  3 11:59:30 orin60 kernel: [185735.451566] CMDFIFO_RDATA 20000000
Apr  3 11:59:30 orin60 kernel: [185735.451576] 5-154c0000.nvenc:
Apr  3 11:59:30 orin60 kernel: [185735.451577] active class 01, offset 0050
Apr  3 11:59:30 orin60 kernel: [185735.451580] DMASTART 0x0000007fffffa000, DMAEND 0x0000000000000ffc
Apr  3 11:59:30 orin60 kernel: [185735.451582] DMAPUT 000004b0 DMAGET 00000494 DMACTL 00000000
Apr  3 11:59:30 orin60 kernel: [185735.451584] CHANNELSTAT 00
Apr  3 11:59:30 orin60 kernel: [185735.451589] JOB, syncpt 54: 787594 timeout: 10000 num_slots: 15 num_handles: 1
Apr  3 11:59:30 orin60 kernel: [185735.451592]     0x0000007fffffa3c0: e0000021: ACQUIRE_MLOCK(index=33)
Apr  3 11:59:30 orin60 kernel: [185735.451599]     0x0000007fffffa3c4: 00000840: SETCL(class=021)
Apr  3 11:59:30 orin60 kernel: [185735.451602]     0x0000007fffffa3c8: 90000000: SETPYLD(data=0000)
Apr  3 11:59:30 orin60 kernel: [185735.451606]     0x0000007fffffa3cc: 7000000c: SETSTRMID(offset=00000c)
Apr  3 11:59:30 orin60 kernel: [185735.451609]     0x0000007fffffa3d0: 20000001: NONINCR(offset=000, [00001036])
Apr  3 11:59:30 orin60 kernel: [185735.451614]     0x0000007fffffa3d8: 004e0045: SETCL(class=001, offset=04e, mask=05, [000c0489, 00000036])
Apr  3 11:59:30 orin60 kernel: [185735.451622]     0x0000007fffffa3e4: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451626]     0x0000007fffffa3e8: 00000840: SETCL(class=021)
Apr  3 11:59:30 orin60 kernel: [185735.451629]     0x0000007fffffa3ec: 90000035: SETPYLD(data=0035)
Apr  3 11:59:30 orin60 kernel: [185735.451632]     0x0000007fffffa3f0: 7000000c: SETSTRMID(offset=00000c)
Apr  3 11:59:30 orin60 kernel: [185735.451635]     0x0000007fffffa3f4: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451639]     0x0000007fffffa3f8: 10100001: INCR(offset=010, [000000c8])
Apr  3 11:59:30 orin60 kernel: [185735.451643]     0x0000007fffffa400: 10110001: INCR(offset=011, [3fec3aa0])
Apr  3 11:59:30 orin60 kernel: [185735.451648]     0x0000007fffffa408: c000005c: GATHER_W(count=005c, addr=[ffe8c000, 0000007f])
Apr  3 11:59:30 orin60 kernel: [185735.451654]     0x0000007fffffa414: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451658]     0x0000007fffffa418: 20000001: NONINCR(offset=000, [00001036])
Apr  3 11:59:30 orin60 kernel: [185735.451662]     0x0000007fffffa420: 004e0045: SETCL(class=001, offset=04e, mask=05, [000c048b, 00000036])
Apr  3 11:59:30 orin60 kernel: [185735.451668]     0x0000007fffffa42c: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451672]     0x0000007fffffa430: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451676]     0x0000007fffffa434: e1000021: RELEASE_MLOCK(index=33)
Apr  3 11:59:30 orin60 kernel: [185735.451684]   GATHER at 0x0000007fffe8c000+0x0, 92 words
Apr  3 11:59:30 orin60 kernel: [185735.451686]     0x0000007fffe8c000: 10100002: INCR(offset=010, [00000080, 00000007])
Apr  3 11:59:30 orin60 kernel: [185735.451693]     0x0000007fffe8c00c: 10100002: INCR(offset=010, [000001c0, 12001803])
Apr  3 11:59:30 orin60 kernel: [185735.451699]     0x0000007fffe8c018: 10100002: INCR(offset=010, [000001c1, 00000005])
Apr  3 11:59:30 orin60 kernel: [185735.451706]     0x0000007fffe8c024: 10100002: INCR(offset=010, [000001de, 00000000])
Apr  3 11:59:30 orin60 kernel: [185735.451713]     0x0000007fffe8c030: 10100002: INCR(offset=010, [000001df, 00000000])
Apr  3 11:59:30 orin60 kernel: [185735.451719]     0x0000007fffe8c03c: 10100002: INCR(offset=010, [000001e0, 3fef3650])
Apr  3 11:59:30 orin60 kernel: [185735.451725]     0x0000007fffe8c048: 10100002: INCR(offset=010, [000001c6, 3fef2480])
Apr  3 11:59:30 orin60 kernel: [185735.451731]     0x0000007fffe8c054: 10100002: INCR(offset=010, [000001c9, 3fef2bf0])
Apr  3 11:59:30 orin60 kernel: [185735.451738]     0x0000007fffe8c060: 10100002: INCR(offset=010, [000001cd, bff38000])
Apr  3 11:59:30 orin60 kernel: [185735.451744]     0x0000007fffe8c06c: 10100002: INCR(offset=010, [000001d0, bff37800])
Apr  3 11:59:30 orin60 kernel: [185735.451750]     0x0000007fffe8c078: 10100002: INCR(offset=010, [00000120, 3ff3fa00])
Apr  3 11:59:30 orin60 kernel: [185735.451756]     0x0000007fffe8c084: 10100002: INCR(offset=010, [000001ce, 3fee5f00])
Apr  3 11:59:30 orin60 kernel: [185735.451786]     0x0000007fffe8c090: 10100002: INCR(offset=010, [000001cf, 3fee5f80])
Apr  3 11:59:30 orin60 kernel: [185735.451792]     0x0000007fffe8c09c: 10100002: INCR(offset=010, [00000100, bfff7000])
Apr  3 11:59:30 orin60 kernel: [185735.451799]     0x0000007fffe8c0a8: 10100002: INCR(offset=010, [00000110, bffe6000])
Apr  3 11:59:30 orin60 kernel: [185735.451805]     0x0000007fffe8c0b4: 10100002: INCR(offset=010, [00000101, bfffb000])
Apr  3 11:59:30 orin60 kernel: [185735.451811]     0x0000007fffe8c0c0: 10100002: INCR(offset=010, [00000111, bfff8000])
Apr  3 11:59:30 orin60 kernel: [185735.451818]     0x0000007fffe8c0cc: 10100002: INCR(offset=010, [000001cc, bffff000])
Apr  3 11:59:30 orin60 kernel: [185735.451824]     0x0000007fffe8c0d8: 10100002: INCR(offset=010, [000001d3, bfffc000])
Apr  3 11:59:30 orin60 kernel: [185735.451830]     0x0000007fffe8c0e4: 10100002: INCR(offset=010, [000001c8, 3ff6fc00])
Apr  3 11:59:30 orin60 kernel: [185735.451837]     0x0000007fffe8c0f0: 10100002: INCR(offset=010, [00000124, 3ffb0000])
Apr  3 11:59:30 orin60 kernel: [185735.451843]     0x0000007fffe8c0fc: 10100002: INCR(offset=010, [00000125, 3fecff00])
Apr  3 11:59:30 orin60 kernel: [185735.451849]     0x0000007fffe8c108: 10100002: INCR(offset=010, [00000126, 3fecfe80])
Apr  3 11:59:30 orin60 kernel: [185735.451856]     0x0000007fffe8c114: 10100002: INCR(offset=010, [00000127, 3fecff80])
Apr  3 11:59:30 orin60 kernel: [185735.451863]     0x0000007fffe8c120: 10100002: INCR(offset=010, [00000122, 3fef1860])
Apr  3 11:59:30 orin60 kernel: [185735.451869]     0x0000007fffe8c12c: 10100002: INCR(offset=010, [00000123, 3ff3fce0])
Apr  3 11:59:30 orin60 kernel: [185735.451876]     0x0000007fffe8c138: 10100002: INCR(offset=010, [000001c7, 3ffa4000])
Apr  3 11:59:30 orin60 kernel: [185735.451882]     0x0000007fffe8c144: 10100002: INCR(offset=010, [000001c4, 3ff3fc80])
Apr  3 11:59:30 orin60 kernel: [185735.451888]     0x0000007fffe8c150: 10100002: INCR(offset=010, [00000121, 3ff3fe00])
Apr  3 11:59:30 orin60 kernel: [185735.451894]     0x0000007fffe8c15c: 10100002: INCR(offset=010, [000000c0, 00000100])
Apr  3 11:59:30 orin60 kernel: [185735.451901]     0x0000007fffe8c168: 20000001: NONINCR(offset=000, [00000436])
Apr  3 11:59:30 orin60 kernel: [185735.451907] JOB, syncpt 55: 139901 timeout: 10000 num_slots: 15 num_handles: 1
Apr  3 11:59:30 orin60 kernel: [185735.451909]     0x0000007fffffa438: e0000021: ACQUIRE_MLOCK(index=33)
Apr  3 11:59:30 orin60 kernel: [185735.451913]     0x0000007fffffa43c: 00000840: SETCL(class=021)
Apr  3 11:59:30 orin60 kernel: [185735.451916]     0x0000007fffffa440: 90000000: SETPYLD(data=0000)
Apr  3 11:59:30 orin60 kernel: [185735.451919]     0x0000007fffffa444: 7000000c: SETSTRMID(offset=00000c)
Apr  3 11:59:30 orin60 kernel: [185735.451922]     0x0000007fffffa448: 20000001: NONINCR(offset=000, [00001037])
Apr  3 11:59:30 orin60 kernel: [185735.451927]     0x0000007fffffa450: 004e0045: SETCL(class=001, offset=04e, mask=05, [0002227c, 00000037])
Apr  3 11:59:30 orin60 kernel: [185735.451933]     0x0000007fffffa45c: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451937]     0x0000007fffffa460: 00000840: SETCL(class=021)
Apr  3 11:59:30 orin60 kernel: [185735.451940]     0x0000007fffffa464: 90000035: SETPYLD(data=0035)
Apr  3 11:59:30 orin60 kernel: [185735.451943]     0x0000007fffffa468: 7000000c: SETSTRMID(offset=00000c)
Apr  3 11:59:30 orin60 kernel: [185735.451947]     0x0000007fffffa46c: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451951]     0x0000007fffffa470: 10100001: INCR(offset=010, [000000c8])
Apr  3 11:59:30 orin60 kernel: [185735.451955]     0x0000007fffffa478: 10110001: INCR(offset=011, [3fec3ba0])
Apr  3 11:59:30 orin60 kernel: [185735.451959]     0x0000007fffffa480: c000005c: GATHER_W(count=005c, addr=[ffe49000, 0000007f])
Apr  3 11:59:30 orin60 kernel: [185735.451965]     0x0000007fffffa48c: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451969]     0x0000007fffffa490: 20000001: NONINCR(offset=000, [00001037])
Apr  3 11:59:30 orin60 kernel: [185735.451973]     0x0000007fffffa498: 004e0045: SETCL(class=001, offset=04e, mask=05, [0002227e, 00000037])
Apr  3 11:59:30 orin60 kernel: [185735.451979]     0x0000007fffffa4a4: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451983]     0x0000007fffffa4a8: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:30 orin60 kernel: [185735.451987]     0x0000007fffffa4ac: e1000021: RELEASE_MLOCK(index=33)
Apr  3 11:59:30 orin60 kernel: [185735.451992]   GATHER at 0x0000007fffe49000+0x0, 92 words
Apr  3 11:59:30 orin60 kernel: [185735.451995]     0x0000007fffe49000: 10100002: INCR(offset=010, [00000080, 00000007])
Apr  3 11:59:30 orin60 kernel: [185735.452001]     0x0000007fffe4900c: 10100002: INCR(offset=010, [000001c0, 12001803])
Apr  3 11:59:30 orin60 kernel: [185735.452007]     0x0000007fffe49018: 10100002: INCR(offset=010, [000001c1, 00000004])
Apr  3 11:59:30 orin60 kernel: [185735.452014]     0x0000007fffe49024: 10100002: INCR(offset=010, [000001de, 00000000])
Apr  3 11:59:30 orin60 kernel: [185735.452020]     0x0000007fffe49030: 10100002: INCR(offset=010, [000001df, 00000000])
Apr  3 11:59:30 orin60 kernel: [185735.452026]     0x0000007fffe4903c: 10100002: INCR(offset=010, [000001e0, 3fef3490])
Apr  3 11:59:30 orin60 kernel: [185735.452033]     0x0000007fffe49048: 10100002: INCR(offset=010, [000001c6, 3feabf40])
Apr  3 11:59:30 orin60 kernel: [185735.452039]     0x0000007fffe49054: 10100002: INCR(offset=010, [000001c9, 3fef2c60])
Apr  3 11:59:30 orin60 kernel: [185735.452046]     0x0000007fffe49060: 10100002: INCR(offset=010, [000001cd, bfeb8000])
Apr  3 11:59:30 orin60 kernel: [185735.452052]     0x0000007fffe4906c: 10100002: INCR(offset=010, [000001d0, bfeb4000])
Apr  3 11:59:30 orin60 kernel: [185735.452058]     0x0000007fffe49078: 10100002: INCR(offset=010, [00000120, 3ff37600])
Apr  3 11:59:30 orin60 kernel: [185735.452064]     0x0000007fffe49084: 10100002: INCR(offset=010, [000001ce, 3ff6cc00])
Apr  3 11:59:30 orin60 kernel: [185735.452071]     0x0000007fffe49090: 10100002: INCR(offset=010, [000001cf, 3ff6dc00])
Apr  3 11:59:30 orin60 kernel: [185735.452077]     0x0000007fffe4909c: 10100002: INCR(offset=010, [00000100, bffa8000])
Apr  3 11:59:30 orin60 kernel: [185735.452083]     0x0000007fffe490a8: 10100002: INCR(offset=010, [00000110, bff90000])
Apr  3 11:59:30 orin60 kernel: [185735.452090]     0x0000007fffe490b4: 10100002: INCR(offset=010, [00000101, bffe8000])
Apr  3 11:59:30 orin60 kernel: [185735.452096]     0x0000007fffe490c0: 10100002: INCR(offset=010, [00000111, bffc0000])
Apr  3 11:59:30 orin60 kernel: [185735.452103]     0x0000007fffe490cc: 10100002: INCR(offset=010, [000001cc, bff88000])
Apr  3 11:59:30 orin60 kernel: [185735.452109]     0x0000007fffe490d8: 10100002: INCR(offset=010, [000001d3, bff70000])
Apr  3 11:59:30 orin60 kernel: [185735.452115]     0x0000007fffe490e4: 10100002: INCR(offset=010, [000001c8, 3ff6ce00])
Apr  3 11:59:30 orin60 kernel: [185735.452121]     0x0000007fffe490f0: 10100002: INCR(offset=010, [00000124, 3ff40000])
Apr  3 11:59:30 orin60 kernel: [185735.452128]     0x0000007fffe490fc: 10100002: INCR(offset=010, [00000125, 3ff3b700])
Apr  3 11:59:30 orin60 kernel: [185735.452134]     0x0000007fffe49108: 10100002: INCR(offset=010, [00000126, 3ff3b300])
Apr  3 11:59:30 orin60 kernel: [185735.452140]     0x0000007fffe49114: 10100002: INCR(offset=010, [00000127, 3ff37500])
Apr  3 11:59:30 orin60 kernel: [185735.452147]     0x0000007fffe49120: 10100002: INCR(offset=010, [00000122, 3ff3b4e0])
Apr  3 11:59:30 orin60 kernel: [185735.452153]     0x0000007fffe4912c: 10100002: INCR(offset=010, [00000123, 3ff3b480])
Apr  3 11:59:30 orin60 kernel: [185735.452159]     0x0000007fffe49138: 10100002: INCR(offset=010, [000001c7, 3fed0000])
Apr  3 11:59:30 orin60 kernel: [185735.452166]     0x0000007fffe49144: 10100002: INCR(offset=010, [000001c4, 3ff37480])
Apr  3 11:59:30 orin60 kernel: [185735.452172]     0x0000007fffe49150: 10100002: INCR(offset=010, [00000121, 3ff37700])
Apr  3 11:59:30 orin60 kernel: [185735.452179]     0x0000007fffe4915c: 10100002: INCR(offset=010, [000000c0, 00000100])
Apr  3 11:59:30 orin60 kernel: [185735.452185]     0x0000007fffe49168: 20000001: NONINCR(offset=000, [00000437])
Apr  3 11:59:30 orin60 kernel: [185735.452190]
Apr  3 11:59:30 orin60 kernel: [185735.452192] 6: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.452194] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.452195] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.452205] 6-15380000.nvjpg:
Apr  3 11:59:30 orin60 kernel: [185735.452207] inactive
Apr  3 11:59:30 orin60 kernel: [185735.452207]
Apr  3 11:59:30 orin60 kernel: [185735.452209] 7: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.452211] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.452212] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.452222] 7-15380000.nvjpg:
Apr  3 11:59:30 orin60 kernel: [185735.452223] inactive
Apr  3 11:59:30 orin60 kernel: [185735.452223]
Apr  3 11:59:30 orin60 kernel: [185735.452226] 8: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.452229] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.452230] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.452240] 8-15540000.nvjpg:
Apr  3 11:59:30 orin60 kernel: [185735.452241] inactive
Apr  3 11:59:30 orin60 kernel: [185735.452241]
Apr  3 11:59:30 orin60 kernel: [185735.452243] 9: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.452245] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.452247] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.452256] 9-15540000.nvjpg:
Apr  3 11:59:30 orin60 kernel: [185735.452258] inactive
Apr  3 11:59:30 orin60 kernel: [185735.452258]
Apr  3 11:59:30 orin60 kernel: [185735.452260] 10: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.452263] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.452264] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.452274] 10-15a50000.ofa:
Apr  3 11:59:30 orin60 kernel: [185735.452276] inactive
Apr  3 11:59:30 orin60 kernel: [185735.452276]
Apr  3 11:59:30 orin60 kernel: [185735.452277] 11: fifo:
Apr  3 11:59:30 orin60 kernel: [185735.452280] CMDFIFO_STAT 00002040
Apr  3 11:59:30 orin60 kernel: [185735.452281] [empty]
Apr  3 11:59:30 orin60 kernel: [185735.452291] 11-15a50000.ofa:
Apr  3 11:59:30 orin60 kernel: [185735.452292] inactive
Apr  3 11:59:30 orin60 kernel: [185735.452292]
Apr  3 11:59:30 orin60 kernel: [185735.452301] tegra-host1x 13e00000.host1x: cdma_timeout_handler: timeout: 54 (54-V4L2_EncThread), HW thresh 787593, done 787594
Apr  3 11:59:40 orin60 kernel: [185745.689827] ---- syncpts ----
Apr  3 11:59:40 orin60 kernel: [185745.689847] id 0 (reserved) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689853] id 1 (1-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689857] id 2 (2-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689860] id 3 (3-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689864] id 4 (4-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689868] id 5 (5-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689872] id 6 (6-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689875] id 7 (7-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689879] id 8 (8-pva_syncpt) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689883] id 9 (9-15340000.vic) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689887] id 10 (10-15480000.nvdec) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689890] id 11 (11-154c0000.nvenc) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689894] id 12 (12-15380000.nvjpg) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689897] id 13 (13-15540000.nvjpg) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689901] id 14 (14-15a50000.ofa) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689906] id 15 (15-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689910] id 16 (16-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689914] id 17 (17-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689918] id 18 (18-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689921] id 19 (19-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689925] id 20 (20-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689928] id 21 (21-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689932] id 22 (22-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689935] id 23 (23-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689939] id 24 (24-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689942] id 25 (25-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689945] id 26 (26-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689949] id 27 (27-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689952] id 28 (28-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689956] id 29 (29-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689959] id 30 (30-nvkms-fence) min 0 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689964] id 31 (31-ga10b_511) min 1160 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689968] id 32 (32-ga10b_510) min 13 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689972] id 33 (33-ga10b_509) min 14 max 10 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689976] id 34 (34-ga10b_508) min 4 max 0 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689980] id 35 (35-videotestsrc0:s) min 4527573 max 4527573 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689985] id 36 (36-ga10b_499_user) min 16885575 max 16885575 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689989] id 37 (37-ga10b_479_user) min 32004 max 32004 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689993] id 38 (38-ga10b_478_user) min 32005 max 32005 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.689997] id 39 (39-ga10b_495_user) min 32006 max 32006 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690001] id 40 (40-ga10b_497_user) min 32003 max 32003 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690005] id 41 (41-ga10b_481_user) min 32006 max 32006 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690009] id 42 (42-ga10b_506_user) min 32004 max 32004 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690013] id 43 (43-ga10b_507_user) min 32004 max 32004 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690017] id 44 (44-ga10b_498_user) min 32006 max 32006 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690021] id 45 (45-ga10b_501_user) min 32002 max 32002 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690025] id 46 (46-ga10b_488_user) min 32002 max 32002 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690029] id 47 (47-ga10b_489_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690033] id 48 (48-ga10b_494_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690037] id 49 (49-ga10b_482_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690041] id 50 (50-ga10b_474_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690045] id 51 (51-ga10b_505_user) min 32000 max 32000 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690050] id 52 (52-videotestsrc14:) min 166154 max 166154 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690055] id 53 (53-videotestsrc15:) min 2935463 max 2935463 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690060] id 54 (54-V4L2_EncThread) min 787593 max 787595 (2 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690066] id 55 (55-V4L2_EncThread) min 139899 max 139902 (2 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690070] id 56 (56-V4L2_EncThread) min 197556 max 197556 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.690074] id 57 (57-V4L2_EncThread) min 3166999 max 3166999 (0 waiters)
Apr  3 11:59:40 orin60 kernel: [185745.691104]
Apr  3 11:59:40 orin60 kernel: [185745.691106] ---- channels ----
Apr  3 11:59:40 orin60 kernel: [185745.691110] 0: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691113] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691115] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691126] 0-15340000.vic:
Apr  3 11:59:40 orin60 kernel: [185745.691128] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691128]
Apr  3 11:59:40 orin60 kernel: [185745.691131] 1: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691133] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691134] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691144] 1-15340000.vic:
Apr  3 11:59:40 orin60 kernel: [185745.691146] active class 00, offset 0100
Apr  3 11:59:40 orin60 kernel: [185745.691150] DMASTART 0x0000007fffffe000, DMAEND 0x0000000000000ffc
Apr  3 11:59:40 orin60 kernel: [185745.691153] DMAPUT 00000070 DMAGET 00000070 DMACTL 00000000
Apr  3 11:59:40 orin60 kernel: [185745.691156] CHANNELSTAT 00
Apr  3 11:59:40 orin60 kernel: [185745.691157]
Apr  3 11:59:40 orin60 kernel: [185745.691159] 2: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691162] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691163] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691174] 2-15480000.nvdec:
Apr  3 11:59:40 orin60 kernel: [185745.691175] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691175]
Apr  3 11:59:40 orin60 kernel: [185745.691177] 3: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691180] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691181] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691191] 3-15480000.nvdec:
Apr  3 11:59:40 orin60 kernel: [185745.691193] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691193]
Apr  3 11:59:40 orin60 kernel: [185745.691195] 4: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691197] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691198] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691208] 4-154c0000.nvenc:
Apr  3 11:59:40 orin60 kernel: [185745.691210] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691210]
Apr  3 11:59:40 orin60 kernel: [185745.691212] 5: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691214] CMDFIFO_STAT 00004000
Apr  3 11:59:40 orin60 kernel: [185745.691217] CMDFIFO_RDATA 20000000
Apr  3 11:59:40 orin60 kernel: [185745.691227] 5-154c0000.nvenc:
Apr  3 11:59:40 orin60 kernel: [185745.691229] active class 01, offset 0050
Apr  3 11:59:40 orin60 kernel: [185745.691231] DMASTART 0x0000007fffffa000, DMAEND 0x0000000000000ffc
Apr  3 11:59:40 orin60 kernel: [185745.691233] DMAPUT 000004b0 DMAGET 00000494 DMACTL 00000000
Apr  3 11:59:40 orin60 kernel: [185745.691235] CHANNELSTAT 00
Apr  3 11:59:40 orin60 kernel: [185745.691239] JOB, syncpt 55: 139901 timeout: 10000 num_slots: 15 num_handles: 1
Apr  3 11:59:40 orin60 kernel: [185745.691243]     0x0000007fffffa438: e0000021: ACQUIRE_MLOCK(index=33)
Apr  3 11:59:40 orin60 kernel: [185745.691249]     0x0000007fffffa43c: 00000840: SETCL(class=021)
Apr  3 11:59:40 orin60 kernel: [185745.691253]     0x0000007fffffa440: 90000000: SETPYLD(data=0000)
Apr  3 11:59:40 orin60 kernel: [185745.691256]     0x0000007fffffa444: 7000000c: SETSTRMID(offset=00000c)
Apr  3 11:59:40 orin60 kernel: [185745.691260]     0x0000007fffffa448: 20000001: NONINCR(offset=000, [00001037])
Apr  3 11:59:40 orin60 kernel: [185745.691265]     0x0000007fffffa450: 004e0045: SETCL(class=001, offset=04e, mask=05, [0002227c, 00000037])
Apr  3 11:59:40 orin60 kernel: [185745.691273]     0x0000007fffffa45c: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:40 orin60 kernel: [185745.691299]     0x0000007fffffa460: 00000840: SETCL(class=021)
Apr  3 11:59:40 orin60 kernel: [185745.691302]     0x0000007fffffa464: 90000035: SETPYLD(data=0035)
Apr  3 11:59:40 orin60 kernel: [185745.691305]     0x0000007fffffa468: 7000000c: SETSTRMID(offset=00000c)
Apr  3 11:59:40 orin60 kernel: [185745.691309]     0x0000007fffffa46c: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:40 orin60 kernel: [185745.691313]     0x0000007fffffa470: 10100001: INCR(offset=010, [000000c8])
Apr  3 11:59:40 orin60 kernel: [185745.691318]     0x0000007fffffa478: 10110001: INCR(offset=011, [3fec3ba0])
Apr  3 11:59:40 orin60 kernel: [185745.691322]     0x0000007fffffa480: c000005c: GATHER_W(count=005c, addr=[ffe49000, 0000007f])
Apr  3 11:59:40 orin60 kernel: [185745.691328]     0x0000007fffffa48c: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:40 orin60 kernel: [185745.691332]     0x0000007fffffa490: 20000001: NONINCR(offset=000, [00001037])
Apr  3 11:59:40 orin60 kernel: [185745.691337]     0x0000007fffffa498: 004e0045: SETCL(class=001, offset=04e, mask=05, [0002227e, 00000037])
Apr  3 11:59:40 orin60 kernel: [185745.691343]     0x0000007fffffa4a4: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:40 orin60 kernel: [185745.691347]     0x0000007fffffa4a8: 20000000: NONINCR(offset=000, [])
Apr  3 11:59:40 orin60 kernel: [185745.691351]     0x0000007fffffa4ac: e1000021: RELEASE_MLOCK(index=33)
Apr  3 11:59:40 orin60 kernel: [185745.691358]   GATHER at 0x0000007fffe49000+0x0, 92 words
Apr  3 11:59:40 orin60 kernel: [185745.691361]     0x0000007fffe49000: 10100002: INCR(offset=010, [00000080, 00000007])
Apr  3 11:59:40 orin60 kernel: [185745.691368]     0x0000007fffe4900c: 10100002: INCR(offset=010, [000001c0, 12001803])
Apr  3 11:59:40 orin60 kernel: [185745.691375]     0x0000007fffe49018: 10100002: INCR(offset=010, [000001c1, 00000004])
Apr  3 11:59:40 orin60 kernel: [185745.691381]     0x0000007fffe49024: 10100002: INCR(offset=010, [000001de, 00000000])
Apr  3 11:59:40 orin60 kernel: [185745.691388]     0x0000007fffe49030: 10100002: INCR(offset=010, [000001df, 00000000])
Apr  3 11:59:40 orin60 kernel: [185745.691394]     0x0000007fffe4903c: 10100002: INCR(offset=010, [000001e0, 3fef3490])
Apr  3 11:59:40 orin60 kernel: [185745.691401]     0x0000007fffe49048: 10100002: INCR(offset=010, [000001c6, 3feabf40])
Apr  3 11:59:40 orin60 kernel: [185745.691408]     0x0000007fffe49054: 10100002: INCR(offset=010, [000001c9, 3fef2c60])
Apr  3 11:59:40 orin60 kernel: [185745.691414]     0x0000007fffe49060: 10100002: INCR(offset=010, [000001cd, bfeb8000])
Apr  3 11:59:40 orin60 kernel: [185745.691421]     0x0000007fffe4906c: 10100002: INCR(offset=010, [000001d0, bfeb4000])
Apr  3 11:59:40 orin60 kernel: [185745.691427]     0x0000007fffe49078: 10100002: INCR(offset=010, [00000120, 3ff37600])
Apr  3 11:59:40 orin60 kernel: [185745.691434]     0x0000007fffe49084: 10100002: INCR(offset=010, [000001ce, 3ff6cc00])
Apr  3 11:59:40 orin60 kernel: [185745.691440]     0x0000007fffe49090: 10100002: INCR(offset=010, [000001cf, 3ff6dc00])
Apr  3 11:59:40 orin60 kernel: [185745.691446]     0x0000007fffe4909c: 10100002: INCR(offset=010, [00000100, bffa8000])
Apr  3 11:59:40 orin60 kernel: [185745.691453]     0x0000007fffe490a8: 10100002: INCR(offset=010, [00000110, bff90000])
Apr  3 11:59:40 orin60 kernel: [185745.691460]     0x0000007fffe490b4: 10100002: INCR(offset=010, [00000101, bffe8000])
Apr  3 11:59:40 orin60 kernel: [185745.691466]     0x0000007fffe490c0: 10100002: INCR(offset=010, [00000111, bffc0000])
Apr  3 11:59:40 orin60 kernel: [185745.691472]     0x0000007fffe490cc: 10100002: INCR(offset=010, [000001cc, bff88000])
Apr  3 11:59:40 orin60 kernel: [185745.691479]     0x0000007fffe490d8: 10100002: INCR(offset=010, [000001d3, bff70000])
Apr  3 11:59:40 orin60 kernel: [185745.691485]     0x0000007fffe490e4: 10100002: INCR(offset=010, [000001c8, 3ff6ce00])
Apr  3 11:59:40 orin60 kernel: [185745.691491]     0x0000007fffe490f0: 10100002: INCR(offset=010, [00000124, 3ff40000])
Apr  3 11:59:40 orin60 kernel: [185745.691498]     0x0000007fffe490fc: 10100002: INCR(offset=010, [00000125, 3ff3b700])
Apr  3 11:59:40 orin60 kernel: [185745.691504]     0x0000007fffe49108: 10100002: INCR(offset=010, [00000126, 3ff3b300])
Apr  3 11:59:40 orin60 kernel: [185745.691511]     0x0000007fffe49114: 10100002: INCR(offset=010, [00000127, 3ff37500])
Apr  3 11:59:40 orin60 kernel: [185745.691517]     0x0000007fffe49120: 10100002: INCR(offset=010, [00000122, 3ff3b4e0])
Apr  3 11:59:40 orin60 kernel: [185745.691524]     0x0000007fffe4912c: 10100002: INCR(offset=010, [00000123, 3ff3b480])
Apr  3 11:59:40 orin60 kernel: [185745.691530]     0x0000007fffe49138: 10100002: INCR(offset=010, [000001c7, 3fed0000])
Apr  3 11:59:40 orin60 kernel: [185745.691537]     0x0000007fffe49144: 10100002: INCR(offset=010, [000001c4, 3ff37480])
Apr  3 11:59:40 orin60 kernel: [185745.691543]     0x0000007fffe49150: 10100002: INCR(offset=010, [00000121, 3ff37700])
Apr  3 11:59:40 orin60 kernel: [185745.691550]     0x0000007fffe4915c: 10100002: INCR(offset=010, [000000c0, 00000100])
Apr  3 11:59:40 orin60 kernel: [185745.691556]     0x0000007fffe49168: 20000001: NONINCR(offset=000, [00000437])
Apr  3 11:59:40 orin60 kernel: [185745.691560]
Apr  3 11:59:40 orin60 kernel: [185745.691563] 6: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691565] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691567] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691577] 6-15380000.nvjpg:
Apr  3 11:59:40 orin60 kernel: [185745.691579] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691579]
Apr  3 11:59:40 orin60 kernel: [185745.691581] 7: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691583] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691584] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691594] 7-15380000.nvjpg:
Apr  3 11:59:40 orin60 kernel: [185745.691596] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691596]
Apr  3 11:59:40 orin60 kernel: [185745.691598] 8: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691600] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691602] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691612] 8-15540000.nvjpg:
Apr  3 11:59:40 orin60 kernel: [185745.691614] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691614]
Apr  3 11:59:40 orin60 kernel: [185745.691615] 9: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691618] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691619] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691629] 9-15540000.nvjpg:
Apr  3 11:59:40 orin60 kernel: [185745.691630] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691630]
Apr  3 11:59:40 orin60 kernel: [185745.691632] 10: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691635] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691636] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691646] 10-15a50000.ofa:
Apr  3 11:59:40 orin60 kernel: [185745.691647] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691647]
Apr  3 11:59:40 orin60 kernel: [185745.691650] 11: fifo:
Apr  3 11:59:40 orin60 kernel: [185745.691652] CMDFIFO_STAT 00002040
Apr  3 11:59:40 orin60 kernel: [185745.691653] [empty]
Apr  3 11:59:40 orin60 kernel: [185745.691663] 11-15a50000.ofa:
Apr  3 11:59:40 orin60 kernel: [185745.691664] inactive
Apr  3 11:59:40 orin60 kernel: [185745.691664]
Apr  3 11:59:40 orin60 kernel: [185745.691674] tegra-host1x 13e00000.host1x: cdma_timeout_handler: timeout: 55 (55-V4L2_EncThread), HW thresh 139899, done 139901

Errors on gstreamer when 2 happens:

Error parsing OBU data
Error parsing frame header
Error parsing OBU data
Error parsing frame header
Error parsing OBU data
Error parsing frame header
Error parsing OBU data

and errors on gstreamer when 3 (complete halt happens):

0:06:46.674550062 29758 0xffff00006240 WARN            v4l2videoenc gstv4l2videoenc.c:1776:gst_v4l2_video_enc_handle_frame:<encoder1> error: Failed to process frame.
0:06:46.674593903 29758 0xffff00006240 WARN            v4l2videoenc gstv4l2videoenc.c:1776:gst_v4l2_video_enc_handle_frame:<encoder1> error: Maybe be due to not enough memory or failing driver

I’m testing with this pipeline:

gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-raw, width=2712, height=1538, framerate=90/1' ! nvvidconv ! queue ! nvv4l2av1enc name=encoder1 bitrate=22000000 maxperf-enable=1 insert-seq-hdr=true idrinterval=15 iframeinterval=256 ! av1parse disable-passthrough=true ! rtpav1pay pt=96 ! udpsink host=192.168.X.X port=5000 sync=0

Receiving with:

gst-launch-1.0 udpsrc port=5000 caps="application/x-rtp, encoding-name=AV1, payload=96" ! rtpav1depay ! av1parse ! dav1ddec ! autovideosink

Resource usage is OK, enough free RAM and CPU left on the Orin NX.

Any hints on what might be happening? It’s clearly deep inside the encoder/hardware…

razvanphp · April 3, 2025, 5:10pm

… this seems to be related to Orin NX AV1 encoding limitations, since if I lower the resolution, it does not throw those errors anymore. I tried increasing both fps and bitrate and it still handles it, but somehow it does not like the 2k resolution.

Running even 2 processes like this in parallel works:

gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-raw, width=1920, height=1080, framerate=90/1' ! nvvidconv ! queue ! nvv4l2av1enc name=encoder1 bitrate=44000000 maxperf-enable=1 insert-seq-hdr=true idrinterval=256 iframeinterval=256 ! av1parse disable-passthrough=true ! rtpav1pay pt=96 ! fakesink

But running just one of this triggers the kernel errors:

gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-raw, width=2712, height=1538, framerate=90/1' ! nvvidconv ! queue ! nvv4l2av1enc name=encoder1 bitrate=22000000 maxperf-enable=1 insert-seq-hdr=true idrinterval=256 iframeinterval=256 ! av1parse disable-passthrough=true ! rtpav1pay pt=96 ! fakesink

DaneLLL · April 8, 2025, 3:28am

Hi,
On Orin NX developer kit(Orin NX module + Orin Nano carrier board) with Jetpack 6.2, we run the command and don’t hit any issue:

$ gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-raw,width=2712, height=1538, framerate=90/1' ! nvvidconv ! queue ! nvv4l2av1enc name=encoder1 bitrate=22000000 maxperf-enable=1 insert-seq-hdr=true idrinterval=256 iframeinterval=256 ! av1parse disable-passthrough=true ! rtpav1pay pt=96 ! fakesink

Do you have developer kit for a try? Would like to align the result.

Topic		Replies	Views
Kernel panic when camera streaming restarts Jetson AGX Orin camera	9	1003	November 13, 2023
Jetpack-4.6 tx2 : v4l2-ctl --stream-mmap needs nvarguscamerasrc to succeed Jetson TX2 camera	31	1916	November 10, 2021
No frame from IMX415 on orin devkit Jetson AGX Orin camera	3	530	November 6, 2023
Orin Nano and tc358743 capture issue Jetson Orin Nano camera	101	5129	March 18, 2024
When the camera driver was upgraded from r35.4.1 to r35.5.0, a kernel crash occurred when vi could not receive data Jetson AGX Orin camera	9	238	September 20, 2024
GMSL Camera fail on Orin JatPack 5.1.2 Jetson AGX Orin camera , board-design	4	347	March 20, 2024
Unable to stream CSI camera video under the new Jetpack 6.2 Jetson Orin NX camera	16	182	March 2, 2025
[L4T R32.5.1] VI got no reply from camera processor while capturing frames Jetson Xavier NX camera	6	736	October 18, 2021
OV2311 Custom Driver Issues Jetson Orin NX camera	7	531	January 31, 2024
4x 1080p60 V4L2src Capture+Encode gStreamer Performance Issues Jetson TX2	16	2606	November 28, 2018

Nvv4l2av1enc keyFrames not working

Related topics