GStreamer Transcoding for MJPEG video on Nano?

Hi all, loving this jetson nano board. The last step in my project is transcoding a raw stream mapping of MJPEG images (at ~25fps) to H264 using the hardware capabilities of this board.

I looked at the GStreamer PDF and I don’t see MJPEG video mentioned anywhere, so I went ahead and used the single image transcode as a jumping off point. This is what I’m using:

gst-launch-1.0 filesrc location=video.mkv ! image/jpeg, width=3840, height=2160, framerate=25/1 ! jpegparse ! jpegdec ! 'video/x-raw, format=(string)I420' ! omxh264enc ! matroskamux ! filesink location=video-compressed.mkv -e

The above doesn’t produce an error, but the FPS of the output file is around 1, or incredibly low. It should be 25-30. Also possibly unsolvable, but the transcode takes a while at about 50% real time speed. IIRC I was able to do a H264 to vp8 at 450% real time.

Am I doing something wrong with one of my pipelines?

Thanks in advance for any insights

EDIT: I think it was VP8 not vp9

Hi qcasey1, it looks like you are using the jpegdec element, which isn’t hardware accelerated.

Instead, try using the nvjpegdec element instead, which is hw-accelerated and included in the GStreamer PDF.

Thanks for the response, but when I use nvjpegdec I get a whole bunch of

NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 12, SyncPointValue = 0)

before gstreamer eventually exits. Also would this effect the output video’s frames per second?

Also it hangs my file transfers, is this a memory issue?

EDIT: my most recent dmesg shows this as a memory issue I think

[68497.308008] sd 0:0:0:0: [sda] tag#0 data out submit err -11 uas-tag 1 inflight: s-out a-cmd s-cmd work
[68497.308011] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x2a 2a 00 0e 8a aa 30 00 00 28 00
[68531.933836] jpegparse0:sink invoked oom-killer: gfp_mask=0x26080c0(GFP_KERNEL|__GFP_ZERO|__GFP_NOTRACK), nodemask=0, order=0, oom_score_adj=0
[68531.946611] jpegparse0:sink cpuset=/ mems_allowed=0
[68531.946624] CPU: 2 PID: 24381 Comm: jpegparse0:sink Not tainted 4.9.140-tegra #1
[68531.946627] Hardware name: NVIDIA Jetson Nano Developer Kit (DT)
[68531.946629] Call trace:
[68531.946638] [<ffffff800808bdb8>] dump_backtrace+0x0/0x198
[68531.946643] [<ffffff800808c37c>] show_stack+0x24/0x30
[68531.946649] [<ffffff800845d7e0>] dump_stack+0x98/0xc0
[68531.946654] [<ffffff8008259190>] dump_header+0x6c/0x1b8
[68531.946659] [<ffffff80081c9b88>] oom_kill_process+0x268/0x498
[68531.946663] [<ffffff80081ca264>] out_of_memory+0x1e4/0x308
[68531.946668] [<ffffff80081cfd8c>] __alloc_pages_nodemask+0x844/0xd38
[68531.946673] [<ffffff800820009c>] __pte_alloc+0x3c/0x128
[68531.946678] [<ffffff80082030b4>] alloc_set_pte+0x48c/0x550
[68531.946682] [<ffffff8008204d04>] handle_mm_fault+0x72c/0xa68
[68531.946687] [<ffffff80080a36b0>] do_page_fault+0x308/0x518
[68531.946691] [<ffffff80080a392c>] do_translation_fault+0x6c/0x80
[68531.946694] [<ffffff8008080954>] do_mem_abort+0x54/0xb0
[68531.946698] [<ffffff80080833c8>] el0_da+0x20/0x24
[68531.946700] Mem-Info:
[68531.946709] active_anon:18 inactive_anon:0 isolated_anon:0
                active_file:422 inactive_file:1203 isolated_file:0
                unevictable:3037 dirty:0 writeback:0 unstable:0
                slab_reclaimable:7231 slab_unreclaimable:12573
                mapped:157 shmem:0 pagetables:2154 bounce:0
                free:116139 free_pcp:162 free_cma:104001
[68531.946716] Node 0 active_anon:72kB inactive_anon:0kB active_file:1688kB inactive_file:4812kB unevictable:12148kB isolated(anon):0kB isolated(file):0kB mapped:628kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:1489 all_unreclaimable? no
[68531.946724] DMA free:442340kB min:22636kB low:28292kB high:33948kB active_anon:72kB inactive_anon:0kB active_file:1492kB inactive_file:4740kB unevictable:0kB writepending:0kB present:2076672kB managed:2042692kB mlocked:0kB slab_reclaimable:876kB slab_unreclaimable:1156kB kernel_stack:416kB pagetables:972kB bounce:0kB free_pcp:648kB local_pcp:0kB free_cma:416004kB
[68531.946726] lowmem_reserve[]: 0 1961 1961 1961
[68531.946743] Normal free:22216kB min:22416kB low:28020kB high:33624kB active_anon:0kB inactive_anon:0kB active_file:72kB inactive_file:72kB unevictable:12148kB writepending:0kB present:2082816kB managed:2008620kB mlocked:0kB slab_reclaimable:28048kB slab_unreclaimable:49136kB kernel_stack:3552kB pagetables:7644kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[68531.946745] lowmem_reserve[]: 0 0 0 0
[68531.946755] DMA: 250*4kB (MEC) 30*8kB (MEC) 16*16kB (MEC) 384*32kB (UMEC) 1*64kB (E) 2*128kB (EC) 2*256kB (EC) 2*512kB (EC) 3*1024kB (UEC) 7*2048kB (UEC) 100*4096kB (C) = 442648kB
[68531.946798] Normal: 774*4kB (ME) 682*8kB (UME) 204*16kB (UMH) 291*32kB (UMH) 3*64kB (U) 5*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 22216kB
[68531.946835] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[68531.946837] 4754 total pagecache pages
[68531.946840] 15 pages in swap cache
[68531.946842] Swap cache stats: add 911696, delete 912013, find 669808/950576
[68531.946844] Free swap  = 1802344kB
[68531.946846] Total swap = 2025648kB
[68531.946849] 1039872 pages RAM
[68531.946851] 0 pages HighMem/MovableOnly
[68531.946852] 27044 pages reserved
[68531.946854] 118784 pages cma reserved
[68531.946857] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[68531.946876] [ 1993]     0  1993     8635        0      14       4      222             0 systemd-journal
[68531.946882] [ 2658]     0  2658     3884        0       8       3      387         -1000 systemd-udevd
[68531.946887] [ 3512] 62583  3512    20988        2      11       3      164             0 systemd-timesyn
[68531.946891] [ 3513]     0  3513     1601        0       7       3      128             0 rpcbind
[68531.946895] [ 3514]   102  3514     2639        0       9       4      183             0 systemd-resolve
[68531.946899] [ 3544]     0  3544     1920        2       7       4      793             0 haveged
[68531.946903] [ 3711]     0  3711    78217        0      22       5      428             0 ModemManager
[68531.946907] [ 3714]   109  3714    54992        0      12       4      878             0 rsyslogd
[68531.946911] [ 3796]     0  3796    25003        2      17       3     1993             0 networkd-dispat
[68531.946915] [ 3801]   114  3801     1622        0       7       3      253             0 avahi-daemon
[68531.946918] [ 3855]     0  3855     2603        2       9       3      184             0 systemd-logind
[68531.946922] [ 3924]     0  3924     1660        0       7       4       94             0 cron
[68531.946926] [ 3934]   103  3934     1799        2       8       3      249          -900 dbus-daemon
[68531.946930] [ 4062]   114  4062     1490        0       7       3       85             0 avahi-daemon
[68531.946935] [ 4166]     0  4166   100370        0      31       4      843             0 NetworkManager
[68531.946939] [ 4180]     0  4180     2354        0       9       3      148             0 wpa_supplicant
[68531.946942] [ 4326]     0  4326    57866        0      15       3      227             0 polkitd
[68531.946947] [ 4698]   111  4698    61942        0      23       4      960             0 whoopsie
[68531.946951] [ 4799]     0  4799   324867        0      34       5      129             0 nvs-service
[68531.946955] [ 4822]     0  4822     1654        0       6       3       86             0 nvmemwarning.sh
[68531.946959] [ 4877]     0  4877     1318        0       6       4       33             0 agetty
[68531.946963] [ 4900]     0  4900     1318        0       6       3       35             0 agetty
[68531.946966] [ 4916]   105  4916     2367        0       8       4      125             0 kerneloops
[68531.946970] [ 4923]     0  4923     1374        0       8       3       36             0 agetty
[68531.946974] [ 4924]     0  4924     1654        0       8       4       76             0 nvgetty.sh
[68531.946977] [ 4932]   105  4932     2367        0       9       4      117             0 kerneloops
[68531.946981] [ 4974]     0  4974     2603        0       8       3      199         -1000 sshd
[68531.946985] [ 5042]     0  5042     1318        0       6       3       35             0 getty
[68531.946990] [ 5168]     0  5168      787        0       5       3       38             0 nvphsd
[68531.946994] [ 5218]     0  5218    19489        0       8       3      104             0 nvphsd
[68531.946998] [ 5438]     0  5438     2094        0       9       3      318             0 dhclient
[68531.947002] [ 5868]  1000  5868     3154        2      11       3      336             0 systemd
[68531.947006] [ 5896]  1000  5896    24439        0      15       4      632             0 (sd-pam)
[68531.947010] [ 7596]     0  7596    68456        0      24       4      483             0 packagekitd
[68531.947013] [ 1100]     0  1100    24742        0      33       3     1215             0 nvargus-daemon
[68531.947017] [ 8106]  1000  8106     1810        0       6       4      194             0 tmux: server
[68531.947021] [ 8109]  1000  8109     1990        2       7       3      420             0 bash
[68531.947026] [22561]     0 22561     3106        2       9       3      249             0 sshd
[68531.947029] [22608]  1000 22608     3106        0       9       3      259             0 sshd
[68531.947033] [22609]  1000 22609     2015        2       8       3      455             0 bash
[68531.947040] [24375]  1000 24375   944854   755756    1611       8    34611             0 gst-launch-1.0
[68531.947043] [24386]     0 24386     3033        0       9       4      240             0 sshd
[68531.947047] [24388]     0 24388      478        0       5       3       22             0 sh
[68531.947051] [24389]     0 24389      453        0       5       4       29             0 run-parts
[68531.947055] [24403]     0 24403      478        0       5       3       21             0 91-release-upgr
[68531.947059] [24404]     0 24404      478        0       4       3       22             0 91-release-upgr
[68531.947062] [24405]     0 24405     2876      113       9       3      942             0 lsb_release
[68531.947066] [24406]     0 24406      458        0       5       3       19             0 cut
[68531.947070] [24407]     0 24407     1074        0       4       3       31             0 sshd
[68531.947074] [24408]     0 24408     1654        0       6       3       85             0 nvmemwarning.sh
[68531.947077] [24409]     0 24409     1654        4       6       3       74             0 nvmemwarning.sh
[68531.947080] [24410]     0 24410     1654        0       6       3       78             0 nvmemwarning.sh
[68531.947084] Out of memory: Kill process 24375 (gst-launch-1.0) score 521 or sacrifice child
[68531.955551] Killed process 24375 (gst-launch-1.0) total-vm:3779416kB, anon-rss:0kB, file-rss:3023024kB, shmem-rss:0kB
[68531.998268] oom_reaper: reaped process 24375 (gst-launch-1.0), now anon-rss:0kB, file-rss:3024688kB, shmem-rss:0kB

It would appear to be running out of memory, perhaps due to the file size or video resolution. You can run tegrastats in the background to keep an eye on the memory usage, and mount a swap file if needed.

You could try inserting queue element(s) into the pipeline and limiting the max-size-buffers, and also as a test reducing the video resolution.

Gotcha. Swap is on, I had 6 but bumped it up to 8G. When running this I watch tegrastats and htop, but gstreamer never swaps to disk and maxes out the 4G of RAM.

Any way around this? Do you have an example of limiting the max-size-buffers? I only see it on the videotestsrc in the doc

Hi,
We have patch for jpeg decoding. Please check https://elinux.org/L4T_Jetson/r32.2.1_patch

Thanks, that looks promising. How would I go about applying that?

Hi @DaneLLL,

Did you mean :

  1. Installing a another version of jpeglib as in this link?
  2. Or may this be tried as well ?
  3. Something else ?

Thanks for clarifying this.

Hi,

With the lib, we are able to run

$ gst-launch-1.0 filesrc location= rat_mjpeg_1080p_24fps_Q15.mkv ! <b>matroskademux</b> ! jpegparse ! nvjpegdec ! 'video/x-raw' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420' ! omxh264enc ! matroskamux ! filesink location=video-compressed.mkv -e

Applying this patch should bring better performance. It requires extra effort of re-building libgstnvjpeg.so but is worth a try.

Thanks @DaneLLL, is there a guide somewhere on how to apply this patch / rebuild libgstnvjpeg.so? Or are there plans to make that mainstream?

Hi,
Please replace

/usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so

Also I have corrected #10. It has to have matroskademux in the pipeline.