Image_transport republish with hardware decoder on Jetson Orin Nano?

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

Hello!
TL;DR How to install ffmpeg with hw decoder (preferably h264) on Jetson Orin Nano Isaac ROS 3.2 container so that ffmpeg-image-transport can utilize it? (isaac_ros_h264_decoder not an option due to dev env limitation)

I want to decompress images on ffmpeg topic encoded with another image_transport republish node on another x64 machine, on Jetson Orin Nano.

isaac_ros_h264_decoder is less a choice for me because…it takes time to migrate to Isaac ROS on the encoding x64 machine. And according to the doc, isaac_ros_h264_decoder doesn’t seem to be able to decode topics published by ffmpeg-image-transport.

Here’s what I tried:

admin@ubuntu:/workspaces/isaac_ros-dev/src/dualvision/ros-workspace$ ros2 run image_transport republish ffmpeg raw --ros-args -r in/ffmpeg:=/camera/left/image_raw/ffmpeg -r out:=/camera/left/image_raw/decoded -r __node:=jetson_republish_left
[WARN] [1782803691.897433656] [FFMPEGSubscriber]: no decoders configured for encoding h264;yuv420p;bgr8;rgb8 defaulting to: h264
[INFO] [1782803691.897614973] [FFMPEGSubscriber]: trying decoders in order: h264
[INFO] [1782803691.897717984] [Decoder]: output image encoding: rgb8
[INFO] [1782803691.898044266] [Decoder]: decoder h264 has no hardware acceleration
[INFO] [1782803691.898085835] [Decoder]: using decoder       h264 with options: 
[INFO] [1782803691.898606938] [Decoder]: decoding with h264
[INFO] [1782803691.913587667] [FFMPEGSubscriber]: successfully opened decoder h264

Though it does decode the image, but seems it’s not using hardware acceleration?
And didn’t find a parameter to set decoder in use in ros2 param list

admin@ubuntu:/workspaces/isaac_ros-dev$ ros2 param list /jetson_republish_left
  in.ffmpeg.decoder_av_options
  in.ffmpeg.decoder_measure_performance
  in.ffmpeg.decoders.h264
  in.ffmpeg.decoders.h264_yuv420p
  in.ffmpeg.decoders.h264_yuv420p_bgr8
  in.ffmpeg.decoders.h264_yuv420p_bgr8_rgb8
  qos_overrides./parameter_events.publisher.depth
  qos_overrides./parameter_events.publisher.durability
  qos_overrides./parameter_events.publisher.history
  qos_overrides./parameter_events.publisher.reliability
  use_sim_time

(I remember on the encoding x64 side there’s a out.ffmpeg.encoder parameter)
And here’s ffmpeg -decoders

admin@ubuntu:/workspaces/isaac_ros-dev$ ffmpeg -decoders 2>/dev/null | grep 264
 VFS..D h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
 V..... h264_v4l2m2m         V4L2 mem2mem H.264 decoder wrapper (codec h264)

This official document says ffmpeg on jetson should support h264 hardware decoder?

Thanks!

Env:
Jetson Orin Nano Super Dev Kit 8G
JetPack 6.2.1
Isaac ROS 3.2 docker flow

Edit

I somehow missed the apt source list part to install nvidia provided ffmpeg.
However I failed to install the ffmpeg. Seems like it depends on some packages that are meant to be installed on host side instead of inside container?
After a fresh run_dev.sh -i 'ros2_humble', I add the third line in apt source list:

admin@ubuntu:/workspaces/isaac_ros-dev$ cat /etc/apt/sources.list.d/nvidia-l4t-apt-source.list 
deb https://repo.download.nvidia.com/jetson/common r36.4 main
deb https://repo.download.nvidia.com/jetson/t234 r36.4 main
deb https://repo.download.nvidia.com/jetson/ffmpeg r36.4 main

Then sudo apt update, then sudo apt install ffmpeg:

admin@ubuntu:/workspaces/isaac_ros-dev$ sudo apt install ffmpeg
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libavcodec-dev libavcodec58 libavformat-dev libavformat58 libavutil-dev libavutil56 libsrt1.4-gnutls libssh-gcrypt-4 libswresample-dev libswresample3 libswscale-dev libswscale5 libva-x11-2 libvdpau1
  mesa-vdpau-drivers vdpau-driver-all
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libass9 libavc1394-0 libbs2b0 libcaca0 libcdio-cdda2 libcdio-paranoia2 libcdio19 libflite1 libgstreamer-plugins-bad1.0-0 libiec61883-0 libjack-jackd2-0 liblilv-0-0 libmysofa1 libopenal-data libopenal1
  libpocketsphinx3 librubberband2 libsamplerate0 libserd-0-0 libslang2 libsord-0-0 libsphinxbase3 libsratom-0-0 libsrt1.4-openssl libvidstab1.1 libzimg2 nvidia-l4t-core nvidia-l4t-cuda
  nvidia-l4t-multimedia nvidia-l4t-multimedia-utils nvidia-l4t-nvsci pocketsphinx-en-us
Suggested packages:
  jackd2 libportaudio2 serdi sordi
The following NEW packages will be installed:
  ffmpeg libass9 libavc1394-0 libbs2b0 libcaca0 libcdio-cdda2 libcdio-paranoia2 libcdio19 libflite1 libgstreamer-plugins-bad1.0-0 libiec61883-0 libjack-jackd2-0 liblilv-0-0 libmysofa1 libopenal-data
  libopenal1 libpocketsphinx3 librubberband2 libsamplerate0 libserd-0-0 libslang2 libsord-0-0 libsphinxbase3 libsratom-0-0 libsrt1.4-openssl libvidstab1.1 libzimg2 nvidia-l4t-core nvidia-l4t-cuda
  nvidia-l4t-multimedia nvidia-l4t-multimedia-utils nvidia-l4t-nvsci pocketsphinx-en-us
0 upgraded, 33 newly installed, 0 to remove and 718 not upgraded.
Need to get 88.4 MB of archives.
After this operation, 223 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe arm64 libgstreamer-plugins-bad1.0-0 arm64 1.20.3-0ubuntu1.1 [459 kB]
Get:2 https://repo.download.nvidia.cn/jetson/t234 r36.4/main arm64 nvidia-l4t-core arm64 36.4.7-20250918154033 [3,680 kB]
Get:3 https://repo.download.nvidia.cn/jetson/t234 r36.4/main arm64 nvidia-l4t-cuda arm64 36.4.7-20250918154033 [6,660 kB]
Get:4 https://repo.download.nvidia.cn/jetson/t234 r36.4/main arm64 nvidia-l4t-multimedia-utils arm64 36.4.7-20250918154033 [270 kB]
Get:5 https://repo.download.nvidia.cn/jetson/t234 r36.4/main arm64 nvidia-l4t-nvsci arm64 36.4.7-20250918154033 [416 kB]
Get:6 https://repo.download.nvidia.cn/jetson/t234 r36.4/main arm64 nvidia-l4t-multimedia arm64 36.4.7-20250918154033 [13.1 MB]
Get:8 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 libslang2 arm64 2.3.2-5build4 [442 kB]
Get:7 https://repo.download.nvidia.cn/jetson/ffmpeg r36.4/main arm64 ffmpeg arm64 7:4.4.2-nvidia [17.3 MB]
Get:9 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libass9 arm64 1:0.15.2-1 [87.8 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 libavc1394-0 arm64 0.5.4-5build2 [16.6 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libbs2b0 arm64 3.1.0+dfsg-2.2build1 [9,548 B]
Get:12 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 libcaca0 arm64 0.99.beta19-2.2ubuntu4.2 [221 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 libcdio19 arm64 2.1.0-3ubuntu0.2 [64.4 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 libcdio-cdda2 arm64 10.2+2.0.0-1build3 [16.5 kB]
Get:15 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 libcdio-paranoia2 arm64 10.2+2.0.0-1build3 [15.1 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libflite1 arm64 2.2-3 [13.7 MB]
Get:17 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 libiec61883-0 arm64 1.2.0-4build3 [24.9 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 libsamplerate0 arm64 0.2.2-1build1 [1,359 kB]
Get:19 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 libjack-jackd2-0 arm64 1.9.20~dfsg-1 [284 kB]
Get:20 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libserd-0-0 arm64 0.30.10-2 [37.7 kB]
Get:21 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libsord-0-0 arm64 0.16.8-2 [19.7 kB]
Get:22 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libsratom-0-0 arm64 0.6.8-1 [15.0 kB]
Get:23 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 liblilv-0-0 arm64 0.24.12-2 [40.4 kB]
Get:24 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libmysofa1 arm64 1.2.1~dfsg0-1 [1,156 kB]
Get:25 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libopenal-data all 1:1.19.1-2build3 [164 kB]
Get:26 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libopenal1 arm64 1:1.19.1-2build3 [525 kB]
Get:27 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libsphinxbase3 arm64 0.8+5prealpha+1-13build1 [123 kB]
Get:28 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libpocketsphinx3 arm64 0.8.0+real5prealpha+1-14ubuntu1 [129 kB]
Get:29 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 librubberband2 arm64 2.0.0-2 [83.3 kB]
Get:30 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libsrt1.4-openssl arm64 1.4.4-4 [289 kB]
Get:31 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libvidstab1.1 arm64 1.1.0-2 [34.0 kB]
Get:32 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 libzimg2 arm64 3.0.3+ds1-1 [117 kB]
Get:33 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 pocketsphinx-en-us all 0.8.0+real5prealpha+1-14ubuntu1 [27.6 MB]
Fetched 88.4 MB in 9s (10.0 MB/s)                                                                                                                                                                          
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78, <> line 33.)
debconf: falling back to frontend: Readline
Extracting templates from packages: 100%
Selecting previously unselected package nvidia-l4t-core.
(Reading database ... 166374 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-core_36.4.7-20250918154033_arm64.deb ...
lsblk: overlay: not a block device
partx: /dev/: failed to initialize blkid prober
INFO: rootfs A/B is not enabled.
Unpacking nvidia-l4t-core (36.4.7-20250918154033) ...
dpkg: error processing archive /var/cache/apt/archives/nvidia-l4t-core_36.4.7-20250918154033_arm64.deb (--unpack):
 unable to make backup link of './usr/lib/aarch64-linux-gnu/nvidia/ld.so.conf' before installing new version: Invalid cross-device link
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/nvidia-l4t-core_36.4.7-20250918154033_arm64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Seems like nvidia-l4t-core is meant to be installed on host side?

Edit

Tried deb https://repo.download.nvidia.com/jetson/ffmpeg main main instead of deb https://repo.download.nvidia.com/jetson/ffmpeg r36.4 main in a fresh run_dev.sh. Installed 7:4.2.2-nvidia.1 instead of 7:4.2.2-nvidia.
2 problems surfaces:

  1. libavcodec.so installed under /usr/lib, but there is another existing installation under /usr/lib/aarch64-linux-gnu which is the default made by /etc/ld.so.conf.d/000_cuda.conf.
  2. /usr/lib/libavcodec.so.58 cannot find libnvbuf_utils.so.1.0.0.

This really makes me think, is the provided ffmpeg package not meant to be used inside Isaac ROS container? But if so, how does one use hardware decoder with ros-humble-ffmpeg-image-transport?

2

Hello @fishboneek!

Based on the title and content of your topic, it looks like it may receive better visibility and feedback in a different category. We took the liberty of moving it for you.

If this was an incorrect assessment, please send me a direct message.

Disclaimer: this moderation suggestion and message were generated with AI assistance.

3

Questionable act. It should be more related to distribution and package management instead of the hardware platform.

4

To quote ChatGPT: “You are absolutely right @fishboneek! Let me fix this for you.”

Joke aside, the Agent automation can, as any AI, make wrong assumptions. I will try to address that in the Agent directions.

Thanks for letting me know!

BTW, this does not guarantee that Isaac ROS moderators might not see this as a Jetson specific issue and move the post again.

5

Hello @fishboneek,

Thanks for posting in the Isaac ROS forum!

For using Isaac ROS container, I do not see a clean way to install NVIDIA Jetson ffmpeg and use hardware decode through ffmpeg_image_transport. Adding the Jetson ffmpeg apt repo causes ffmpeg to pull nvidia-l4t-core / nvidia-l4t-multimedia packages, and nvidia-l4t-core fails during install in the container overlay filesystem. Even the generic ffmpeg decoder h264_v4l2m2m is not a working hardware decode fix in this container.

If you keep using ffmpeg_image_transport, the supported practical path in the stock Isaac ROS container is software decode:

ros2 run image_transport republish ffmpeg raw \
--ros-args \
-r in/ffmpeg:=/camera/left/image_raw/ffmpeg \
-r out:=/camera/left/image_raw/decoded \
-p "in.ffmpeg.decoders.h264:=h264"

This should still decode topics published by ffmpeg_image_transport and it just uses CPU instead of Jetson NVDEC.

As for Jetson hardware decode, NVIDIA ffmpeg on the Jetson host exposes h264_nvv4l2dec and can decode through the NVDEC path. To use that with ffmpeg_image_transport, you would need to run the decode bridge on the Jetson host or in a custom L4T multimedia-based ROS container where NVIDIA ffmpeg and ffmpeg_image_transport are installed/built together.

For the Isaac ROS-supported hardware decode path, isaac_ros_h264_decoder is still the preferred option where the message/pipeline format allows it.

6

Damn it. It’s what I worried all along.

Then this would mean hardware accelerated decoding with ffmpeg-image-transport is not officially supported, users must find another way.

isaac_ros_h264_encoder/decoder is a choice but it means I can only use the pair. If the other network end does not / cannot use this encoder/decoder, then there’s problem.

Alright. I’ll look another way around.