ROS / Realsense D435 on Jetson TX2: Only 15fps FHD availlable

Hi devs,

i try to capture frames with an Intel Realsense D435 with my Jetson TX2 developer kit JetPack-L4T-3.2 (28.2).

Modifications:

  • Installed ROS (with JetsonHacks instructions)
  • Patched Kernel (with JetsonHacks instructions)
  • Installed Librealsense (with JetsonHacks instructions)
  • Installed realsense2_camera for ROS (with JetsonHacks instructions)
  • Installed latest firmware to Realsense D435

The camera is working fine. Depth stream seems to be good, IR1 and IR2 streams seems to be good and control comand can be applied via ROS-node or realsense-viewer.

My problem:
My color stream in HD or FHD is only capturing with 15 fps while i set up 30 fps. The specifications say, that HD or FHD 30 fps is available.I checked the fps with rqt. What can i do?

This is the output of a new launched realsense2_camera nodelet:

nvidia@tegra-ubuntu:~/ros_catkin_ws$ roslaunch launcher realsense_d435.launch 
... logging to /home/nvidia/.ros/log/681c40d2-747d-11e8-a4c9-00044b8d1193/roslaunch-tegra-ubuntu-9348.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

WARNING: ignoring defunct <master /> tag
started roslaunch server http://localhost:38379/

SUMMARY
========

PARAMETERS
 * /d435/realsense2_camera/accel_fps: 1000
 * /d435/realsense2_camera/accel_optical_frame_id: camera_accel_opti...
 * /d435/realsense2_camera/align_depth: False
 * /d435/realsense2_camera/camera_aligned_depth_to_color_frame_id: camera_aligned_de...
 * /d435/realsense2_camera/camera_aligned_depth_to_fisheye_frame_id: camera_aligned_de...
 * /d435/realsense2_camera/camera_aligned_depth_to_infra1_frame_id: camera_aligned_de...
 * /d435/realsense2_camera/camera_aligned_depth_to_infra2_frame_id: camera_aligned_de...
 * /d435/realsense2_camera/color_fps: 30
 * /d435/realsense2_camera/color_height: 1080
 * /d435/realsense2_camera/color_optical_frame_id: camera_color_opti...
 * /d435/realsense2_camera/color_width: 1920
 * /d435/realsense2_camera/depth_fps: 6
 * /d435/realsense2_camera/depth_height: 480
 * /d435/realsense2_camera/depth_optical_frame_id: camera_depth_opti...
 * /d435/realsense2_camera/depth_width: 640
 * /d435/realsense2_camera/enable_color: True
 * /d435/realsense2_camera/enable_depth: True
 * /d435/realsense2_camera/enable_fisheye: False
 * /d435/realsense2_camera/enable_imu: True
 * /d435/realsense2_camera/enable_infra1: False
 * /d435/realsense2_camera/enable_infra2: False
 * /d435/realsense2_camera/enable_pointcloud: False
 * /d435/realsense2_camera/enable_sync: True
 * /d435/realsense2_camera/fisheye_fps: 30
 * /d435/realsense2_camera/fisheye_height: 1080
 * /d435/realsense2_camera/fisheye_optical_frame_id: camera_fisheye_op...
 * /d435/realsense2_camera/fisheye_width: 1920
 * /d435/realsense2_camera/gyro_fps: 1000
 * /d435/realsense2_camera/gyro_optical_frame_id: camera_gyro_optic...
 * /d435/realsense2_camera/infra1_fps: 6
 * /d435/realsense2_camera/infra1_height: 480
 * /d435/realsense2_camera/infra1_optical_frame_id: camera_infra1_opt...
 * /d435/realsense2_camera/infra1_width: 640
 * /d435/realsense2_camera/infra2_fps: 6
 * /d435/realsense2_camera/infra2_height: 480
 * /d435/realsense2_camera/infra2_optical_frame_id: camera_infra2_opt...
 * /d435/realsense2_camera/infra2_width: 640
 * /d435/realsense2_camera/json_file_path: 
 * /d435/realsense2_camera/serial_no: 
 * /rosdistro: kinetic
 * /rosversion: 1.12.13
 
NODES
  /d435/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [9358]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 681c40d2-747d-11e8-a4c9-00044b8d1193
process[rosout-1]: started with pid [9371]
started core service [//rosout]
process[d435/realsense2_camera_manager-2]: started with pid [9388]
process[d435/realsense2_camera-3]: started with pid [9389]
[ INFO] [1529494261.587218125]: Initializing nodelet with 6 worker threads.
[ INFO] [1529494261.709804001]: RealSense ROS v2.0.3
[ INFO] [1529494261.709915744]: Running with LibRealSense v2.10.4
[ INFO] [1529494261.898911216]: getParameters...
[ INFO] [1529494262.331291437]: setupDevice...
[ INFO] [1529494262.331432365]: JSON file is not provided
[ INFO] [1529494262.331518733]: ROS Node Namespace: d435
[ INFO] [1529494262.331627052]: Device Name: Intel RealSense D435
[ INFO] [1529494262.331671404]: Device Serial No: 817512071398
[ INFO] [1529494262.331714956]: Device FW version: 05.09.11.00
[ INFO] [1529494262.331769836]: Device Product ID: 0x0B07
[ INFO] [1529494262.331804780]: Enable PointCloud: Off
[ INFO] [1529494262.331835244]: Align Depth: Off
[ INFO] [1529494262.331869932]: Sync Mode: On
[ INFO] [1529494262.332827881]: Device Sensors: 
[ INFO] [1529494262.333756614]: Stereo Module was found.
[ INFO] [1529494262.334108965]: RGB Camera was found.
[ INFO] [1529494262.335530561]: (Gyro, 0) sensor isnt supported by current device! -- Skipping...
[ INFO] [1529494262.335657152]: (Accel, 0) sensor isnt supported by current device! -- Skipping...
[ INFO] [1529494262.335854336]: setupPublishers...
[ INFO] [1529494262.354970375]: Expected frequency for depth = 6.00000
[ INFO] [1529494262.386906697]: Expected frequency for color = 30.00000
[ INFO] [1529494262.413189467]: setupStreams...
 20/06 11:31:03,402 WARNING [548398092288] (sensor.cpp:315) Unregistered Media formats : [ UYVY ]; Supported: [ ]
[ INFO] [1529494263.409226226]: depth stream is enabled - width: 640, height: 480, fps: 6
 20/06 11:31:03,609 WARNING [548398092288] (backend-v4l2.cpp:1158) Pixel format 36315752-1a66-a242-9065-d01814a likely requires patch for fourcc code RW16!
 20/06 11:31:03,612 WARNING [548398092288] (sensor.cpp:315) Unregistered Media formats : [ RW16 ]; Supported: [ ]
[ INFO] [1529494263.634823094]: color stream is enabled - width: 1920, height: 1080, fps: 30
[ INFO] [1529494264.677144936]: publishStaticTransforms...
 20/06 11:31:04,677 WARNING [548023628176] (ds5-timestamp.cpp:64) UVC metadata payloads not available. Please refer to installation chapter for details.
[ INFO] [1529494264.679750848]: RealSense Node Is Up!
[ WARN] [1529494264.688789477]: Frame metadata isnt available! (frame_timestamp_domain = RS2_TIMESTAMP_DOMAIN_SYSTEM_TIME)
 20/06 11:31:05,124 WARNING [547851469200] (ds5-timestamp.cpp:64) UVC metadata payloads not available. Please refer to installation chapter for details.
 20/06 11:31:05,506 ERROR [548398092288] (types.h:179) xioctl(VIDIOC_S_CTRL) failed Last Error: Numerical result out of range
[ WARN] [1529494265.508154573]: Reconfigure callback failed with exception xioctl(VIDIOC_S_CTRL) failed Last Error: Numerical result out of range:

Does anybody has an idea what to do? I stick in the mud with this fault…
Thanks to everybody!

I think the lack of direct hardware support is hindering D435 capture or software display just isn’t quick enough. So in short I am not sure but I just downloaded all this stuff for use with a D435 last night and
after recompiling the kernel I can no longer use nvpmodel.
Going to rebuild from scratch again.

Bob

What does “I can no longer use nvpmodel” mean?

I dont have the feeling, that the Jetson isnt quick enough! I can read out with OpenCv AND write down to file via GStreamer (with H264 encoding) and the framerates behaviour is stable: 30 fps depth stream and 15 fps color stream. If i stop the writing process, the color FPS stays at 15 fps. I Think the camera is publishing with 15 fps, but i dont know my fault…

This is what ROS-node says:

[ INFO] [1529494263.634823094]: color stream is enabled - width: 1920, height: 1080, fps: 30

Why is this not happening?

Does somebody know about this messages?

[ WARN] [1529494264.688789477]: Frame metadata isnt available! (frame_timestamp_domain = RS2_TIMESTAMP_DOMAIN_SYSTEM_TIME)
 20/06 11:31:05,124 WARNING [547851469200] (ds5-timestamp.cpp:64) UVC metadata payloads not available. Please refer to installation chapter for details.
 20/06 11:31:05,506 ERROR [548398092288] (types.h:179) xioctl(VIDIOC_S_CTRL) failed Last Error: Numerical result out of range
[ WARN] [1529494265.508154573]: Reconfigure callback failed with exception xioctl(VIDIOC_S_CTRL) failed Last Error: Numerical result out of range:

I have used the realsense-viewer and watch the camera easily deliver 30fps.
As soon as I added the depth elements it slowed down.

In general I am not sure how efficient the driver or software is when it comes
to display or even transfer of data. Are they in separate threads?
Do they take advantage of matrix multiplication in CUDA vs C.
And why is it 1920 x 1080 when the camera is 720p?

nvpmodel -m 0 lets you run at full speed.

There is no other process… I CAN start the “Writing” process (im not doing) and I CAN start a display (im not doing) but that dont affect the fps. To avoid disaccords: Im JUST starting the realsense2_camera node in ROS and the fps stays at 15 fps. IF i start more processes, the fps STAYS at 15 fps all the time. So i think the camera is configured to 15 fps, but i dont know why.

What do you mean? Look at the specs: It is able to stream FHD (1920 x 1080). By the way: I tried FHD and HD (720p) -> same behaviour (15 fps).

Im doing that!

Interesting update:

I figured out, the framerate depends on the exposure time settings. I used “rqt_reconfigure” to reconfigure the ros node. First of all i disabled auto-exposure. After that i set exposure value to 400. This speed up the framerate up tp ~28 fps.

The interesting thing:
The exposure range is 41-10000. When setting value to 400, framerate is nearly 30 (FHD and HD). When increasing or decreasing the value a bit by ~10, the framerate is falling down.

I cannot explain this behaviour…