Intel RealSense D435 USB3 Camera on TX2 - V4L2 Drops Frames at High Resolutions

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi all,

Using either the built-in Jetson TX2 USB port or a PCIe uPD720202 based card, I can’t acquire at resolutions higher than 848x480 without having dropped frames with this camera. By dropped frames, I mean every second or third frame comes back smaller than expected, containing just a few lines of pixel data.

I’m testing using my own V4L2 application, but the same appears if I use yavta for example.

I should point out that the camera uses bulk mode and not isochronous mode:

lsusb -d '8086:0b07' -v | grep 'Transfer Type'
          Transfer Type            Interrupt
          Transfer Type            Bulk
          Transfer Type            Bulk
FIXME: alloc bigger buffer for device capability descriptors
          Transfer Type            Bulk

lsusb -t shows 5000M speed

/:  Bus 10.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Video, Driver=uvcvideo, 5000M
    |__ Port 1: Dev 2, If 1, Class=Video, Driver=uvcvideo, 5000M
    |__ Port 1: Dev 2, If 2, Class=Video, Driver=uvcvideo, 5000M
    |__ Port 1: Dev 2, If 3, Class=Video, Driver=uvcvideo, 5000M
    |__ Port 1: Dev 2, If 4, Class=Video, Driver=uvcvideo, 5000M

Here is my yavta output:

./yavta --capture=20 --size 1920x1080 -t 1/30  /dev/video4
Device /dev/video4 opened: Intel(R) RealSense(TM) 435 (usb-0000:06:00.0-1).
Video format set: width: 1920 height: 1080 buffer size: 4147200
Video format: YUYV (56595559) 1920x1080
Current frame rate: 1/30
Setting frame rate to: 1/30
Frame rate set: 1/30
8 buffers requested.
length: 4147200 offset: 0
Buffer 0 mapped at address 0x7fa016f000.
length: 4147200 offset: 4149248
Buffer 1 mapped at address 0x7f9fd7a000.
length: 4147200 offset: 8298496
Buffer 2 mapped at address 0x7f9f985000.
length: 4147200 offset: 12447744
Buffer 3 mapped at address 0x7f9f590000.
length: 4147200 offset: 16596992
Buffer 4 mapped at address 0x7f9f19b000.
length: 4147200 offset: 20746240
Buffer 5 mapped at address 0x7f9eda6000.
length: 4147200 offset: 24895488
Buffer 6 mapped at address 0x7f9e9b1000.
length: 4147200 offset: 29044736
Buffer 7 mapped at address 0x7f9e5bc000.
0 (0) [-] 0 4147200 bytes 185437.733973 1542228849.119820
Warning: bytes used 65784 != image size 4147200
1 (1) [-] 1 65784 bytes 185437.739973 1542228849.120196
2 (2) [-] 2 4147200 bytes 185437.801172 1542228849.187045
Warning: bytes used 65784 != image size 4147200
3 (3) [-] 3 65784 bytes 185437.807220 1542228849.187409
4 (4) [-] 4 4147200 bytes 185437.868362 1542228849.254179
Warning: bytes used 65784 != image size 4147200
5 (5) [-] 5 65784 bytes 185437.874355 1542228849.254540
6 (6) [-] 6 4147200 bytes 185437.935557 1542228849.321373
Warning: bytes used 65784 != image size 4147200

Lower resolutions work fine. Anyone have any hints?

2

Can you please try ‘sudo ./jetson_clocks.sh’?

Also please try to run the sample tegra_multimedia_api\samples\12_camera_v4l2_cuda and check the result.

3

Thanks for the info, I will build and try this. I did run jetson_clocks.sh I should add, but it didn’t help.

Best,

  • Alex
4

I have tried the 12_camera_v4l2_cuda example by putting the stock JetPack L4T 3.2.1 image on my Jetson TX2, then applying a kernel patched to support the camera’s pixel formats in V4L2, and I ran the sample application.

When using resolutions of 848x480 or lower, the video display looked fine. But 1280x720 and 1920x1080 showed a flickering image - every second frame appeared ok, but the bad ones seemed to contain just a few rows of pixel information.

I tried to take a picture of it, but it doesn’t show up well on a cell phone camera.

This works fine on an Intel PC with Ubuntu 16.04

5

Hi precosky,
Can you try to connect D435 to USB3 HUB with external power supply and then connect the HUB to TX2 USB3 port?
Would like to check if is is an issue in power supply.

Besides, do you see any error/warning in dmesg?

6

Hi Dane,

I’ll get a powered hub (probably in the new week) and try this and I’ll update this once I’ve tried it.

In dmesg, I see just the following after issuing a command to yavta to acquire some images:

./yavta --capture=80 /dev/video2 -t 1/30 --size=1920x1080

[Nov23 20:21] xhci-tegra 3530000.xhci: exiting ELPG
[  +0.027320] xhci-tegra 3530000.xhci: Firmware timestamp: 2017-12-07 10:50:08 UTC, Version: 55.09 release
[  +0.013391] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 6
[  +0.007157] xhci-tegra 3530000.xhci: exiting ELPG done
[  +5.183299] xhci-tegra 3530000.xhci: entering ELPG
[  +0.009272] xhci-tegra 3530000.xhci: entering ELPG done
7

The prints look normal. Please try powered hub.

8

Hi precosky,

Have you tried the powered hub? Any good news?

Thanks

9

Hi kayccc,

I have now acquired a powered hub, and plugged it in with its AC adaptor and connected the cable to the on-board USB port of the TX2 board. One realsense camrea is connected to the hub and nothing else.

Unfortunately the problem remains:

➜ dexr@precosky_tx2_a yavta git:(master) ✗ ./yavta /dev/video2 --capture=10 --size 1920x1080 -f YUYV -t 1/15
Device /dev/video2 opened: Intel(R) RealSense™ 435 (usb-3530000.xhci-1.3.1).
Video format set: width: 1920 height: 1080 buffer size: 4147200
Video format: YUYV (56595559) 1920x1080
Current frame rate: 1/30
Setting frame rate to: 1/15
Frame rate set: 1/15
8 buffers requested.
length: 4147200 offset: 0
Buffer 0 mapped at address 0x7f90622000.
length: 4147200 offset: 4149248
Buffer 1 mapped at address 0x7f9022d000.
length: 4147200 offset: 8298496
Buffer 2 mapped at address 0x7f8fe38000.
length: 4147200 offset: 12447744
Buffer 3 mapped at address 0x7f8fa43000.
length: 4147200 offset: 16596992
Buffer 4 mapped at address 0x7f8f64e000.
length: 4147200 offset: 20746240
Buffer 5 mapped at address 0x7f8f259000.
length: 4147200 offset: 24895488
Buffer 6 mapped at address 0x7f8ee64000.
length: 4147200 offset: 29044736
Buffer 7 mapped at address 0x7f8ea6f000.
0 (0) [-] 0 4147200 bytes 3349.283954 1544117756.370382
Warning: bytes used 65784 != image size 4147200
1 (1) [-] 1 65784 bytes 3349.296568 1544117756.371402
2 (2) [-] 2 4147200 bytes 3349.418348 1544117756.504680
Warning: bytes used 65784 != image size 4147200
3 (3) [-] 3 65784 bytes 3349.430931 1544117756.505247
4 (4) [-] 4 4147200 bytes 3349.552707 1544117756.633103
Warning: bytes used 65784 != image size 4147200
5 (5) [-] 5 65784 bytes 3349.559297 1544117756.633683
6 (6) [-] 6 4147200 bytes 3349.687100 1544117756.766974
Warning: bytes used 65784 != image size 4147200
7 (7) [-] 7 65784 bytes 3349.693162 1544117756.767392
8 (0) [-] 8 4147200 bytes 3349.821518 1544117756.907716
Warning: bytes used 65784 != image size 4147200
9 (1) [-] 9 65784 bytes 3349.833992 1544117756.908719
Captured 9 frames in 0.539176 seconds (16.692138 fps, 39068727.094678 B/s).

I’m getting support now through NVOnline but I’ll keep this updated if I learn anything.

10

Can you check the operating temperature of the device?

11

The cameras are just sitting on a desk. I’ve added temperature monitoring as per Read device temperature · Issue #866 · IntelRealSense/librealsense · GitHub
and reading a steady 28 deg C

12

bro, i know NOTHING about video capture. Im a total newb starting in robotics and i have a D435 and was trying to figure out what the heck V4L2-CTL really does.

when i ran your yavta command on my laptop i got the errors you are also getting.
yavta --capture=10 --size 1920x1080 -t 1/30 /dev/video0 -Fimage.jpg -fYUYV

Warning: bytes used 0 != image size 1843200 for plane 0
0 (0) [E] any 0 0 B 0.000000 72139.139180 -0.000 fps ts mono/SoE
Warning: bytes used 0 != image size 1843200 for plane 0
1 (1) [E] any 0 0 B 0.000000 72139.139556 0.000 fps ts mono/SoE
Warning: bytes used 0 != image size 1843200 for plane 0
2 (2) [E] any 0 0 B 0.000000 72139.139635 0.000 fps ts mono/SoE

but then i ran it against /dev/video4
and i got all good output

image
image1212×668 146 KB

it seems you maybe running your command against the wrong device since the D435 manifests itself as multiple devices in linux

v4l2-ctl --list-devices
Intel(R) RealSense™ Depth Ca (usb-0000:00:14.0-1):
/dev/video0
/dev/video1
/dev/video2
/dev/video3
/dev/video4
/dev/video5
/dev/media0
/dev/media1

Integrated_Webcam_HD: Integrate (usb-0000:00:1d.0-1.6):
/dev/video6
/dev/video7

and the numbered order of these devices is not gauranteed when you reboot the machine

when i run
v4l2-ctl --list-formats -d 0

i get the depth format
[0]: 'Z16 ’ (16-bit Depth)

when i run

v4l2-ctl --list-formats -d 4

i get the RGBish format
[0]: ‘YUYV’ (YUYV 4:2:2)

image
image477×594 56.6 KB

so as you can see your command is asking for apples and you are pointing it to oranges?
e.g. you are asking for YUYV from a Z16 source
yavta --capture=10 --size 1920x1080 -t 1/30 /dev/video4 -Fimage.jpg -fYUYV

you can also run
yavta --enum-formats /dev/video0

on all the video devices.

image
image1182×951 157 KB

1 Like
14

Ooooo OOoooh.
Seems like the issue is deeper than that. When i run the same command again

yavta --capture=10 --size 1920x1080 -t 1/30 /dev/video4 -Fimage -fYUYV

i get your warning messages.

but if i unplug the USB and plug it back in and run it again its fine again until the second time i run the command.

image
image1189×801 175 KB

I got it to work by doing the instructions at

Linux Ubuntu Installation

now you get

v4l2-ctl --list-formats --device=/dev/video0
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘YUYV’
Name : YUYV 4:2:2

Index       : 1
Type        : Video Capture
Pixel Format: 'MJPG' (compressed)
Name        : Motion-JPEG

instead of before when i was getting

v4l2-ctl --list-formats --device=/dev/video0
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

[0]: 'Z16 ' (16-bit Depth)
[1]: '' (00000050-0000-0010-8000-00aa003)

./yavta --enum-formats /dev/video0
Device /dev/video0 opened: Integrated_Webcam_HD: Integrate (usb-0000:00:1d.0-1.6).

  • Available formats:
    Format 0: YUYV (56595559)
    Type: Video capture (1)
    Name: YUYV 4:2:2
    Frame size: 640x480 (1/30, 1/30)
    Frame size: 160x120 (1/30)
    Frame size: 320x180 (1/30)
    Frame size: 320x240 (1/30)
    Frame size: 424x240 (1/30)
    Frame size: 640x360 (1/30)
    Frame size: 640x480 (1/30, 1/30)

    Format 1: MJPG (47504a4d)
    Type: Video capture (1)
    Name: Motion-JPEG
    Frame size: 848x480 (1/30)
    Frame size: 960x540 (1/30)
    Frame size: 1280x720 (1/30)

Video format: YUYV (56595559) 640x480