Does anyone have any experience getting the Intel Realsense R200 camera (http://click.intel.com/intel-realsense-developer-kit-r200.html) to work with the TX1?
There is a cross-platform API for the camera (https://github.com/IntelRealSense/librealsense), and I’d like to find a way to see if it’s compatible with TX1.
Does anyone have any thoughts about or want to work on this together?
Thanks!
Here are the installation instructions for Ubuntu 14.04: librealsense/installation.md at master · IntelRealSense/librealsense · GitHub
I’m having trouble installing the Video4Linux backend because of the uvcvideo.ko patch that is required and I can’t patch for this kernel for some reason. I keep getting the error “Unable to find a source package for linux-image-3.10.67-g458d45c” when I run the patching script (/scripts/patch-uvcvideo-ubuntu-mainline.sh).
Any thoughts are welcome!
No thoughts as of yet, but I am having the same issue.
You can run the Intel Realsense on a Jetson using a UVC backend. Here’s a script:
You’ll need to switch over to the installLibrealsense directory and:
$ ./installLibrealsense.sh
librealsense will be placed in the home folder.
I’m still working through some of the issues and the repositories will change over time. I have modified the Qt Creator projects so that they (mostly) work with the uvc backend.
Note that the camera needs to be replugged after a program is run, that’s one of the issues that I’m currently working on.
it is good to read, Kangalow, that you have done the necessary work to bypass that ARM limitation and have brought the RealSense camera to tx1. Thank you!
I can not take credit, Intel and another Jetson community member did the heavy lifting. I just wrote up a little install script before I start some projects.
Thanks Kangalow!
Although I’m running into two errors (“error: unrecognized command line option ‘-mssse3’” and “[…/bin/debug/obj/image.o] Error 1”) when I try to build all.pro. Have you run into these errors before?
Thanks!
Yes, the -msse3 error is for Intel processors and needs to be commented out. image.c has a lot of msse code in it, so it won’t compile if that flag is set. I thought that I fixed the all.pro correctly, but apparently not. Commenting the -msse3 flag will work in the short run.
The makefile to build the binaries in the root directory should have correctly.
Okay, thanks! Where is the -mssse3 flag? I found it in Makefile.realsense, but when I deleted it there, I get a new error: “tmmintrin.h: No such file or directory”, which is in image.cpp. I guess that’s what you were referring to? Do you know how I can get around this?
I’m confused. Are you using the file ‘Makefile’ in the root directory in the branch ‘jetson’? Are you compiling on the TX1 itself, or cross compiling?
Whoops, I did something really dumb. I had the original version of librealsense (intended for Linux x86) installed from a previous attempt, and was probing that directory instead of the one your scripts installed! It is building now woohoo, and you were right, the mssse3 flag is not present in this version of librealsense! Great work kangalow and others!
Do you know how I can access the IR only video stream? Thanks!!
Look in the examples folders. There are two IR sensors, and are typically accessed like:
rs_enable_stream(dev, RS_STREAM_INFRARED, 640, 480, RS_FORMAT_Y8, 60, &e);
check_error();
rs_enable_stream(dev, RS_STREAM_INFRARED2, 640, 480, RS_FORMAT_Y8, 60, NULL); /* Pass NULL to ignore errors */
check_error();
Hello, I have tried various methods of getting the V4L2 Backend functioning to no avail. Just thought I’d post my efforts, maybe helping others. I am however total linux novice, am surprised I even got so far TBH…
What I managed:
Results:
When I plug in a R200:
dmesg shows:
[ 155.355061] usb 3-2: new SuperSpeed USB device number 7 using tegra-xhci
[ 155.382711] usb 3-2: New USB device found, idVendor=8086, idProduct=0a80
[ 155.382838] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 155.382933] usb 3-2: Product: Intel RealSense 3D Camera R200
[ 155.383022] usb 3-2: Manufacturer: Intel Corp
[ 155.383103] usb 3-2: SerialNumber: SN_2481011935
[ 155.397762] uvcvideo: Unknown video format 49323159-0000-0010-8000-00aa00389b71
[ 155.397902] uvcvideo: Unknown video format 20493859-0000-0010-8000-00aa00389b71
[ 155.398065] uvcvideo: Found UVC 1.10 device Intel RealSense 3D Camera R200 (8086:0a80)
[ 155.412721] uvcvideo: Unknown video format 2036315a-0000-0010-8000-00aa00389b71
[ 155.412799] uvcvideo: Found UVC 1.10 device Intel RealSense 3D Camera R200 (8086:0a80)
[ 155.415389] uvcvideo: Unable to create debugfs 3-7 directory.
[ 155.416728] input: Intel RealSense 3D Camera R200 as /devices/platform/tegra-xhci/usb3/3-2/3-2:1.2/input/input14
[ 155.424863] uvcvideo: Unknown video format 30315752-0000-0010-8000-00aa00389b71
[ 155.424954] uvcvideo: Found UVC 1.10 device Intel RealSense 3D Camera R200 (8086:0a80)
[ 155.427627] uvcvideo: Unable to create debugfs 3-7 directory.
[ 155.429017] input: Intel RealSense 3D Camera R200 as /devices/platform/tegra-xhci/usb3/3-2/3-2:1.4/input/input15
[ 155.487884] uvcvideo: Unknown video format 49323159-0000-0010-8000-00aa00389b71
[ 155.487896] uvcvideo: Unknown video format 20493859-0000-0010-8000-00aa00389b71
[ 155.487936] uvcvideo: Found UVC 1.10 device Intel RealSense 3D Camera R200 (8086:0a80)
[ 155.490536] systemd-udevd[2126]: Failed to apply ACL on /dev/video0: No such file or directory
[ 155.616228] uvcvideo: Unknown video format 2036315a-0000-0010-8000-00aa00389b71
[ 155.616409] uvcvideo: Found UVC 1.10 device Intel RealSense 3D Camera R200 (8086:0a80)
[ 155.619937] uvcvideo: Unable to create debugfs 3-7 directory.
[ 155.622699] input: Intel RealSense 3D Camera R200 as /devices/platform/tegra-xhci/usb3/3-2/3-2:1.2/input/input16
[ 155.746497] uvcvideo: Unknown video format 30315752-0000-0010-8000-00aa00389b71
[ 155.746688] uvcvideo: Found UVC 1.10 device Intel RealSense 3D Camera R200 (8086:0a80)
[ 155.750889] uvcvideo: Unable to create debugfs 3-7 directory.
[ 155.754255] input: Intel RealSense 3D Camera R200 as /devices/platform/tegra-xhci/usb3/3-2/3-2:1.4/input/input17
lsusb shows:
Bus 003 Device 007: ID 8086:0a80 Intel Corp.
At boot dmesg says about uvcvideo:
[ 6.898575] usbcore: registered new interface driver uvcvideo
And the final straw, when I run cpp-headless, or cpp-capture:
./cpp-headless
There are 1 connected RealSense devices.
Using device 0, an Intel RealSense R200
Serial number: 2481011935
Firmware version: 1.0.72.06
...
[ 490.172470] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[ 490.180668] pgd = ffffffc00007d000
[ 490.184137] [00000000] *pgd=000000017fc06003, *pmd=000000017fc07003, *pte=00e0000050041407
[ 490.192625] Internal error: Oops: 96000045 [#1] PREEMPT SMP
[ 490.198258] Enter nvdumper_crash_setup_regs
[ 490.202724] nvdumper: all registers are saved.
[ 490.202747] nvdumper: all registers are saved.
[ 490.202771] nvdumper: all registers are saved.
[ 490.216272] nvdumper: all registers are saved.
[ 490.220776] Modules linked in: bnep rfcomm bcmdhd cfg80211 nvhost_vi bluedroid_pm
[ 490.228696] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.67-rex+ #1
[ 490.235210] task: ffffffc0010f1870 ti: ffffffc0010e4000 task.ti: ffffffc0010e4000
[ 490.242788] PC is at uvc_video_clock_decode+0x108/0x190
[ 490.248088] LR is at uvc_video_clock_decode+0xf0/0x190
And the system crashes… It would appear that maybe I didn’t correctly patch uvcvideo, but i checked modinfo and it has loaded the one i compiled from the modified files. Gi diff reports:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
index 0c2bc73..0ef7c9a 100644
--- a/arch/arm64/kernel/vdso32/Makefile
+++ b/arch/arm64/kernel/vdso32/Makefile
@@ -11,7 +11,7 @@ obj-vdso32 := $(addprefix $(obj)/, $(obj-vdso32))
GCOV_PROFILE := n
-ccflags-y := -shared -fPIC -fno-common -fno-builtin -march=armv7-a
+ccflags-y := -shared -fPIC -fomit-frame-pointer -fno-common -fno-builtin -march=armv7-a
ccflags-y += -nostdlib -Wl,-soname=linux-vdso32.so.1 \
$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
asflags-y := -D__VDSO32__ -s
diff --git a/drivers/media/usb/uvc/uvc_driver.c b/drivers/media/usb/uvc/uvc_driver.c
index 363cdbf..3440ba9 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -108,6 +108,56 @@ static struct uvc_format_desc uvc_fmts[] = {
.fcc = V4L2_PIX_FMT_Y16,
},
{
+ .name = "Greyscale 8 L/R (Y8I)",
+ .guid = UVC_GUID_FORMAT_Y8I,
+ .fcc = V4L2_PIX_FMT_Y8I,
+ },
+ {
+ .name = "Greyscale 12 L/R (Y12I)",
+ .guid = UVC_GUID_FORMAT_Y12I,
+ .fcc = V4L2_PIX_FMT_Y12I,
+ },
+ {
+ .name = "Depth data 16-bit (Z16)",
+ .guid = UVC_GUID_FORMAT_Z16,
+ .fcc = V4L2_PIX_FMT_Z16,
+ },
+ {
+ .name = "Raw data 10-bit (RW10)",
+ .guid = UVC_GUID_FORMAT_RW10,
+ .fcc = V4L2_PIX_FMT_RW10,
+ },
+ {
+ .name = "Depth 16-bit (INVZ)",
+ .guid = UVC_GUID_FORMAT_INVZ,
+ .fcc = V4L2_PIX_FMT_INVZ,
+ },
+ {
+ .name = "Depth:IR 16:8 24-bit (INZI)",
+ .guid = UVC_GUID_FORMAT_INZI,
+ .fcc = V4L2_PIX_FMT_INZI,
+ },
+ {
+ .name = "Depth 16-bit (INVR)",
+ .guid = UVC_GUID_FORMAT_INVR,
+ .fcc = V4L2_PIX_FMT_INVR,
+ },
+ {
+ .name = "Depth:IR 16:8 24-bit (INRI)",
+ .guid = UVC_GUID_FORMAT_INRI,
+ .fcc = V4L2_PIX_FMT_INRI,
+ },
+ {
+ .name = "Infrared 8-bit (INVI)",
+ .guid = UVC_GUID_FORMAT_INVI,
+ .fcc = V4L2_PIX_FMT_INVI,
+ },
+ {
+ .name = "FlickerIR 8-bit (RELI)",
+ .guid = UVC_GUID_FORMAT_RELI,
+ .fcc = V4L2_PIX_FMT_RELI,
+ },
+ {
.name = "RGB Bayer",
.guid = UVC_GUID_FORMAT_BY8,
.fcc = V4L2_PIX_FMT_SBGGR8,
diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h
index af505fd..4af23e7 100644
--- a/drivers/media/usb/uvc/uvcvideo.h
+++ b/drivers/media/usb/uvc/uvcvideo.h
@@ -91,6 +91,36 @@
#define UVC_GUID_FORMAT_Y16 \
{ 'Y', '1', '6', ' ', 0x00, 0x00, 0x10, 0x00, \
0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
+#define UVC_GUID_FORMAT_Y8I \
+ { 'Y', '8', 'I', ' ', 0x00, 0x00, 0x10, 0x00, \
+ 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
+#define UVC_GUID_FORMAT_Y12I \
+ { 'Y', '1', '2', 'I', 0x00, 0x00, 0x10, 0x00, \
+ 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
+#define UVC_GUID_FORMAT_Z16 \
+ { 'Z', '1', '6', ' ', 0x00, 0x00, 0x10, 0x00, \
+ 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
+#define UVC_GUID_FORMAT_RW10 \
+ { 'R', 'W', '1', '0', 0x00, 0x00, 0x10, 0x00, \
+ 0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
+#define UVC_GUID_FORMAT_INVZ \
+ { 'I', 'N', 'V', 'Z', 0x90, 0x2d, 0x58, 0x4a, \
+ 0x92, 0x0b, 0x77, 0x3f, 0x1f, 0x2c, 0x55, 0x6b}
+#define UVC_GUID_FORMAT_INZI \
+ { 'I', 'N', 'Z', 'I', 0x66, 0x1a, 0x42, 0xa2, \
+ 0x90, 0x65, 0xd0, 0x18, 0x14, 0xa8, 0xef, 0x8a}
+#define UVC_GUID_FORMAT_INVR \
+ { 'I', 'N', 'V', 'R', 0x90, 0x2d, 0x58, 0x4a, \
+ 0x92, 0x0b, 0x77, 0x3f, 0x1f, 0x2c, 0x55, 0x6b}
+#define UVC_GUID_FORMAT_INRI \
+ { 'I', 'N', 'R', 'I', 0x90, 0x2d, 0x58, 0x4a, \
+ 0x92, 0x0b, 0x77, 0x3f, 0x1f, 0x2c, 0x55, 0x6b}
+#define UVC_GUID_FORMAT_INVI \
+ { 'I', 'N', 'V', 'I', 0xdb, 0x57, 0x49, 0x5e, \
+ 0x8e, 0x3f, 0xf4, 0x79, 0x53, 0x2b, 0x94, 0x6f}
+#define UVC_GUID_FORMAT_RELI \
+ { 'R', 'E', 'L', 'I', 0x14, 0x13, 0x43, 0xf9, \
+ 0xa7, 0x5a, 0xee, 0x6b, 0xbf, 0x01, 0x2e, 0x23}
#define UVC_GUID_FORMAT_BY8 \
{ 'B', 'Y', '8', ' ', 0x00, 0x00, 0x10, 0x00, \
0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
@@ -100,7 +130,6 @@
#define UVC_GUID_FORMAT_M420 \
{ 'M', '4', '2', '0', 0x00, 0x00, 0x10, 0x00, \
0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
-
#define UVC_GUID_FORMAT_H264 \
{ 'H', '2', '6', '4', 0x00, 0x00, 0x10, 0x00, \
0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}
diff --git a/drivers/platform/tegra/tegra21_clocks.c b/drivers/platform/tegra/tegra21_clocks.c
index 0a0584f..cf0e277 100644
--- a/drivers/platform/tegra/tegra21_clocks.c
+++ b/drivers/platform/tegra/tegra21_clocks.c
@@ -1062,7 +1062,7 @@ static struct clk_ops tegra_super_ops = {
*/
static void tegra21_cpu_clk_init(struct clk *c)
{
- c->state = (!is_lp_cluster() == (c->u.cpu.mode == MODE_G)) ? ON : OFF;
+ c->state = ((!is_lp_cluster()) == (c->u.cpu.mode == MODE_G)) ? ON : OFF;
}
static int tegra21_cpu_clk_enable(struct clk *c)
diff --git a/drivers/usb/host/xhci-mem.c b/drivers/usb/host/xhci-mem.c
index 677f032..032b014 100644
--- a/drivers/usb/host/xhci-mem.c
+++ b/drivers/usb/host/xhci-mem.c
@@ -1436,8 +1436,9 @@ int xhci_endpoint_init(struct xhci_hcd *xhci,
break;
case USB_SPEED_HIGH:
/* Some devices get this wrong */
- if (usb_endpoint_xfer_bulk(&ep->desc))
- max_packet = 512;
+//Duo3D Kernel Patch, applied 13/05/2016
+ //if (usb_endpoint_xfer_bulk(&ep->desc))
+ // max_packet = 512;
/* bits 11:12 specify the number of additional transaction
* opportunities per microframe (USB 2.0, section 9.6.6)
*/
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 51f3740..2692503 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -401,6 +401,18 @@ struct v4l2_pix_format {
#define V4L2_PIX_FMT_VC1_ANNEX_L v4l2_fourcc('V', 'C', '1', 'L') /* SMPTE 421M Annex L compliant stream */
#define V4L2_PIX_FMT_VP8 v4l2_fourcc('V', 'P', '8', '0') /* VP8 */
+/* Intel RealSense-specific formats */
+#define V4L2_PIX_FMT_Y8I v4l2_fourcc('Y', '8', 'I', ' ') /* Greyscale 8-bit L/R interleaved */
+#define V4L2_PIX_FMT_Y12I v4l2_fourcc('Y', '1', '2', 'I') /* Greyscale 12-bit L/R interleaved */
+#define V4L2_PIX_FMT_Z16 v4l2_fourcc('Z', '1', '6', ' ') /* Depth data 16-bit */
+#define V4L2_PIX_FMT_RW10 v4l2_fourcc('R', 'W', '1', '0') /* Raw data 10-bit */
+#define V4L2_PIX_FMT_INVZ v4l2_fourcc('I', 'N', 'V', 'Z') /* 16 Depth */
+#define V4L2_PIX_FMT_INZI v4l2_fourcc('I', 'N', 'Z', 'I') /* 24 Depth/IR 16:8 */
+#define V4L2_PIX_FMT_INVR v4l2_fourcc('I', 'N', 'V', 'R') /* 16 Depth */
+#define V4L2_PIX_FMT_INRI v4l2_fourcc('I', 'N', 'R', 'I') /* 24 Depth/IR 16:8 */
+#define V4L2_PIX_FMT_INVI v4l2_fourcc('I', 'N', 'V', 'I') /* 8 IR */
+#define V4L2_PIX_FMT_RELI v4l2_fourcc('R', 'E', 'L', 'I') /* 8 IR alternating on off illumination */
+
/* Vendor-specific formats */
#define V4L2_PIX_FMT_CPIA1 v4l2_fourcc('C', 'P', 'I', 'A') /* cpia1 YUV */
#define V4L2_PIX_FMT_WNVA v4l2_fourcc('W', 'N', 'V', 'A') /* Winnov hw compress */
NOTE: Ignore the other diffs, they are from adding DuoMLX support (WIP) and just to get git to compile…
I assume this was a cross-compile…which tool chains were used?
Hello linuxdev,
the environment used was setup identical to here.
I am suspicious of my modifications as I did not completely reflash the jetson, I merely recompiled the Image and zImage, copied these into my /boot/ dir on the device (same version string) and then recompiled modules and copied the entire modules output folder over the target devices folder. The module seems to have compiled because modinfo uvcvideo shows a new version (not vermagic, version as defined with (i added the dirty)
MODULE_VERSION("1.1.1-dirty");
However the same problem persists with uvcvideo: Unknown video format. Interesting to note: the hex string is revered for the first block. But I did some googling and this seems normal. EG:
uvcvideo: Unknown video format 20493859-0000-0010-8000-00aa00389b71, the part in bold converted to chars reads: I8Y. But in the patch applied to uvcvideo.h it is Y8I . Just a quirk I noticed, but I assume this is normal behavior.
@RNZNobody and linuxdev, any update? I met the same issue with RNZNobody.
I do not know of updates, but something I did not think about until re-reading this has me thinking about part of one of the other posts:
“However the same problem persists with uvcvideo: Unknown video format. Interesting to note: the hex string is revered for the first block. But I did some googling and this seems normal. EG:
uvcvideo: Unknown video format 20493859-0000-0010-8000-00aa00389b71, the part in bold converted to chars reads: I8Y. But in the patch applied to uvcvideo.h it is Y8I . Just a quirk I noticed, but I assume this is normal behavior.”
Having bytes reversed might mean big-endian/little-endian mismatch. When directly reading bytes of the kernel or its drivers from the file system, they appear backwards because the kernel image is ordered like big endian within the actual Image file, but use of assembler and data is little endian (disassembling re-orders things, but chunks of data which are not disassembled my have different requirements).
Sorry, my research has taken me in a different direction to the R200 for now, so this is where I left it. But what linuxdev noticed in my earlier findings does make a lot of sense actually! Sadly I won’t have the time to go back and check that myself (nor would I know how to solve litle/BIG ordering issues), but I hope I documented my efforts enough that they are repeatable.
Also that was on 23.2 or so. Have not tried on newer firmware.
Good Luck
Likely the values would be read from the disk as if it were a network, then one of these C functions would be used:
htonl, htons, ntohl, ntohs
See “man htonl”, or one of the others above. Data from a disk can be treated exactly like data from a network.
From reading the above, it seems like the UVC patches were not applied correctly. This may be due to 32-bit issues.
For the 64-bit L4T 24.2.1, I have the RealSense R200 up and running on the TX1 using a patched UVC driver. The gory details:
Another issue not mentioned here is that you have to turn off USB autosuspend for the camera to work correctly.