Hi, I had compile and install librealsense on my jetson nano, firt i tried version 1.12.1, my realsense F200 works fine, but the SR300 can not get depth image, no error, just a blank image. then i tried version2, got an error like “The requested pixel format ‘INZI’ is not natively supported by the Linux kernel…”, how do i patch the kernel?
thanks
Hi rogercao0112,
We haven’t tested with Intel RealSense series cameras on Nano yet, you can refer to from this repo [url]https://github.com/jetsonhacks/buildLibrealsense2TX[/url] to see if can help.
Thanks
Look at this repository instead:
https://github.com/JetsonHacksNano/installLibrealsense
It is still a work in progress, but it’s pretty close. Having trouble identifying the D435i camera.
I’m writing a driver for the donkeycar robotics project. I’ve used https://github.com/JetsonHacksNano/installLibrealsense to install on nano. I used the installlibrealsense.sh script. Realsense-viewer worked well. demos worked well. The python code could not import pyrealsense2. Pip install pyrealsense2 did not work (could not find it). I rebuild from source, after which python code could import pyrealsense2; however the code could not connect to the camera - I have a Intel Realsense D435i. Error is:
File "realsense435i.py", line 68, in __init__
profile = pipeline.start(config)
RumtimeError: Couldn't resolve requests
This code runs find on my MSI gaming laptop running Ubuntu 18.04. See the code here:https://raw.githubusercontent.com/autorope/donkeycar/RealSense-part/donkeycar/parts/realsense435i.py
version I used is pasted below:
"""
Author: Ed Murphy
File: realsense435i.py
Date: April 14 2019
Notes: Donkeycar part for the Intel Realsense depth cameras D435 and D435i.
"""
import time
import logging
import numpy as np
import pyrealsense2 as rs
WIDTH = 424
HEIGHT = 240
CHANNELS = 3
class RealSense435i(object):
"""
Donkeycar part for the Intel Realsense depth cameras D435 and D435i.
The Intel Realsense D435i camera is a device which uses an imu, twin fisheye cameras,
and an Movidius chip to stream a depth map along with an rgb image and optionally,
accelerometer and gyro data (the 'i' variant has an IMU, the non-i variant does not)
NOTE: this ALWAYS captures 424 pixels wide x 240 pixels high x RGB at 60fps.
If an image width, height or depth are passed with different values,
the the outputs will be scaled to the requested size on the way out.
"""
def __init__(self, width = WIDTH, height = HEIGHT, channels = CHANNELS, enable_rgb=True, enable_depth=True, enable_imu=False, device_id = None):
self.device_id = device_id # "923322071108" # serial number of device to use or None to use default
self.enable_imu = enable_imu
self.enable_rgb = enable_rgb
self.enable_depth = enable_depth
self.width = width
self.height = height
self.channels = channels
self.resize = (width != WIDTH) or (height != height) or (channels != CHANNELS)
if self.resize:
print("The output images will be resized from {} to {}. This requires opencv.".format((WIDTH, HEIGHT, CHANNELS), (self.width, self.height, self.channels)))
# Configure streams
self.imu_pipeline = None
if self.enable_imu:
self.imu_pipeline = rs.pipeline()
imu_config = rs.config()
if None != self.device_id:
imu_config.enable_device(self.device_id)
imu_config.enable_stream(rs.stream.accel, rs.format.motion_xyz32f, 63) # acceleration
imu_config.enable_stream(rs.stream.gyro, rs.format.motion_xyz32f, 200) # gyroscope
imu_profile = self.imu_pipeline.start(imu_config)
self.pipeline = None
if self.enable_depth or self.enable_rgb:
self.pipeline = rs.pipeline()
config = rs.config()
# if we are provided with a specific device, then enable it
if None != self.device_id:
config.enable_device(self.device_id)
if self.enable_depth:
config.enable_stream(rs.stream.depth, 848, 480, rs.format.z16, 60) # depth
if self.enable_rgb:
config.enable_stream(rs.stream.color, 424, 240, rs.format.bgr8, 60) # rgb
# Start streaming
profile = self.pipeline.start(config)
# Getting the depth sensor's depth scale (see rs-align example for explanation)
if self.enable_depth:
depth_sensor = profile.get_device().first_depth_sensor()
depth_scale = depth_sensor.get_depth_scale()
print("Depth Scale is: ", depth_scale)
if self.enable_rgb:
# Create an align object
# rs.align allows us to perform alignment of depth frames to others frames
# The "align_to" is the stream type to which we plan to align depth frames.
align_to = rs.stream.color
self.align = rs.align(align_to)
time.sleep(1) # let camera warm up
# initialize frame state
self.color_image = None
self.depth_image = None
self.acceleration_x = None
self.acceleration_y = None
self.acceleration_z = None
self.gyroscope_x = None
self.gyroscope_y = None
self.gyroscope_z = None
self.frame_count = 0
self.start_time = time.time()
self.frame_time = self.start_time
def _poll(self):
last_time = self.frame_time
self.frame_time = time.time() - self.start_time
self.frame_count += 1
#
# get the frames
#
try:
if self.enable_rgb or self.enable_depth:
frames = self.pipeline.wait_for_frames(200 if (self.frame_count > 1) else 10000) # wait 10 seconds for first frame
if self.enable_imu:
imu_frames = self.imu_pipeline.wait_for_frames(200 if (self.frame_count > 1) else 10000)
except Exception as e:
logging.error(e)
return
#
# convert camera frames to images
#
if self.enable_rgb or self.enable_depth:
# Align the depth frame to color frame
aligned_frames = self.align.process(frames) if self.enable_depth and self.enable_rgb else None
depth_frame = aligned_frames.get_depth_frame() if aligned_frames is not None else frames.get_depth_frame()
color_frame = aligned_frames.get_color_frame() if aligned_frames is not None else frames.get_color_frame()
# Convert images to numpy arrays
self.depth_image = np.asanyarray(depth_frame.get_data()) if self.enable_depth else None
self.color_image = np.asanyarray(color_frame.get_data()) if self.enable_rgb else None
if self.resize:
import cv2
if self.width != WIDTH or self.height != HEIGHT:
self.color_image = cv2.resize(self.color_image, (self.width, self.height), cv2.INTER_NEAREST) if self.enable_rgb else None
self.depth_image = cv2.resize(self.depth_image, (self.width, self.height), cv2.INTER_NEAREST) if self.enable_depth else None
if self.channels != CHANNELS:
self.color_image = cv2.cvtColor(self.color_image, cv2.COLOR_RGB2GRAY) if self.enable_rgb else None
#
# output imu data as discrete values in the same order as the Mpu6050 code
# so we can be compatible with any other parts that consume imu data.
#
if self.enable_imu:
acceleration = imu_frames.first_or_default(rs.stream.accel, rs.format.motion_xyz32f).as_motion_frame().get_motion_data()
self.acceleration_x = acceleration.x
self.acceleration_y = acceleration.y
self.acceleration_z = acceleration.z
gyroscope = imu_frames.first_or_default(rs.stream.gyro, rs.format.motion_xyz32f).as_motion_frame().get_motion_data()
self.gyroscope_x = gyroscope.x
self.gyroscope_y = gyroscope.y
self.gyroscope_z = gyroscope.z
logging.debug("imu frame {} in {} seconds: \n\taccel = {}, \n\tgyro = {}".format(
str(self.frame_count),
str(self.frame_time - last_time),
str((self.acceleration_x, self.acceleration_y, self.acceleration_z)),
str((self.gyroscope_x, self.gyroscope_y, self.gyroscope_z))))
def update(self):
"""
When running threaded, update() is called from the background thread
to update the state. run_threaded() is called to return the latest state.
"""
while self.running:
self._poll()
def run_threaded(self):
"""
Return the lastest state read by update(). This will not block.
All 4 states are returned, but may be None if the feature is not enabled when the camera part is constructed.
For gyroscope, x is pitch, y is yaw and z is roll.
:return: (rbg_image: nparray, depth_image: nparray, acceleration: (x:float, y:float, z:float), gyroscope: (x:float, y:float, z:float))
"""
return self.color_image, self.depth_image, self.acceleration_x, self.acceleration_y, self.acceleration_z, self.gyroscope_x, self.gyroscope_y, self.gyroscope_z
def run(self):
"""
Read and return frame from camera. This will block while reading the frame.
see run_threaded() for return types.
"""
self._poll()
return self.run_threaded()
def shutdown(self):
self.running = False
time.sleep(0.1)
if self.imu_pipeline is not None:
self.imu_pipeline.stop()
if self.pipeline is not None:
self.pipeline.stop()
#
# self test
#
if __name__ == "__main__":
show_opencv_window = False # True to show images in opencv window: note that default donkeycar environment is not configured for this.
if show_opencv_window:
import cv2
enable_rgb = True
enable_depth = True
enable_imu = True
device_id = None
width = 212
height = 120
channels = 3
profile_frames = 0 # set to non-zero to calculate the max frame rate using given number of frames
try:
#
# for D435i, enable_imu can be True, for D435 enable_imu should be false
#
camera = RealSense435i(
width=width, height=height, channels=channels,
enable_rgb=enable_rgb, enable_depth=enable_depth, enable_imu=enable_imu, device_id=device_id)
frame_count = 0
start_time = time.time()
frame_time = start_time
while True:
#
# read data from camera
#
color_image, depth_image, acceleration_x, acceleration_y, acceleration_z, gyroscope_x, gyroscope_y, gyroscope_z = camera.run()
# maintain frame timing
frame_count += 1
last_time = frame_time
frame_time = time.time()
if enable_imu and not profile_frames:
print("imu frame {} in {} seconds: \n\taccel = {}, \n\tgyro = {}".format(
str(frame_count),
str(frame_time - last_time),
str((acceleration_x, acceleration_y, acceleration_z)),
str((gyroscope_x, gyroscope_y, gyroscope_z))))
# Show images
if show_opencv_window and not profile_frames:
cv2.namedWindow('RealSense', cv2.WINDOW_AUTOSIZE)
if enable_rgb or enable_depth:
# make sure depth and color images have same number of channels so we can show them together in the window
if 3 == channels:
depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET) if enable_depth else None
else:
depth_colormap = cv2.cvtColor(cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET), cv2.COLOR_RGB2GRAY) if enable_depth else None
# Stack both images horizontally
images = None
if enable_rgb:
images = np.hstack((color_image, depth_colormap)) if enable_depth else color_image
elif enable_depth:
images = depth_colormap
if images is not None:
cv2.imshow('RealSense', images)
# Press esc or 'q' to close the image window
key = cv2.waitKey(1)
if key & 0xFF == ord('q') or key == 27:
cv2.destroyAllWindows()
break
if profile_frames > 0:
if frame_count == profile_frames:
print("Aquired {} frames in {} seconds for {} fps".format(str(frame_count), str(frame_time - start_time), str(frame_count / (frame_time - start_time))))
break
else:
time.sleep(0.05)
finally:
camera.shutdown()
Unfortunately the forum doesn’t support markdown. You need to use the “code” bb tags.
Update: If I don’t pass config for the pipeline (so everything is defaults), then it will work. If I use any config, even with values that match defaults, it does not work correctly; it fails trying to start the pipeline. It would be great to have an early check for valid config values and fail with a clear message if config is not valid for the camera/library. Again, this code all works on my gaming pc running ubuntu 18.04, but not on Jetson Nano running 18.04. librealsense 2.31 and 2.32 both had this problem on nano.