Intel realsense d435i on NX devkit

Andrey1984 · May 30, 2020, 7:42am

Have anyone managed to get the camera working with NX?
In case of installation with apt-get method:

   sudo apt-key adv --keyserver keys.gnupg.net --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key 
    sudo add-apt-repository "deb http://realsense-hw-public.s3.amazonaws.com/Debian/apt-repo bionic main" -u
sudo apt update -y
sudo apt-get install librealsense2-utils librealsense2-dev -y""
rs-hello-realsense RealSense error calling rs2_pipeline_wait_for_frames(pipe:0x557f97e110): Frame didn't arrive within 15000

rs-distance works, however.

In case of building the latest sdk from source:
first it requires to add cuda paths to .bashrc and reload terminal to continue

export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
export PATH=$PATH:$CUDA_HOME/bin

wget https://github.com/IntelRealSense/librealsense/archive/v2.34.1.tar.gz
  tar -xvf v2.34.1.tar.gz 
  cd librealsense-2.34.1/
mkdir build
cd build
cmake -DBUILD_EXAMPLES=true -DFORCE_LIBUVC=true -DBUILD_WITH_CUDA=true ..
make -j6
sudo make install
rs-distance
There are 1 connected RealSense devices.
rs_error was raised when calling rs2_create_device(info_list:0x55ba4f63c0, index:0):
    failed to set power state

da0909 · June 1, 2020, 1:19am

the d415 can not work on NX, too. expect for a bug fix

DaneLLL · June 1, 2020, 2:12am

Hi,
Please try

$tar -xvf librealsense-2.19.1.tar.gz
$cd librealsense
$sudo apt update
$sudo apt upgrade
$sudo apt-get install git libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev
$sudo apt-get install libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev
$sudo cp config/99-realsense-libusb.rules /etc/udev/rules.d/
$sudo udevadm control --reload-rules && udevadm trigger
$mkdir build
$cd build
$cmake ../ -DCMAKE_BUILD_TYPE=Release -DBUILD_EXAMPLES=true
$sudo make uninstall
$make clean
$make
$sudo make install

And run realsense-viewer.

Andrey1984 · June 1, 2020, 8:57am

@DaneLLL Thank you for your response. I shall test the proposed solution. However, this implementation doesn’t include cuda, does it?

-DBUILD_WITH_CUDA=true

Could you also extend how to merge two partitions, both 14 gb which appear instantly after flashing NX devit with sdkmanager, please?
If to install CUDA using the sdkmanager the space will exhaust on the system partition so that the system partition capacity will be merely enough to build the intelrealsense sdk.
Attempt #1

       wget https://github.com/IntelRealSense/librealsense/archive/v2.35.0.tar.gz
        tar -xvf v2.35.0.tar.gz 
        cd librealsense-2.35.0/
         sudo apt update -y && sudo apt upgrade -y
        sudo apt-get install git libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev -y
        sudo apt-get install libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev -y
        sudo cp config/99-realsense-libusb.rules /etc/udev/rules.d/
        sudo udevadm control --reload-rules && udevadm trigger
        mkdir build
        cd build
        cmake ../ -DCMAKE_BUILD_TYPE=Release -DBUILD_EXAMPLES=true
        sudo make uninstall
        make clean
        make -j6
        sudo make install

Testing realsense-viewer:

Testing rs-distance: FAILED

$ rs-distance 
There are 0 connected RealSense devices.

Attempt N2;
The same but with the proposed 2.19.1 version
The results seems the same

/dev/2191/librealsense-2.19.1/build$ rs-distance 
There are 0 connected RealSense devices.

However, when I add apt repository and install librealsense2 with it it will work:

 /usr/bin/rs-distance
There are 1 connected RealSense devices.

Using device 0, an Intel RealSense D435I

However running /usr/bin/realsense-viewer [ version from apt-repository seems to fail]

DaneLLL · June 1, 2020, 11:13pm

Hi,
Is the issue specific to

-DBUILD_WITH_CUDA=true

Or you simply follow the steps and it does no twork?

Andrey1984 · June 2, 2020, 8:46am

Hi @DaneLLL ,
Thank you for following up.
This time I simply followed the steps. Without the cuda argument. First I built 2.35 version; tested with rs-distance, found that it won’t detect the device.
Then I built the exact version you proposed 2.19.1. It would neither found the realsense device.
Then I installed apt repository 2.32 version. It uses custom udev rule located in /lib/udev/rules.d/
and it seems work somehow. To a certain extent.

jmlilly4 · June 4, 2020, 9:36pm

I’m having a similar issue with the same hardware, a D435i and a Xavier NX. I can confirm following the steps listed above leaves realsense-viewer unable to detect the Realsense. I am able to detect and read from the device by installing version 2.32 from the apt repository as guided here (Intel’s Linux instructions; notably, I have to skip the dkms package but it “works” without this step) and here (the install script simply adds the same repository and apt-get installs the same packages).

However, while I’m able to run realsense-viewer and enable the depth and color streams successfully, when I disable and re-enable them, they stop working. Realsense-viewer reports “no frames received” for each until I either reconnect the hardware to the Jetson or use the “Hardware reset” option in realsense-viewer. People are reporting similar problems on this git issue. I’ve cross-posted my info there as well.

DaneLLL · June 4, 2020, 11:11pm

Hi,
Please check

We have updated the firmware and there is improvement. You may give it a try.

jmlilly4 · June 5, 2020, 3:19pm

That seems to have done it! Thank you!

Andrey1984 · June 6, 2020, 6:56pm

Dockerized d435i for ROS2;
Dockerfile:

FROM nvcr.io/nvidia/l4t-base:r32.4.2
RUN apt-get update -y
RUN apt install -y wget curl git
#RUN /bin/bash  -c 'echo "deb http://packages.ros.org/ros/ubuntu bionic main" > /etc/apt/sources.list.d/ros-latest.list'
#RUN  apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
RUN apt-get update -y
RUN locale-gen en_US en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 && LANG=en_US.UTF-8
RUN apt update && apt install curl gnupg2 lsb-release
RUN curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc |  apt-key add - && sh -c 'echo "deb http://packages.ros.org/ros2/ubuntu `lsb_release -cs` main" > /etc/apt/sources.list.d/ros2-latest.list'
RUN apt update -y
RUN apt-get update -y
ENV TZ America/Los_Angeles
RUN DEBIAN_FRONTEND="noninteractive" apt-get install -y tzdata git curl cmake wget mc mlocate 
RUN DEBIAN_FRONTEND="noninteractive" apt install -y ros-dashing-ros-base
RUN apt update -y 
RUN DEBIAN_FRONTEND="noninteractive" apt install -y ros-dashing-cv-bridge ros-dashing-librealsense2 ros-dashing-message-filters ros-dashing-image-transport
RUN DEBIAN_FRONTEND="noninteractive" apt install -y  libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev
RUN DEBIAN_FRONTEND="noninteractive" apt install -y  ros-dashing-realsense-camera-msgs ros-dashing-realsense-ros2-camera
RUN dpkg-reconfigure --frontend noninteractive tzdata
RUN rm -rf /var/lib/apt/lists
WORKDIR /home/nvidia/ros2_ws
RUN  mkdir -p ~/src && cd ~/src && git clone https://github.com/intel/ros2_intel_realsense.git
RUN  sh -c 'echo "deb [arch=amd64,arm64] http://repo.ros2.org/ubuntu/main `lsb_release -cs` main" > /etc/apt/sources.list.d/ros2-latest.list'
RUN curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc |  apt-key add -
RUN apt-get update -y
RUN apt install -y python3-colcon-common-extensions
RUN cd ~ &&  /bin/bash -c '. /opt/ros/dashing/setup.bash; colcon build --base-paths src/ros2_intel_realsense'

docker build . -t ros2_realsense

docker run -it --rm --net=host   -e DISPLAY=$DISPLAY --ipc=host --privileged -v /tmp/.X11-unix/:/tmp/.X11-unix/ -v /tmp/argus_socket:/tmp/argus_socket --cap-add SYS_PTRACE ros2_realsense /bin/bash -c '.  /opt/ros/dashing/setup.bash; source ~/install/local_setup.bash; realsense_ros2_camera'

However, when I am trying to implement it for ROS it seems built, executes, but throw multiple errors after the execution.
Any insights?
Dockerfile for ROS1

FROM nvcr.io/nvidia/l4t-base:r32.4.2
RUN apt-get update -y
RUN apt install -y wget software-properties-common apt-utils  curl git
RUN /bin/bash  -c 'echo "deb http://packages.ros.org/ros/ubuntu bionic main" > /etc/apt/sources.list.d/ros-latest.list'
RUN  apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
RUN apt-key adv --keyserver keys.gnupg.net --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE ||  apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key
RUN apt-key adv --keyserver keys.gnupg.net --recv-key C8B3A55A6F3EFCDE ||  apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key C8B3A55A6F3EFCDE
RUN add-apt-repository "deb http://realsense-hw-public.s3.amazonaws.com/Debian/apt-repo bionic main" -u
RUN apt-get update -y
ENV TZ America/Los_Angeles
RUN DEBIAN_FRONTEND="noninteractive" apt-get install -y tzdata git curl cmake wget mc mlocate 
RUN DEBIAN_FRONTEND="noninteractive" apt install -y ros-melodic-desktop clang-format ros-melodic-desktop-full python-pip python3-pip  python-rosdep python-rosinstall python-rosinstall-generator python-wstool build-essential 
RUN apt update -y
#RUN rm -rf /var/lib/apt/lists
RUN  apt-get install -y librealsense2-dev librealsense2-utils
RUN apt install -y ros-melodic-ddynamic-reconfigure
RUN dpkg-reconfigure --frontend noninteractive tzdata
RUN rm -rf /var/lib/apt/lists
WORKDIR /home/nvidia/ws
RUN mkdir src && cd src &&  git clone https://github.com/intel-ros/realsense.git -b development
RUN rosdep init && rosdep update


RUN /bin/bash -c '. /opt/ros/melodic/setup.bash;  cd /home/nvidia/ws; rosdep install --from-paths src --ignore-src; catkin_make'

docker build . -t librealsense1ros

docker run -it --rm --net=host   -e DISPLAY=$DISPLAY --ipc=host --privileged -v /tmp/.X11-unix/:/tmp/.X11-unix/ -v /tmp/argus_socket:/tmp/argus_socket --cap-add SYS_PTRACE librealsense1ros /bin/bash -c '.  /opt/ros/melodic/setup.bash; source devel/setup.bash; roslaunch realsense2_camera rs_camera.launch'

Expected frequency for depth = 30.00000                                                                                                            [ INFO] [1591514606.959555082]: Expected frequency for infra1 = 30.00000                                                                                                           [ INFO] [1591514607.039217389]: Expected frequency for infra2 = 30.00000                                                                                                           [ INFO] [1591514607.111038886]: Expected frequency for color = 30.00000                                                                                                            [ INFO] [1591514607.195585966]: setupStreams...                                                                                                                                    [ INFO] [1591514607.263808675]: insert Depth to Stereo Module                                                                                                                      [ INFO] [1591514607.263963491]: insert Color to RGB Camera                                                                                                                         [ INFO] [1591514607.264124643]: insert Infrared to Stereo Module                                                                                                                   [ INFO] [1591514607.264218211]: insert Infrared to Stereo Module                                                                                                                   [ INFO] [1591514607.264299684]: insert Gyro to Motion Module                                                                                                                       [ INFO] [1591514607.264366820]: insert Accel to Motion Module                                                                                                                      [ INFO] [1591514607.634880751]: SELECTED BASE:Depth, 0                                                                                                                             [ INFO] [1591514607.686268687]: RealSense Node Is Up!                                                                                                                               07/06 00:23:27,697 WARNING [546861736320] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61                              07/06 00:23:27,781 ERROR [546056429952] (synthetic-stream.cpp:48) Exception was thrown during user processing callback!                                                            07/06 00:23:27,790 WARNING [546861736320] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11               07/06 00:23:27,810 ERROR [546056429952] (synthetic-stream.cpp:48) Exception was thrown during user processing callback!                                                            07/06 00:23:27,818 ERROR [546316472704] (synthetic-stream.cpp:48) Exception was thrown during user processing callback!                                                            07/06 00:23:27,843 ERROR [546056429952] (synthetic-stream.cpp:48) Exception was thrown during user processing callback!

linked thread https://forums.developer.nvidia.com/t/realsensed400-camera-can-not-work-on-jetson-xavier-nx/125138/19v

DaneLLL · June 18, 2020, 4:55am

Hi @Andrey1984
So it works if you launch through realsense-viewer. When you proceed to install ROS2, it fails?

There is a project running ROS2 on Jetson Nano:
https://developer.nvidia.com/embedded/community/jetson-projects

But the camera is RPi cam ver.2, which goes through Argus. It is different from using USB cameras, but probably it still can be a reference.

Andrey1984 · June 18, 2020, 8:15am

No, it doesn’t fail totally. And yes it fails with graphics vizualization.
However, it hits a major roadblock of libGL processing which seems to be a common issue for running visual applications from l4t docker:

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to ‘/tmp/runtime-root’
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
[ERROR] [rviz2]: Failed to create an OpenGL context. BadValue (integer parameter out of range for operation)

Maybe you have an idea how to configure the libGL/swrast? In the docker container? Previously I got to the same issie while trying to use Autoware.Auto on Xavier AGX. Implementing autoware auto examples on nvidia Jetson aarch64 - ROS Answers: Open Source Q&A Forum

dan.novischi · January 8, 2023, 7:02pm

Hi,

If you can… you can try to replicate what I have done for the Jetson Nano in Realsense D435i Installation for ROS Noetic on Ubuntu 18.04. In the tutorial I build everything from source.

If you give it a go let us know how it goes.

Andrey1984 · January 29, 2023, 6:54pm

@dan.novischi
the correct url seems
https://github.com/dnovischi/jetson-tutorials/blob/main/jetson-nano-realsense-notetic-install.md

github.com

dnovischi/jetson-tutorials/blob/main/jetson-nano-realsense-notetic-install.md

# Realsense D435i Installation for ROS Noetic on Ubuntu 18.04 and Ubuntu 20.04

In this tutorial I will describe the steps needed to fully install the Realsense D435i on the official Jetson Nano Ubuntu 18.04 distribution and Ubuntu 20.04 , with the latest Nvidia Jetpack installed based on a ROS Noetic deployment. If you don't have such a distribution installed, please complete either the [Jetson Nano Ubuntu 18.04 Full Install](jetson-nano-ubuntu-18-04-install.md) or the [Ubuntu 20.04 Install Through Distribution Upgrade](jetson-nano-ubuntu-20-04-install.md), and the [ROS Noetic Install on Jetson Nano for Ubuntu 18.04 and Ubuntu 20.04](jetson-nano-ros-noetic-install.md) tutorials.

The tutorial assumes that the aforementioned installs are done, while it details the steps to properly install [librealsense](https://github.com/IntelRealSense/librealsense), [realsense-ros warpper](https://github.com/IntelRealSense/realsense-ros/tree/ros1-legacy) and the corresponding [realsense firmare](https://dev.intelrealsense.com/docs/firmware-releases).

All the steps described in this tutorial are exactly the same for both Ubuntu 18.04 and Ubuntu 20.04.

Please note that the ROS Realsense warpper 2.3.2 that is compatible with ROS Noetic [2] requires librealsense version 2.50.0 [7]. While the recommaned Realsense D435i Ffirmware version is 5.13.0.50 [3].

## Librealsense Installation

1. Disconnect the Realsense USB cable from the Jetson Nano.
2. Make sure your Jetson is up to date

```bash
sudo apt-get update
sudo apt-get upgrade
```

This file has been truncated. show original

dan.novischi · January 29, 2023, 11:04pm

The links have changed due to the tutorials updates in the repository. You can find the repository below. For Realsense follow the steps in tutorial 4. You may want to star/watch the repo to receive the notification about updates.