Realsense Camera fails on Joystick and mapping application - RealSense error calling rs2_pipeline_start_with_config: Couldn't resolve requests

gerardom · July 21, 2020, 9:15pm

Hi, I’m testing the joystick application for Jetson Nano, and even though the control works, the Intel Realsense camera doesn’t seem to function. This is the log that I’m getting:

2020-07-21 13:56:23.033 WARN engine/alice/backend/codelet_canister.cpp@225: Codelet ‘websight/isaac.sight.AliceSight’ was not added to scheduler because no tick method is specified.
2020-07-21 13:56:23.131 WARN packages/realsense/RealsenseCamera.cpp@42: Realsense recommended firmware version is 05.11.15.00, currently using firmware version 05.12.05.00
2020-07-21 13:56:23.511 ERROR engine/alice/components/Codelet.cpp@229: Component ‘2d_kaya.kaya_hardware.camera/realsense’ of type ‘isaac::RealsenseCamera’ reported FAILURE:

RealSense error calling rs2_pipeline_start_with_config(pipe:0x7f88068d40, config:0x7f880e8d20): Couldn't resolve requests

2020-07-21 13:56:23.511 ERROR engine/alice/backend/event_manager.cpp@42: Stopping node ‘2d_kaya.kaya_hardware.camera’ because it reached status ‘FAILURE’

I have tried also on the mapping application, and is the same issue (web interface doesn’t show video). The camera works fine if I start the realsense-viewer manually.

Has anyone experienced this issue?

MikhailFilimonov · July 22, 2020, 10:17am

Hi @gerardom and thank you for reaching out to us.
Please make sure you’re using the RealSense D435 (not D435i) camera with firmware 5.11.15

gerardom · July 22, 2020, 7:00pm

Thanks Mikhail, I’m using the Kaya original BOM (including the RealSense 435), and have downrev to FW 5.11.15. The warning to use this specific version went away, but the error is still the same.

log:
2020-07-22 11:50:26.507 INFO packages/sight/WebsightServer.cpp@217: Please open Chrome Browser and navigate to http://:3000

2020-07-22 11:50:26.507 WARN engine/alice/backend/codelet_canister.cpp@225: Codelet ‘websight/isaac.sight.AliceSight’ was not added to scheduler because no tick method is specified.

2020-07-22 11:50:27.040 ERROR engine/alice/components/Codelet.cpp@229: Component ‘2d_kaya.kaya_hardware.camera/realsense’ of type ‘isaac::RealsenseCamera’ reported FAILURE:

RealSense error calling rs2_pipeline_start_with_config(pipe:0x7f6c06ef90, config:0x7f6c0e9690): Couldn’t resolve requests

2020-07-22 11:50:27.040 ERROR engine/alice/backend/event_manager.cpp@42: Stopping node ‘2d_kaya.kaya_hardware.camera’ because it reached status ‘FAILURE’

2020-07-22 11:50:27.040 WARN engine/alice/backend/codelet_canister.cpp@225: Codelet ‘2d_kaya.kaya_hardware.camera/realsense’ was not added to scheduler because no tick method is specified.

Any other ideas?

MikhailFilimonov · July 22, 2020, 9:06pm

Thank you for providing the details on this matter.
Does the apps/samples/realsense_camera work normally on the robot?
Try running sudo jetson_clocks and put Jetson Nano into maximum power mode with sudo nvpmodel -m 0 before starting an Isaac application.
Please share the JetPack version you’re running and the output of v4l2-ctl --list-devices

gerardom · August 5, 2020, 11:38pm

I’m running Jetpack 4.3, and the realsense-camera app from the sample directory has similar issues

kaya@Kaya:~$ sudo jetson_clocks
[sudo] password for kaya:
kaya@Kaya:~$ sudo nvpmodel -m 0
kaya@Kaya:~$ cd deploy/media/realsense_camera-pkg/
kaya@Kaya:~/deploy/media/realsense_camera-pkg$ ./apps/samples/realsense_camera/realsense_camera
2020-08-05 16:24:01.467 INFO engine/alice/tools/websight.cpp@166: Loading websight…
2020-08-05 16:24:01.468 WARN engine/alice/backend/application_json_loader.cpp@174: This application does not have an explicit scheduler configuration. One will be autogenerated to the best of the system’s abilities if possible.
2020-08-05 16:24:01.469 INFO engine/alice/backend/redis_backend.cpp@40: Successfully connected to Redis server.

2020-08-05 16:24:01.469 WARN engine/alice/backend/backend.cpp@201: This application does not have an execution group configuration. One will be autogenerated to the best of the systems abilities if possible.
2020-08-05 16:24:01.469 WARN engine/gems/scheduler/scheduler.cpp@337: No default execution groups specified. Attempting to create scheduler configuration for 4 remaining cores. This may be non optimal for the system and application.
2020-08-05 16:24:01.469 INFO engine/gems/scheduler/scheduler.cpp@290: Scheduler execution groups are:
2020-08-05 16:24:01.469 INFO engine/gems/scheduler/scheduler.cpp@299: BlockerGroup: Cores = [3], Workers = No
2020-08-05 16:24:01.469 INFO engine/gems/scheduler/scheduler.cpp@299: WorkerGroup: Cores = [0, 1, 2], Workers = Yes
2020-08-05 16:24:01.508 INFO engine/alice/backend/modules.cpp@226: Loaded module ‘packages/realsense/librealsense_module.so’: Now has 45 components total
2020-08-05 16:24:01.527 INFO engine/alice/backend/modules.cpp@226: Loaded module ‘packages/rgbd_processing/librgbd_processing_module.so’: Now has 51 components total
2020-08-05 16:24:01.544 INFO engine/alice/backend/modules.cpp@226: Loaded module ‘packages/sight/libsight_module.so’: Now has 54 components total
2020-08-05 16:24:01.567 INFO engine/alice/backend/modules.cpp@226: Loaded module ‘packages/viewers/libviewers_module.so’: Now has 83 components total
2020-08-05 16:24:01.567 DEBUG engine/alice/application.cpp@348: Loaded 83 components: isaac::RealsenseCamera, isaac::alice::BufferAllocatorReport, isaac::alice::ChannelMonitor, isaac::alice::CheckJetsonPerformanceModel, isaac::alice::CheckOperatingSystem, isaac::alice::Config, isaac::alice::ConfigBridge, isaac::alice::ConfigLoader, isaac::alice::Failsafe, isaac::alice::FailsafeHeartbeat, isaac::alice::InteractiveMarkersBridge, isaac::alice::JsonToProto, isaac::alice::LifecycleReport, isaac::alice::MessageLedger, isaac::alice::MessagePassingReport, isaac::alice::NodeStatistics, isaac::alice::Pose, isaac::alice::Pose2Comparer, isaac::alice::PoseFromFile, isaac::alice::PoseInitializer, isaac::alice::PoseMessageInjector, isaac::alice::PoseToFile, isaac::alice::PoseToMessage, isaac::alice::PoseTree, isaac::alice::PoseTreeJsonBridge, isaac::alice::PoseTreeRelink, isaac::alice::ProtoToJson, isaac::alice::PyCodelet, isaac::alice::Random, isaac::alice::Recorder, isaac::alice::RecorderBridge, isaac::alice::Replay, isaac::alice::ReplayBridge, isaac::alice::Scheduling, isaac::alice::Sight, isaac::alice::SightChannelStatus, isaac::alice::Subgraph, isaac::alice::Subprocess, isaac::alice::TcpPublisher, isaac::alice::TcpSubscriber, isaac::alice::Throttle, isaac::alice::TimeOffset, isaac::alice::TimeSynchronizer, isaac::alice::UdpPublisher, isaac::alice::UdpSubscriber, isaac::map::Map, isaac::map::ObstacleAtlas, isaac::map::OccupancyGridMapLayer, isaac::map::PolygonMapLayer, isaac::map::WaypointMapLayer, isaac::navigation::DistanceMap, isaac::navigation::NavigationMap, isaac::navigation::RangeScanModelClassic, isaac::navigation::RangeScanModelFlatloc, isaac::rgbd_processing::DepthEdges, isaac::rgbd_processing::DepthImageFlattening, isaac::rgbd_processing::DepthImageToPointCloud, isaac::rgbd_processing::DepthNormals, isaac::rgbd_processing::DepthPoints, isaac::rgbd_processing::FreespaceFromDepth, isaac::sight::AliceSight, isaac::sight::SightWidget, isaac::sight::WebsightServer, isaac::viewers::BinaryMapViewer, isaac::viewers::ColorCameraViewer, isaac::viewers::DepthCameraViewer, isaac::viewers::Detections3Viewer, isaac::viewers::DetectionsViewer, isaac::viewers::FiducialsViewer, isaac::viewers::FlatscanViewer, isaac::viewers::GoalViewer, isaac::viewers::ImageKeypointViewer, isaac::viewers::LidarViewer, isaac::viewers::MosaicViewer, isaac::viewers::ObjectViewer, isaac::viewers::OccupancyMapViewer, isaac::viewers::PointCloudViewer, isaac::viewers::PoseTrailViewer, isaac::viewers::SegmentationCameraViewer, isaac::viewers::SegmentationViewer, isaac::viewers::SkeletonViewer, isaac::viewers::TensorViewer, isaac::viewers::TrajectoryListViewer,
2020-08-05 16:24:01.570 WARN engine/alice/application.cpp@164: The function Application::findComponentByName is deprecated. Please use getNodeComponentOrNull instead. Note that the new method requires a node name instead of a component name. (argument: ‘websight/isaac.sight.AliceSight’)
2020-08-05 16:24:01.570 INFO engine/alice/application.cpp@255: Starting application ‘realsense_camera’ (instance UUID: ‘bebf82d2-d772-11ea-aee2-5d24f609e236’) …
2020-08-05 16:24:01.570 DEBUG engine/gems/scheduler/execution_groups.cpp@476: Launching 0 pre-start job(s)
2020-08-05 16:24:01.570 DEBUG engine/gems/scheduler/execution_groups.cpp@485: Replaying 0 pre-start event(s)
2020-08-05 16:24:01.570 DEBUG engine/gems/scheduler/execution_groups.cpp@476: Launching 0 pre-start job(s)
2020-08-05 16:24:01.570 DEBUG engine/gems/scheduler/execution_groups.cpp@485: Replaying 0 pre-start event(s)
2020-08-05 16:24:01.570 INFO engine/alice/backend/asio_backend.cpp@33: Starting ASIO service
2020-08-05 16:24:01.572 INFO packages/sight/WebsightServer.cpp@216: Sight webserver is loaded
2020-08-05 16:24:01.572 INFO packages/sight/WebsightServer.cpp@217: Please open Chrome Browser and navigate to http://:3000
2020-08-05 16:24:01.572 WARN engine/alice/backend/codelet_canister.cpp@225: Codelet ‘websight/isaac.sight.AliceSight’ was not added to scheduler because no tick method is specified.
2020-08-05 16:24:01.573 WARN engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-08-05 16:24:01.573 WARN engine/alice/backend/codelet_canister.cpp@225: Codelet ‘_check_operating_system/isaac.alice.CheckOperatingSystem’ was not added to scheduler because no tick method is specified.
2020-08-05 16:24:01.573 WARN engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-08-05 16:24:01.573 WARN engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-08-05 16:24:02.015 ERROR engine/alice/components/Codelet.cpp@229: Component ‘camera/realsense’ of type ‘isaac::RealsenseCamera’ reported FAILURE:

** RealSense error calling rs2_pipeline_start_with_config(pipe:0x7f640572f0, config:0x7f640d4950): Couldn’t resolve requests**

2020-08-05 16:24:02.015 ERROR engine/alice/backend/event_manager.cpp@42: Stopping node ‘camera’ because it reached status ‘FAILURE’
2020-08-05 16:24:02.015 WARN engine/alice/backend/codelet_canister.cpp@225: Codelet ‘camera/realsense’ was not added to scheduler because no tick method is specified.
2020-08-05 16:24:11.570 INFO engine/alice/backend/allocator_backend.cpp@57: Optimized memory CPU allocator.
2020-08-05 16:24:11.570 INFO engine/alice/backend/allocator_backend.cpp@66: Optimized memory CUDA allocator.
2020-08-05 16:24:20.997 INFO packages/sight/WebsightServer.cpp@113: Server connected / 1
^C2020-08-05 16:25:03.648 INFO engine/alice/application.cpp@274: Stopping application ‘realsense_camera’ (instance UUID: ‘bebf82d2-d772-11ea-aee2-5d24f609e236’) …
2020-08-05 16:25:03.648 INFO engine/alice/backend/backend.cpp@134: Backend is shutting down…
2020-08-05 16:25:03.659 INFO packages/sight/WebsightServer.cpp@124: Disconnected
2020-08-05 16:25:03.659 INFO packages/sight/WebsightServer.cpp@450: =====================================================================================
2020-08-05 16:25:03.659 INFO packages/sight/WebsightServer.cpp@451: | Websight Statistics Report |
2020-08-05 16:25:03.659 INFO packages/sight/WebsightServer.cpp@452: =====================================================================================
2020-08-05 16:25:03.659 INFO packages/sight/WebsightServer.cpp@453: | Name | Bandwidth | Total | Count |
2020-08-05 16:25:03.659 INFO packages/sight/WebsightServer.cpp@454: -------------------------------------------------------------------------------------
2020-08-05 16:25:03.659 INFO packages/sight/WebsightServer.cpp@473: =====================================================================================
2020-08-05 16:25:03.669 INFO engine/alice/backend/asio_backend.cpp@49: Stopping ASIO service
2020-08-05 16:25:03.679 DEBUG engine/gems/scheduler/execution_groups.cpp@505: Stopping all threads for execution group BlockerGroup…
2020-08-05 16:25:03.679 DEBUG engine/gems/scheduler/execution_groups.cpp@509: Stopping all threads DONE
2020-08-05 16:25:03.679 DEBUG engine/gems/scheduler/execution_groups.cpp@505: Stopping all threads for execution group WorkerGroup…
2020-08-05 16:25:03.679 DEBUG engine/gems/scheduler/execution_groups.cpp@509: Stopping all threads DONE
2020-08-05 16:25:03.679 INFO engine/alice/backend/backend.cpp@149: Backend is shutting down… DONE
2020-08-05 16:25:03.679 INFO engine/alice/backend/backend.cpp@153: codelets:
|=====================================================================================================================|
| Job Statistics Report (regular) |
|=====================================================================================================================|
| Name | Count | Time (Median - 90% - Max) [ms] | Load (%) | Late (%) |
|---------------------------------------------------------------------------------------------------------------------|
| isaac.alice.BufferAllocatorReport | 21 | 11.10 | 16.55 | 24.13 | 0.3 % | 0.0 % |
| isaac.alice.ConfigBridge | 1 | 1.19 | 1.19 | 1.19 | 0.0 % | 0.0 % |
| InteractiveMarkersBridge | 1242 | 0.04 | 0.06 | 2.29 | 0.1 % | 0.0 % |
| isaac.alice.LifecycleReport | 249 | 0.02 | 0.03 | 9.21 | 0.0 % | 0.0 % |
| isaac.alice.MessagePassingReport | 21 | 28.60 | 45.33 | 76.96 | 1.0 % | 0.0 % |
| PoseTreeJsonBridge | 1242 | 0.15 | 0.18 | 1.61 | 0.2 % | 0.0 % |
| NodeStatistics | 63 | 3.17 | 3.67 | 4.91 | 0.2 % | 0.0 % |
| depth_to_pointcloud | 0 | 0.00 | 0.00 | -inf | 0.0 % | 0.0 % |
| viewer | 0 | 0.00 | 0.00 | -inf | 0.0 % | 0.0 % |
| color | 0 | 0.00 | 0.00 | -inf | 0.0 % | 0.0 % |
| depth | 0 | 0.00 | 0.00 | -inf | 0.0 % | 0.0 % |
| left_ir | 0 | 0.00 | 0.00 | -inf | 0.0 % | 0.0 % |
| right_ir | 0 | 0.00 | 0.00 | -inf | 0.0 % | 0.0 % |
| WebsightServer | 11745 | 5.23 | 5.32 | 60.31 | 98.2 % | 0.0 % |
|=====================================================================================================================|
kaya@Kaya:~/deploy/media/realsense_camera-pkg$

Got to work the v4l2-ctl command, and this is the output:

kaya@Kaya:~$ v4l2-ctl --list-devices
Intel(R) RealSense™ Depth Ca (usb-70090000.xusb-2.3):
/dev/video0
/dev/video1
/dev/video2

kaya@Kaya:~$

any other advice?

Thanks

gerardom · August 6, 2020, 4:38am

Updated the Jetpack to 4.4 and still the same issue.

gerardom · October 5, 2020, 11:25pm

@MikhailFilimonov, any other pointers on how to fix this? I have reinstalled again with the latest SDK, and image, and getting the same error. Thanks

Andrey1984 · October 10, 2020, 8:56pm

we are able to use realsense d435, but not d435i
it can be used with sight, as per my experience
it requires to use specific firmware version of the device tough
e.g.

./run ./apps/samples/stereo_vo/svo_realsense.py

gerardom · October 12, 2020, 6:41pm

@Andrew1984 I’m using the Intel RealSense D435, and found what is the problem for this issue. The constrain is that the camera is able to support specific resolutions/refresh rate, and it will throw the rs2_pipeline_start_with_config when the program calls for a configuration that can’t be supported it. I found the information on the Real Sense forum here. The issue with the kaya on the joystick application is that the resolution on the file kaya_hardware.subgraph.json calls for 848x480 (“cols”: 848,) and that is not supported on the camera that I have. I updated it to 640 and the application starts now.

To find the supported modes on the D435 you need to run the command rs-enumerate-devices -o

Log example below:

user@KayaDevice:~$ rs-enumerate-devices -o
Device info:
Name : Intel RealSense D435
Serial Number : xxxxxxxxxx91
Firmware Version : 05.12.08.200
Recommended Firmware Version : 05.12.08.200
Physical Port : /sys/devices/70090000.xusb/usb1/1-2/1-2.3/1-2.3:1.0/video4linux/video0
Debug Op Code : 15
Advanced Mode : YES
Product Id : 0B07
Camera Locked : YES
Usb Type Descriptor : 2.1
Product Line : D400
Asic Serial Number : 939423026988
Firmware Update Id : 939423026988

Options for Stereo Module
Supported options: min max step default
Exposure : 1 … 165000 1 8500
Gain : 16 … 248 1 16
Enable Auto Exposure : 0 … 1 1 1
Visual Preset : 0 … 6 1 0
Laser Power : 0 … 360 30 150
Emitter Enabled : 0 … 2 1 1
Frames Queue Size : 0 … 32 1 16
Error Polling Enabled : 0 … 1 1 0
Output Trigger Enabled : 0 … 1 1 0
Depth Units : 1e-06… 0.01 1e-06 0.001
Stereo Baseline : 50.1575… 50.1575 0 50.1575
Inter Cam Sync Mode : 0 … 258 1 0
Emitter On Off : 0 … 1 1 0
Global Time Enabled : 0 … 1 1 1
Emitter Always On : 0 … 1 1 0
Hdr Enabled : 0 … 1 1 0
Sequence Name : 0 … 3 1 1
Sequence Size : 2 … 2 1 2
Sequence Id : 0 … 2 1 0

Options for RGB Camera
Supported options: min max step default
Backlight Compensation : 0 … 1 1 0
Brightness : -64 … 64 1 0
Contrast : 0 … 100 1 50
Exposure : 1 … 10000 1 166
Gain : 0 … 128 1 64
Gamma : 100 … 500 1 300
Hue : -180 … 180 1 0
Saturation : 0 … 100 1 64
Sharpness : 0 … 100 1 50
White Balance : 2800 … 6500 10 4600
Enable Auto Exposure : 0 … 1 1 1
Enable Auto White Balance : 0 … 1 1 1
Frames Queue Size : 0 … 32 1 16
Power Line Frequency : 0 … 2 1 3
Auto Exposure Priority : 0 … 1 1 0
Global Time Enabled : 0 … 1 1 1

Stream Profiles supported by Stereo Module
Supported modes:
stream resolution fps format
Infrared 1 1280x720 @ 6Hz Y8
Infrared 1 848x480 @ 10Hz Y8
Infrared 1 848x480 @ 6Hz Y8
Infrared 1 640x480 @ 30Hz Y8
Infrared 1 640x480 @ 15Hz Y8
Infrared 1 640x480 @ 6Hz Y8
Infrared 1 640x360 @ 30Hz Y8
Infrared 1 480x270 @ 60Hz Y8
Infrared 1 480x270 @ 30Hz Y8
Infrared 1 480x270 @ 15Hz Y8
Infrared 1 480x270 @ 6Hz Y8
Infrared 1 256x144 @ 90Hz Y8
Depth 1280x720 @ 6Hz Z16
Depth 848x480 @ 10Hz Z16
Depth 848x480 @ 6Hz Z16
Depth 640x480 @ 30Hz Z16
Depth 640x480 @ 15Hz Z16
Depth 640x480 @ 6Hz Z16
Depth 640x360 @ 30Hz Z16
Depth 480x270 @ 60Hz Z16
Depth 480x270 @ 30Hz Z16
Depth 480x270 @ 15Hz Z16
Depth 480x270 @ 6Hz Z16
Depth 256x144 @ 90Hz Z16

Stream Profiles supported by RGB Camera
Supported modes:
stream resolution fps format
Color 1280x720 @ 15Hz RGB8
Color 1280x720 @ 15Hz Y16
Color 1280x720 @ 15Hz BGRA8
Color 1280x720 @ 15Hz RGBA8
Color 1280x720 @ 15Hz BGR8
Color 1280x720 @ 15Hz YUYV
Color 1280x720 @ 10Hz RGB8
Color 1280x720 @ 10Hz Y16
Color 1280x720 @ 10Hz BGRA8
Color 1280x720 @ 10Hz RGBA8
Color 1280x720 @ 10Hz BGR8
Color 1280x720 @ 10Hz YUYV
Color 1280x720 @ 6Hz RGB8
Color 1280x720 @ 6Hz Y16
Color 1280x720 @ 6Hz BGRA8
Color 1280x720 @ 6Hz RGBA8
Color 1280x720 @ 6Hz BGR8
Color 1280x720 @ 6Hz YUYV
Color 640x480 @ 30Hz RGB8
Color 640x480 @ 30Hz Y16
Color 640x480 @ 30Hz BGRA8
Color 640x480 @ 30Hz RGBA8
Color 640x480 @ 30Hz BGR8
Color 640x480 @ 30Hz YUYV
Color 640x480 @ 15Hz RGB8
Color 640x480 @ 15Hz Y16
Color 640x480 @ 15Hz BGRA8
Color 640x480 @ 15Hz RGBA8
Color 640x480 @ 15Hz BGR8
Color 640x480 @ 15Hz YUYV
Color 640x480 @ 6Hz RGB8
Color 640x480 @ 6Hz Y16
Color 640x480 @ 6Hz BGRA8
Color 640x480 @ 6Hz RGBA8
Color 640x480 @ 6Hz BGR8
Color 640x480 @ 6Hz YUYV
Color 424x240 @ 60Hz RGB8
Color 424x240 @ 60Hz Y16
Color 424x240 @ 60Hz BGRA8
Color 424x240 @ 60Hz RGBA8
Color 424x240 @ 60Hz BGR8
Color 424x240 @ 60Hz YUYV
Color 424x240 @ 30Hz RGB8
Color 424x240 @ 30Hz Y16
Color 424x240 @ 30Hz BGRA8
Color 424x240 @ 30Hz RGBA8
Color 424x240 @ 30Hz BGR8
Color 424x240 @ 30Hz YUYV
Color 424x240 @ 15Hz RGB8
Color 424x240 @ 15Hz Y16
Color 424x240 @ 15Hz BGRA8
Color 424x240 @ 15Hz RGBA8
Color 424x240 @ 15Hz BGR8
Color 424x240 @ 15Hz YUYV
Color 424x240 @ 6Hz RGB8
Color 424x240 @ 6Hz Y16
Color 424x240 @ 6Hz BGRA8
Color 424x240 @ 6Hz RGBA8
Color 424x240 @ 6Hz BGR8
Color 424x240 @ 6Hz YUYV

user@KayaDevice:~$

Andrey1984 · October 12, 2020, 9:26pm

@gerardom
Thank you for sharing!