Apriltag with intel realsense d435?

Andrey1984 · November 13, 2020, 8:54am

eprecated. Set tick_period to the desired tick paramater
2020-11-13 03:48:52.530 PANIC packages/sensors/V4L2Camera.cpp@102: [V4L2Camera] Failed to open the camera device at: /dev/video0
====================================================================================================
|                            Isaac application terminated unexpectedly                             |
====================================================================================================

sudo ls /dev/video*
[sudo] password for agx: 
ls: cannot access '/dev/video*': No such file or directory

Andrey1984 · November 16, 2020, 4:49pm

 ./apps/samples/april_tags/april_tags
2020-11-16 10:08:51.702 INFO  engine/alice/tools/websight.cpp@166: Loading websight...
2020-11-16 10:08:51.703 WARN  engine/alice/backend/application_json_loader.cpp@174: This application does not have an explicit scheduler configuration. One will be autogenerated to the best of the system's abilities if possible.
2020-11-16 10:08:51.704 WARN  engine/alice/backend/redis_backend.cpp@33: 
**
**  Could not connect to redis server: Connection refused.
**
**  Metadata will not be available for this run.
**
**  You can start a redis server by running: external\redis\redis-server
**
2020-11-16 10:08:51.704 WARN  engine/alice/backend/backend.cpp@201: This application does not have an execution group configuration. One will be autogenerated to the best of the systems abilities if possible.
2020-11-16 10:08:51.704 WARN  engine/gems/scheduler/scheduler.cpp@341: No default execution groups specified. Attempting to create scheduler configuration for 8 remaining cores. This may be non optimal for the system and application.
2020-11-16 10:08:51.704 INFO  engine/gems/scheduler/scheduler.cpp@291: Scheduler execution groups are:
2020-11-16 10:08:51.704 INFO  engine/gems/scheduler/scheduler.cpp@300: __BlockerGroup__: Cores = [4, 5], Workers = No
2020-11-16 10:08:51.704 INFO  engine/gems/scheduler/scheduler.cpp@300: __WorkerGroup__: Cores = [7, 6, 1, 0, 2, 3], Workers = Yes
2020-11-16 10:08:51.715 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/perception/libapril_tags_module.so': Now has 45 components total
2020-11-16 10:08:51.725 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/sensors/libv4l2_camera_module.so': Now has 46 components total
2020-11-16 10:08:51.735 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/sight/libsight_module.so': Now has 49 components total
2020-11-16 10:08:51.753 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/viewers/libviewers_module.so': Now has 78 components total
2020-11-16 10:08:51.754 DEBUG engine/alice/application.cpp@348: Loaded 78 components: isaac::V4L2Camera, isaac::alice::BufferAllocatorReport, isaac::alice::ChannelMonitor, isaac::alice::CheckJetsonPerformanceModel, isaac::alice::CheckOperatingSystem, isaac::alice::Config, isaac::alice::ConfigBridge, isaac::alice::ConfigLoader, isaac::alice::Failsafe, isaac::alice::FailsafeHeartbeat, isaac::alice::InteractiveMarkersBridge, isaac::alice::JsonToProto, isaac::alice::LifecycleReport, isaac::alice::MessageLedger, isaac::alice::MessagePassingReport, isaac::alice::NodeStatistics, isaac::alice::Pose, isaac::alice::Pose2Comparer, isaac::alice::PoseFromFile, isaac::alice::PoseInitializer, isaac::alice::PoseMessageInjector, isaac::alice::PoseToFile, isaac::alice::PoseToMessage, isaac::alice::PoseTree, isaac::alice::PoseTreeJsonBridge, isaac::alice::PoseTreeRelink, isaac::alice::ProtoToJson, isaac::alice::PyCodelet, isaac::alice::Random, isaac::alice::Recorder, isaac::alice::RecorderBridge, isaac::alice::Replay, isaac::alice::ReplayBridge, isaac::alice::Scheduling, isaac::alice::Sight, isaac::alice::SightChannelStatus, isaac::alice::Subgraph, isaac::alice::Subprocess, isaac::alice::TcpPublisher, isaac::alice::TcpSubscriber, isaac::alice::Throttle, isaac::alice::TimeOffset, isaac::alice::TimeSynchronizer, isaac::alice::UdpPublisher, isaac::alice::UdpSubscriber, isaac::map::Map, isaac::map::ObstacleAtlas, isaac::map::OccupancyGridMapLayer, isaac::map::PolygonMapLayer, isaac::map::WaypointMapLayer, isaac::navigation::DistanceMap, isaac::navigation::NavigationMap, isaac::navigation::RangeScanModelClassic, isaac::navigation::RangeScanModelFlatloc, isaac::perception::AprilTagsDetection, isaac::sight::AliceSight, isaac::sight::SightWidget, isaac::sight::WebsightServer, isaac::viewers::BinaryMapViewer, isaac::viewers::ColorCameraViewer, isaac::viewers::DepthCameraViewer, isaac::viewers::Detections3Viewer, isaac::viewers::DetectionsViewer, isaac::viewers::FiducialsViewer, isaac::viewers::FlatscanViewer, isaac::viewers::GoalViewer, isaac::viewers::ImageKeypointViewer, isaac::viewers::LidarViewer, isaac::viewers::MosaicViewer, isaac::viewers::ObjectViewer, isaac::viewers::OccupancyMapViewer, isaac::viewers::PointCloudViewer, isaac::viewers::PoseTrailViewer, isaac::viewers::SegmentationCameraViewer, isaac::viewers::SegmentationViewer, isaac::viewers::SkeletonViewer, isaac::viewers::TensorViewer, isaac::viewers::TrajectoryListViewer, 
2020-11-16 10:08:51.756 WARN  engine/alice/application.cpp@164: The function Application::findComponentByName is deprecated. Please use `getNodeComponentOrNull` instead. Note that the new method requires a node name instead of a component name. (argument: 'websight/isaac.sight.AliceSight')
2020-11-16 10:08:51.756 INFO  engine/alice/application.cpp@255: Starting application 'april_tags' (instance UUID: 'a2e5700c-281d-11eb-8a2e-838e2f326747') ...
2020-11-16 10:08:51.756 DEBUG engine/gems/scheduler/execution_groups.cpp@476: Launching 0 pre-start job(s)
2020-11-16 10:08:51.756 DEBUG engine/gems/scheduler/execution_groups.cpp@485: Replaying 0 pre-start event(s)
2020-11-16 10:08:51.757 DEBUG engine/gems/scheduler/execution_groups.cpp@476: Launching 0 pre-start job(s)
2020-11-16 10:08:51.757 DEBUG engine/gems/scheduler/execution_groups.cpp@485: Replaying 0 pre-start event(s)
2020-11-16 10:08:51.757 INFO  engine/alice/backend/asio_backend.cpp@33: Starting ASIO service
2020-11-16 10:08:51.758 INFO  packages/sight/WebsightServer.cpp@216: Sight webserver is loaded
2020-11-16 10:08:51.758 INFO  packages/sight/WebsightServer.cpp@217: Please open Chrome Browser and navigate to http://<ip address>:3000
2020-11-16 10:08:51.759 WARN  engine/alice/backend/codelet_canister.cpp@225: Codelet 'websight/isaac.sight.AliceSight' was not added to scheduler because no tick method is specified.
2020-11-16 10:08:51.759 WARN  engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-11-16 10:08:51.759 ERROR engine/alice/components/Codelet.cpp@229: Component '_check_operating_system/isaac.alice.CheckOperatingSystem' of type 'isaac::alice::CheckOperatingSystem' reported FAILURE:

    '/etc/nv_tegra_release' does not start with '# R32 (release), REVISION: 4.3'. Please flash L4T using Jetpack 4.4.

2020-11-16 10:08:51.759 ERROR engine/alice/backend/event_manager.cpp@42: Stopping node '_check_operating_system' because it reached status 'FAILURE'
2020-11-16 10:08:51.759 WARN  engine/alice/backend/codelet_canister.cpp@225: Codelet '_check_operating_system/isaac.alice.CheckOperatingSystem' was not added to scheduler because no tick method is specified.
2020-11-16 10:08:51.759 WARN  engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-11-16 10:08:51.759 WARN  engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-11-16 10:08:51.865 PANIC packages/sensors/V4L2Camera.cpp@139: [V4L2Camera] Unsupported width, 800, driver changed to 848
====================================================================================================
|                            Isaac application terminated unexpectedly                             |
====================================================================================================
#01 engine/alice/tools/main(+0x83d78) [0x557b1bfd78]
#02 engine/alice/tools/main(+0x1559a0) [0x557b2919a0]
#03 engine/alice/tools/main(+0x155ca8) [0x557b291ca8]
#04 linux-vdso.so.1(__kernel_rt_sigreturn+0) [0x7f9a6e46c0]
#05 /lib/aarch64-linux-gnu/libc.so.6(raise+0xb0) [0x7f9a2094f8]
====================================================================================================
Minidump written to: /tmp/b2642f92-0320-40f4-4b2d7e8c-fc218d17.dmp
Aborted (core dumped)

hemals · November 16, 2020, 5:08pm

I’m gathering this was an issue getting the RealSense to be detected on launch? Can you run the sample application for the Realsense Camera correctly on your AGX Xavier? The first stack trace looks like the camera is not configured properly with UDEV rules, though your aarch64 device should be setup properly with JetPack. See here if you haven’t already. The second stack trace indicates that your target is not on the latest JetPack 4.4.X which is needed for 2020.1NX. Was this unexpected (you are running the right JetPack but still getting this message)?

Andrey1984 · November 16, 2020, 5:13pm

@hemals , thank you for following up!
the second trace is after powering off the JEtson then powering it on, also after symlinking /dev/video2 that is the device reflected at Jetson to /dev/video0 that seems the default device querried by IsaaC.
However, as you can see from the trace log above:
'/etc/nv_tegra_release' does not start with '# R32 (release), REVISION: 4.3'. Please flash L4T using Jetpack 4.4.
Moreover, the output of the command depicting the version is
cat /etc/nv_tegra_release # R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020
As you might have noticed the IsaaC app seems looking for 4.3 but the revision is 4.4
Udev rules are rewritten;

 ls /dev/video*
/dev/video2

Andrey1984 · November 16, 2020, 5:27pm

ls /dev/video* -1a | xargs -n1 v4l2-ctl --all -d
Driver Info (not using libv4l2):
   Driver name   : uvcvideo
   Card type     : Intel(R) RealSense(TM) Depth Ca
   Bus info      : usb-3610000.xhci-1
   Driver version: 4.9.140
   Capabilities  : 0x84200001
   	Video Capture
   	Streaming
   	Extended Pix Format
   	Device Capabilities
   Device Caps   : 0x04200001
   	Video Capture
   	Streaming
   	Extended Pix Format
Priority: 2
Video input : 0 (Camera 6: ok)
Format Video Capture:
   Width/Height      : 848/480
   Pixel Format      : 'YUYV'
   Field             : None
   Bytes per Line    : 1696
   Size Image        : 814080
   Colorspace        : Default
   Transfer Function : Default (maps to Rec. 709)
   YCbCr/HSV Encoding: Default (maps to ITU-R 601)
   Quantization      : Default (maps to Limited Range)
   Flags             : 
Crop Capability Video Capture:
   Bounds      : Left 0, Top 0, Width 848, Height 480
   Default     : Left 0, Top 0, Width 848, Height 480
   Pixel Aspect: 1/1
Selection: crop_default, Left 0, Top 0, Width 848, Height 480
Selection: crop_bounds, Left 0, Top 0, Width 848, Height 480
Streaming Parameters Video Capture:
   Capabilities     : timeperframe
   Frames per second: 60.000 (60/1)
   Read buffers     : 0
                    brightness 0x00980900 (int)    : min=-64 max=64 step=1 default=0 value=0
                      contrast 0x00980901 (int)    : min=0 max=100 step=1 default=50 value=50
                    saturation 0x00980902 (int)    : min=0 max=100 step=1 default=64 value=64
                           hue 0x00980903 (int)    : min=-180 max=180 step=1 default=0 value=0
white_balance_temperature_auto 0x0098090c (bool)   : default=1 value=1
                         gamma 0x00980910 (int)    : min=100 max=500 step=1 default=300 value=300
                          gain 0x00980913 (int)    : min=0 max=128 step=1 default=64 value=64
          power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=3 value=3
     white_balance_temperature 0x0098091a (int)    : min=2800 max=6500 step=10 default=4600 value=4600 flags=inactive
                     sharpness 0x0098091b (int)    : min=0 max=100 step=1 default=50 value=50
        backlight_compensation 0x0098091c (int)    : min=0 max=1 step=1 default=0 value=0
                 exposure_auto 0x009a0901 (menu)   : min=0 max=3 default=3 value=3
             exposure_absolute 0x009a0902 (int)    : min=1 max=10000 step=1 default=166 value=166 flags=inactive
        exposure_auto_priority 0x009a0903 (bool)   : default=0 value=0

and yes the camera gets detected at launch and works with realsense viewer

hemals · November 16, 2020, 5:50pm

Instead of symlinking /dev/video2, you should be able to set the parameter “device_id” to “2” in “isaac.V4L2Camera” (see //apps//samples/april_tags/april_tags.app.json). That parameter gets mapped to a device string of “/dev/video{device_id}”.

For the JetPack issue, you are correct. 2020.1NX is looking for the wrong jetson_release_version string when searching for JetPack 4.4. In //engine/alice/components/CheckOperatingSystem.hpp, you should be able to modify the default or configure the parameter with the correct value "# R32 (release), REVISION: 4.4" for now. This has been fixed already for the next Isaac SDK release which is coming soon.

Andrey1984 · November 16, 2020, 5:54pm

but it then still halts with some resolution related error; as per the trace above.
is Apriltag supported on Intel realsense dxx ?

Andrey1984 · November 16, 2020, 6:12pm

so I also change the resolution in the file from

 "rows": 448,
        "cols": 800,

also there is no alice components at the jetson

april_tags-pkg/engine/alice$ ls
tools

again trying

2020-11-16 13:18:04.574 WARN  engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-11-16 13:18:04.574 ERROR engine/alice/components/Codelet.cpp@229: Component '_check_operating_system/isaac.alice.CheckOperatingSystem' of type 'isaac::alice::CheckOperatingSystem' reported FAILURE:

    '/etc/nv_tegra_release' does not start with '# R32 (release), REVISION: 4.3'. Please flash L4T using Jetpack 4.4.

2020-11-16 13:18:04.574 ERROR engine/alice/backend/event_manager.cpp@42: Stopping node '_check_operating_system' because it reached status 'FAILURE'
2020-11-16 13:18:04.574 WARN  engine/alice/backend/codelet_canister.cpp@225: Codelet '_check_operating_system/isaac.alice.CheckOperatingSystem' was not added to scheduler because no tick method is specified.
2020-11-16 13:18:04.574 WARN  engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-11-16 13:18:04.574 WARN  engine/alice/components/Codelet.cpp@53: Function deprecated. Set tick_period to the desired tick paramater
2020-11-16 13:18:14.570 INFO  engine/alice/backend/allocator_backend.cpp@57: Optimized memory CPU allocator.
2020-11-16 13:18:14.570 INFO  engine/alice/backend/allocator_backend.cpp@66: Optimized memory CUDA allocator.

However, it sems appeared in websight on 3000 port after editing the resolution

hemals · November 17, 2020, 9:33pm

The supported resolution seems to be 848x480 for the D435 from the output of v4l2-ctl as you listed. After updating these values in the isaac.V4L2Cameracomponent configuration to rows=480 and cols=848, you are getting the JetPack version error still even after modifying CheckOperatingSystem.hpp and rebuilding on x86 host and then redeploying to your Jetson? The error should not block or affect execution and you said you are seeing the camera working in WebSight?

Andrey1984 · November 17, 2020, 10:05pm

Thank you for fololwing up!
the camera shows up in the websight after changing the resolution.
I will try editing the file CheckOperatingSystem.hpp at Host PC then redeploying to jetson.
Before I just edited files at the Jetson Robot
yes the OS version was more like warnning than terminating error, as it seems to me
Thank you very much