cuVSLAM as a medium for Visual Inertial Odometry (VIO)

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

Good Afternoon Folks,

This is essentially a cross post from the PX4 Forums, but relates to the NVIDIA forum because it attempts to use cuVSLAM (developed by NVIDIA) as a first class method for implementing Visual Inertial Odometry (VIO). So please forgive me if this is out of scope.

My current lab team is attempting to go this route for position estimation and has had limited success with this method. I performed all of the original methology, setup, and troubleshooting that the authors of the paper recommend for setting up the camera which involved calibration, making sure you had clean feature point generation and Lucas-Kanade tracking frame by frame. On the bench I did some ground tests of checking orientation through one of their many provided examples by moving ~20cm in each direction (N,S,E,W, Up, Down) and had reasonable looking transformation frames.

Once I established what I thought were good bench tests, I (with the help of Claude) wrapped up the cuVSLAM library into a ROS 2 node, followed the PX4 guide for integrating vision sensors as a form of odometry data, and tried to run some position hold flights. On Thursday night, I ran some tests in the lab and it performed wonderfully. Since then, it’s been incredibly finicky with odd takeoff interactions, weird ovular oscillatory behavior in the XZ-plane of all places (tries to lock a position, goes up, goes down and forward, more down and back, up, and further back, then back forward and higher up if that makes sense), and loss of direction.

For the drone, I’m using a stereo camera in the OAK-D S2 which is mounted about 7cm from the floor at a 45 degree angle which is seemingly the best angle of mounting for VIO. When I fire up the drone though I get very clean position tracking at (0,0,0) +/- 1/1000 of a meter, and I’ve been able to leave it sit for a couple minutes with no real position change, which is a good thing that the LOCAL_POSITION_NED doesn’t skew off. But when I go into the air, I get massive changes in position almost as if the scaling is wrong. When I looking back at the ROS bags though, there are tons of feature points. The only other possible error I thought was occuring was loss of too many frames, but I ran isolated tests with cuVSLAM the core process running, maximum power, didn’t lose any frames (thus would’ve been relying on IMU data solely), and it still had problems.

For what it’s worth, I’ll try to post an update if I ever find a solution to my problems and add the copious notes I’ve been taking on the whole process, just wanted to see if anybody else on the forum had attempted this.

Specs

Our current setup:

  • Flight Controller: OrangeCube+
  • Flight Stack: PX4-Autopilot
  • Companion Computer: Jetson Orin Nano Devkit
  • Propellers: 13 inch
  • Battery: 6S
  • PX4 Version: v1.16.0
  • Jetpack Version: 6.2
  • ROS 2 Version: Humble
  • GPS-Denied: True

Logs

Here you can find parameters, logs, and rosbags linked in the google drive if it assists anybody in the future. LMK if you don’t have viewing permission and would like it, it should be that anyone with a link can view:

2

Hello @CursedRock17!

Based on the title and content of your topic, it looks like it may receive better visibility and feedback in a different category. We took the liberty of moving it for you.

If this was an incorrect assessment, please send me a direct message.

Disclaimer: this moderation suggestion and message were generated with AI assistance.

7

Hello @CursedRock17,

Thanks for posting in the Isaac ROS forum!

From your description and linked code, I would first check the PX4 external-vision integration path rather than cuVSLAM feature tracking itself. PX4 integration is outside the main Isaac ROS support scope, but one simple issue I noticed is in the VehicleOdometry publisher:

msg.timestamp = int(timestamp_us)
msg.timestamp_sample = int(timestamp_us)

timestamp_sample should represent the camera measurement/capture time, not the ROS publish time. If PX4 receives the wrong sample time, EKF2_EV_DELAY handling can be wrong during flight. I would try fixing that first, then retest safely with the propellers removed.

I would also recommend verifying these before flight:

  • Move the vehicle forward: PX4 external-vision X should increase.
  • Move right: Y should increase.
  • Move up: NED Z should decrease.
  • Yaw should align without introducing roll/pitch offset.
  • Temporarily disable EV velocity and EV yaw fusion; fuse position only first.
  • Confirm your down45 camera mount transform exactly matches the physical OAK-D mounting.
  • Increase EV variances initially and avoid always publishing quality=100 until the estimator innovations are confirmed.

If position-only fusion is stable, you can add yaw and velocity fusion back one at a time.

Good luck with your work!

8

Hey @vchuang, I really appreciate the detailed response. It supported some notions that I wasn’t just crazy. I rolled out those timestamping changes in my ROS code, and upated my EKF2_EV_DELAY, which both had marginal success.

I’d like to leave an update before I head into the weekend. More detailed notes have been added to the drive regarding cuVSLAM in general, integration with PX4, and hardware integration.

The change that had the largest benefit transitioning from a 45* mount to a 90* mount which mounts our Stereo Camera forward (shares a transform frame with the drone). I’m still able to get lot’s of feature points on take off and in the air, I’m also able to get position locks.

As I mention somewhere in my bringup notes, I was able to run non-flying movement tests with the camera and my ROS topic (/slam/odometry) had just a couple centimeters of odometry drift and a couple degrees rotational drift, which IMO is pretty good and supports the results provided in the cuVSLAM paper. I’m also able to execute a simple external mode which takes off, hovers for 10 seconds and lands. I was able to recreate this multiple times, where both takeoff and arming failed due to inconsistent/unstable position information that I would get with the 45* mount.

So defintely big strides of progress, but the issue I currently face is “toilet bowling”, which has been documented before. So essentially oscillations in the XY-plane (a video + logs are now in the drive if it’s hard to mentally visualize). I’ve experimented with changing EKF2_EV_DELAY to aptly reflect our gap in feature frames and IMU, but it had minimal effect. This is a drone meant for indoors behavior and I’ve got no compass enabled, which is lilkely the problem, but that’s a problem for Monday. So improvement, but still struggles. Hopefully I’ll have a better update next week!

9

Thanks for sharing your progress.
This could be helpful for other forum users trying a similar Isaac ROS + PX4 setup.

10

Of course, anything that I can do to prevent anybody’s struggles in the future, I’m interested in. As promised, feedback from today’s flight was success! We were able to run position hold flights in a various lighting conditions and run autonomous takeoff, hover, and land through PX4/ROS 2 with absolutely no toilet bowling and completely stable position hold.

After asserting that the IMU was calibrated, we upped the frame rate of the stereo pair in the OAK-D up to 45 Hz, so that if we dropped frames, we could meet the alleged minimum 30 Hz that PX4 has recommended. Then we made sure to update EKF2_EV_DELAY accordingly (ours was 50ms, but probably could be tuned better).

For proof, flight footage has been added to OAK_D_CUVSLAM/videos in the drive along with our params file. Hopefully this serves usefully to those in the future, and thanks for the help.

11

Thanks for sharing the update and final test result.
It is great to hear that the setup is now working with stable position hold and autonomous takeoff/hover/land. I’ll consider this topic already resolved.

Note: the Google Drive link you provided does not appear to be publicly accessible now. If you intended to share the footage/params for others to reference, you might wanna check the sharing permissions.