Kaya gmapping issues

I recently built a Kaya and can run the joystick app without any issues (well, the logs have some Dynamixel errors, but it drives around okay). However, when I run the gmapping app, the robot barely moves and stops moving at all shortly after. I thought that maybe it was draining the battery quickly, but I can switch back to the joystick app and it works well.

Any advice on what to check? Thanks!

Can anyone share expected battery life for driving Kaya with just the joystick app? I’m wondering if I have something eating power in general or causing servo overload.

That’s interesting. You are deploying and running gmapping_distributed_kaya.app on Kaya, right? (Not gmapping_distributed_host.app)

What is your cpu usage looks like? What is the output of sudo nvpmodel -q when run on Jetson?

Thanks for the response and sorry for the delay.

Correct.

I’ve tried both 5W and 10W (MAX-N) mode. It doesn’t seem to make a difference.

Here are logs from gmapping_distributed_host, gmapping_distributed_kaya, and tegrastats during a test run:
https://filebin.ca/4rqt8kXMoKWx/gmapping_kaya.log
https://filebin.ca/4rqtEzggr8QN/gmapping_host.log
https://filebin.ca/4rqsDIknCOTa/tegrastats.log

I was driving with the joystick and it stopped within a minute. I’ve noted a couple of things in the logs.

First, when starting the app on the kaya, there are a few errors from Dynamixel about input voltage, but then it starts up and runs. This always happens, even in the joystick app.

Second, when it dies, there is a Dynamixel overload error. I’ve tested this while the kaya was sitting on a block, so the wheels can spin freely. I’ve only run this for a few minutes like this, but I don’t get the error, so presumably it’s the motor meeting resistance when actually driving? While testing on the bench, with free spinning wheels, I tested voltage across the power hub and it doesn’t vary much from 12V. I even tried giving the wheels a little resistance with my hands, but didn’t want to be too aggressive.

I have also learned that I can then restart the app and it works again (well, until it fails again). I was misinterpreting the error as a power issue, but the battery is not actually drained. It seems to be something with the Dynamixel motors, but I’m not sure how to diagnose them further.

Thanks so much for any information to help me get past this. I’m happy to run any other tests and share details back here, but I’m not really sure what to do.

I’ve maybe learned something new here. I was poking into the other kaya config versions and tried v4 instead of v2. The main difference that I see here is the max angular speed is increased. With this, I can drive around with the mapping app longer without hitting the overload error. It still happens, but after a few minutes.

It still moves much slower than in the joystick app, though.

“few errors from Dynamixel about input voltage” in the beginning of the app run can be ignored. Your diagnosis is spot on! Overload error means the motor(s) are not happy and they want to stop to protect themselves. Given they are spinning happily on a block, I wonder if there is an angle to one of the wheels that apply an undesirable torque. Are your holonomic wheels spinning on ground smoothly to any direction?

Our hardware team wonders if you are driving on carpet, which would require wheels more torque to turn. Also, “if the wheels have got hair/dirt stuck behind them, etc. Specifically the overload error is related to current, meaning that the motors are trying to output more torque than they are capable of without being damaged.”

Yes, I’m driving on carpet, as most of my house is carpet. Hrmm, I guess that’s going to really limit what I can do with this bot at home. Do you have any suggestions on how I might modify the kaya (either hardware or config) to allow me to drive in such an environment? Are there perhaps other motors that I could use? I see that there is a AX-18A, but I’m not sure if those would be compatible. The next step up in torque for the MX line is 3-4 times the cost and power usage, so those aren’t a great option.

Could you clarify a few things from the docs for the KayaBaseDriver (https://docs.nvidia.com/isaac/isaac/doc/component_api.html#isaac-kayabasedriver)?

  • What is the unit and range of max_safe_speed and max_angular_speed?
  • What is the default torque_limit? It says that the default is some constant "kDefaultTorqueLimit", but I can't find reference to what that actual value is.

Also, out of curiosity, if I were wanting to go in another direction here, it might be interesting to change the kaya design to 4 wheels. Is there any plan to expose the code for KayaBaseDriver? As it is, it’s basically a black box, so it would be difficult for me to learn from the example to design my own robot on top of Isaac SDK.

Thanks!

Let me refer to our hardware team.

In Isaac, units are meters, radians, seconds unless stated otherwise. So, they should be meters/seconds and radians/second.

kMaxTorqueLimit is a fixed value of 0x3FF, not a parameter. It comes from the specifications of dynamixel. It’s not configurable to protect the motors.

Good point, I’ll bring this up.

Thanks so much! I’ll look forward to the response from the hardware team.

I’m assuming that what you’re referring to as kMaxTorqueLimit is equivalent to setting torque_limit to the maximum of 1.0, but I was curious what the default (kDefaultTorqueLimit) is. Presumably that’s not the max by default, but maybe so? I was asking, because maybe that’s one way that I can deal with this. If, say, the default is 0.3, perhaps I could get by with 0.5 or 0.7. If it’s already at 1.0, then I guess I’m already at the limit.

Oh, I thought you asked kMaxTorqueLimit. Yes, you are right.

torque_limit (which equals kDefaultTorqueLimit = 1.0 by default) * kMaxTorqueLimit (which is hard-coded to 0x3FF)

is written to “Max Torque” value:
http://support.robotis.com/en/product/actuator/dynamixel/mx_series/mx-12w.htm#Actuator_Address_0E

Setting torque_limit to something smaller like 0.3 sounds very promising now I look at the code again.

You are troubleshooting this better than I can help you, even though you don’t see the code :D

Haha, well, I’m struggling with this nonetheless. I’d love to see the code :)

I’ve tried changing torque_limit and it doesn’t seem to have any effect at all. In fact, in the gmapping app, none of the KayaBaseDriver config values seem to have any effect (I think there’s something to this, as they do have an effect in the joystick app and the joystick app drives around much better and faster).

I did, however, learn something interesting from reading that Dynamixel documentation. It mentions that the “Max Torque” value set on power on will be used as the initial value for “Torque Limit”. As far as I can tell, from experimenting, KayaBaseDriver never changes “Torque Limit”, so I can use Dynamixel Wizard 2 to change the value of “Max Torque” (which was previously set to 100%) to a lower value. This actually seems to control the maximum torque. Setting it to around 90% seemed to help quite a lot, but still eventually failed with an overload error.

I’ll have to experiment more to understand why the gmapping app is so different from the joystick app, but I’d appreciate any insight that you guys might have here.

BTW, I still have issues on a tile floor instead of carpet. From your experience, does the kaya require a very smooth surface to perform well? I’m wondering if I still have some hardware issue or if I’m just expecting too much from this little bot.

Thanks!

Thanks for keeping us posted. We can improve Kaya driver with your findings!

I’ve certainly saw overload error before, but rarely. I’ve experienced overheat error more, which happens if I drive Kaya fast manually for a long time.

gmapping app intentioanlly sets lower limits for the speed to generate more accurate maps:

grep speed apps/kaya/gmapping_distributed_kaya.app.json
“angular_speed_max”: 0.75,
“linear_speed_max”: 0.15

Interesting that torque_limit doesn’t have any effect. Which file did you modify? Could you open Sight (located at kaya_ip:3000 by default) and check the value there please?

It isn’t so much that you can’t drive on carpet with Kaya, but you will likely see these kinds of errors more frequently, as it has to work harder. Hair & dirt can also get caught behind the wheels and bind them up. An overload error effectively translates into not having enough torque/current to accelerate the wheels. Moving at higher speeds might improve performance because you have some more momentum working for you after initial acceleration. As Oguz said though, we have seen overheat much more often than overload, which doesn’t happen too often. I would not expect to see an overload error ever happen when operating on tile flooring.

Are you able to isolate the error to one of your three motors? It may be that one of them is damaged or obstructed in some way. Maybe a loose connection could cause this type of error as well – I’m not certain, as it originates from the dynamixels themselves.

Isaac can support any of the Robotis Dynamixel motors. You should be aware that some of the use 3-wire connections (TTL), while others use 4-wire connections (RS-485) though, and that the mounting points can change between models. This could require changes to some of Kaya’s parts. You would also need to make sure to select a servo which operates in “wheel” mode (is capable of continuous rotation without end stops), and has encoder feedback for speed/position. And of course, you would need to account for any variable changes relating to encoder & speed settings, etc.