Hi, I’ve found a tutorial online of how to make a simulation in isaac gym and isaac sim but i dont know how to use that to then configure the jetson and wire servos for the quadruped robot. also I don’t know how to connect the gamepad to control the movement of the quadruped. Any help will be much appreciated.
these are the videos I’ve found:
YouTube. com/watch?v=02euh9dC2tw
YouTube. com/watch?v=pxPFr58gHmQ
so do I need to save or export that simulation? if so, then what after that?
kind regards
also do I need ros? if so, how do I connect all the softwares together and also the hardware to the right connectors?
HI Yousuf… the neural network shape and weights in isaacgym are what you need to export from isaac gym to your jeston nano and then run inference on this on your nano.
You need to code some communication between the nano and your servos. I have implemented the servo communication for lx16a servos on my github GitHub - sujitvasanth/lx16a-Jetson-in-C: LX16A this works on the jetson
I dont think anyone has an example of on the edge implementation of inference of the neural networks produced from isaac gym as this is the final step of the sim-2-real pathway but I will eventually have to implement this - but i’m still finalizing my neural network to utilize an mpu6050 in simulation.
It will probably need to be a 2 way process with your robot as you will need to pull some sensor data such as joint positions, contact sensors, camera or imu data from your robot to feed into your neural network for it to behave interactively.
@devs: has anyone written any on the edge neural network inference code for use on the nano?..presumably you don’t need the whole isaaacgym api, just pytorch, or ideally in c or using the opencv neural network api’s which are highly optimized
Thank you so much for your reply. I found the Github for a really cool robot. Would you know how I could control its movements with a wireless gamepad like an xbox controller [forward, back, left, right, turn left, turn right etc].
Kind regards
So which file in that GitHub would need editing? And which lines would I need to edit into what? Would I need a new simulation file, although it looks like the current code controls the inverse kinematic using camera but doesn’t include any walking. Also how would I setup the Jetson to work with the various folders given in that GitHub, for example the main folders are ‘mediapipe, platform, simulations, src’.
kind regards
I’ve now watched the video on pygame, thanks. so I’ll map the buttons to commands in the robot code. only thing is I don’t know which lines to change in the original code and I don’t know which file the lines are in, and also I don’t know which lines are responsible for walk and movement if they exist at all. also could I run some kind of simulation on my Mac to test the updated code to double check everything works before 3d printing and buying the parts. also I don’t know how to use Jetson, would I need to basically put everything [the GitHub files] on sd card and run on the Jetson as if I would on my Mac and it would just simply communicate with the servos connected to the Jetson?
Hi Yousuf… suggest you use a large language model chatbot such as chatGPT to help you analyse the code and show you the changes to make - GPT3.5 and 4 will be really good at helping you generate the code and will even show you how to get the jetson up and running. it can explain the code and the stepops necessary to get things up and running for you.
I have an example of a walking quadruped in simulation here - you could use the webots library to simulate your robot and code but this is not isaac gym so is off topic DIY Quadruped Walking Robot Dog using LX16A - YouTube