Hi, I am using Jetson Nano 4gb B01 module in my project. So we have developed a functionality where the device captures data from the sensors and transmits data to the cloud using MQTT protocol. The cloud sends back MQTT responses and device takes action. Now, we want to implement LLM inside device. I am able to run tinyllama using ollama. So for a use case where we receive prompt from the app. The prompt will be sent to device and device will direct this to the LLM inside device. How to integrate the LLM inside device with MQTT so that the interactions will happen in an autonomous way?
Would it be possible for you to describe a use case scenario?
Right now I am thinking of your MQTT client being subscribed to a topic that is published by the user app, where it sends the request you need. But I would love to have a better understanding of what you want to achieve so I can provide a better solution tailored for your system.
regards,
Andrew
Embedded Software Engineer at ProventusNova
So there will be few mqtt packets. Those mqtt packets we need to send to LLM and depending on some function logic, the llm should analyze packet parameters and give some decisions, How can we do this?
In your case I would divide the problem in 2 smaller parts.
Challenge #1. MQTT communication
Challenge #2. LLM media pipeline
In challenge #1 I would focus on creating the client/server infrastructure for sending and receiving messages through MQTT. I would do this with Python, since it is usually the simplest to implement.
As per challenge #2, you need to work on feeding the proper data into your LLM through a dynamic way.
Now, a few more questions.
At this point, what parts of the solution do you currently have working ?
Is the LLM fed images captured by the NVIDIA Jetson?
Are you able to share a bit of your current source code to see where you are at?
regards,
Andrew
Embedded Software Engineer at ProventusNova
Have you take a look at Jetson Platform Services? NVIDIA already has services that allow you to connect multiple systems through a microservices architecture. You can have a MQTT microservice interacting with your server, and forwarding packets to another microservice running an LLM. And you can select an LLM from either TAO, NVIDIA examples, or doing transfer learning for your custom functionalities.
RidgeRun has already created demos and supported customers using this architecture, integrating custom LLMs with other microservices to create complex systems. You can take a look at our documentation here:
Do it from scratch is not a good recommendation, it is preferable to leverage what NVIDIA already provides, so you can speed up development. Feel free to contact us at support@ridgerun.com if you need help with that!