AI-driven Interactive Lab Assistant w/ OpenCV & ChatGPT

Robotics & Edge Computing Jetson & Embedded Systems Jetson Projects
, , , , ,
1

Hi everyone,

Recently, I published my AI-driven lab assistant project with Jetson Nano on Hackster.

After scrutinizing recent research papers on distance education, I noticed few methods focus on applying object detection with artificial intelligence to provide auto-generated insightful suggestions and guidance, especially for lab-related studies. Since lab-related studies rely on experiments with different types of lab equipment, I decided to build an AI-driven interactive lab assistant counseling students regarding lab equipment, their proper utilization, and history.

Since lab equipment can be opaque, transparent, in various shapes, and change volume depending on the research subject, it is arduous to construct a valid and notable data set for object detection models. Thus, I decided to employ OpenCV modification features to convert images to generate meaningful data samples for diverse lab equipment regardless of their complex structures. Depending on the given lab equipment, I utilized canny edge detection, erosion and dilation morphological operators, Gaussian blur, and color space conversions so as to generate unique and meaningful samples.

After completing my data set, I built my object detection model with Edge Impulse to detect diverse lab equipment from my laboratory. Then, I deployed and uploaded the model on Jetson Nano as a Linux (AARCH64) application (.eim). Therefore, this lab assistant is capable of detecting lab equipment by running the model independently without any additional procedures or latency.

After running the object detection model successfully, I decided to employ the OpenAI API to obtain ChatGPT-powered insightful suggestions and guidance as auto-generated lessons so as to inform the user (student) of the detected lab equipment thoroughly. To enable ChatGPT to generate effective lessons not dependent on a single approach, I provided different inquiries (topics) related to the given lab equipment automatically via the user interface.

If interested, you can inspect the project tutorial, including code files, STL files, and thorough instructions.

1
11920Ă—1440 218 KB

5 Likes
2

The “AI-driven Interactive Lab Assistant with OpenCV & ChatGPT” combines cutting-edge artificial intelligence technologies to revolutionize laboratory operations. By integrating OpenCV, a powerful computer vision library, with ChatGPT, a state-of-the-art natural language processing model, this assistant offers a seamless and intelligent interface for laboratory tasks.

OpenCV empowers the assistant with advanced visual capabilities, allowing it to process images and videos in real-time. With OpenCV, the assistant can perform tasks such as object detection, image segmentation, and gesture recognition. This functionality enables it to monitor experiments, analyze samples, and interact with laboratory equipment efficiently.

ChatGPT, developed by OpenAI, serves as the conversational interface of the assistant. Utilizing its natural language understanding capabilities, ChatGPT enables users to communicate with the assistant through text or speech. Users can ask questions, seek guidance on experimental procedures, or request assistance with data analysis tasks. ChatGPT’s ability to generate human-like responses ensures a smooth and intuitive interaction experience.

Together, these technologies create a versatile and responsive assistant that enhances laboratory workflows and facilitates scientific research. Whether it’s providing real-time feedback, answering inquiries, or guiding users through complex procedures, the AI-driven Interactive Lab Assistant with OpenCV & ChatGPT redefines the way laboratories operate, fostering efficiency, collaboration, and innovation within the scientific community.

1 Like
3

Love your cyberdeck @lionelmoen4 ! Well done, cool project! I think this is very useful to have lab assistants like this, and hopefully to improve automation. I understand that on Jetson Nano 4GB you would use ChatGPT, but for others on Orin Nano 8GB (or larger) you can easily swap out ChatGPT for a locally-run LLM by running an OpenAI-compatible server on your Jetson (packages like llama.cpp, text-generation-webui, and MLC will run OpenAI servers for you)