AI Maintenance Assistant
An AI maintenance assistant that uses Object Detection and LLMs to answer common maintenance questions.
Overview
A comprehensive hardware and software solution designed to assist technicians during complex maintenance tasks. By integrating a Raspberry Pi with a finetuned SSD Object Detection model, this system identifies machinery components in real-time. It leverages Large Language Models (LLMs) to provide context-aware answers to maintenance questions and utilizes a projector to overlay visual guidance directly onto the equipment.
Tech Stack
The Challenge
Expert supervision isn't always available, and technicians often struggle to identify specific components in complex machinery using inefficient paper manuals.
The Solution
We developed an AR-capable AI assistant that identifies components in real-time using Computer Vision. By projecting information directly onto the equipment and allowing natural language Q&A via LLMs, technicians receive immediate, hands-free guidance.
System Architecture
The system follows a distributed client-server model. The Raspberry Pi functions as the edge device, managing the camera input, calibration, and projector output for the AR overlay. The backend consists of a Dockerized server environment that hosts the computation-heavy SSD Object Detection model and LLM logic. Python scripts facilitate real-time network communication, transmitting images for inference and returning bounding box coordinates and textual guidance to the edge device for immediate visualization.
Key Features
Object Detection
Fine-tuned SSD Object Detection on a custom-curated dataset of maintenance tasks.
Raspberry Pi
Integration of Raspberry Pi for real-time object detection and visual feedback.
Docker
Dockerized architecture ensuring consistent deployment across edge (Pi) and server environments
Python Scripts
Python scripts are used to control the Raspberry Pi and the server.
Development Timeline
Finetuning Object Detection
Fine-tuned SSD Object Detection on a custom-curated dataset of maintenance tasks.
Development of Raspberry Pi
Developed the Python backend for the Server and the Raspberry Pi.
Added more features to the Raspberry Pi
Added image checking, camera calibration, network-wide server scanner and visualization of the object detection with a beamer.
Future Roadmap Research
Researched the viability of implementing thesis findings versus training a new scene graph generation model.