Problem
Industrial maintenance still leans on printed manuals and expert supervision. New technicians spend much of a repair flipping between dense documentation and the machine in front of them, which slows onboarding and invites mistakes.
Research project · TU Dresden · 2024 - present
AI Maintenance Assistant
AR repair guidance for industrial machinery: real-time object detection on an edge device, with instructions projected straight onto the machine.
- Python
- Tensorflow
- Docker
- Flask
- Raspberry Pi
- SSD Object Detection
- LLMs
- Git
- AR Projection
- Platform
- Raspberry Pi 3
- Hardware
A hardware-software system that assists technicians during machinery repairs. A Raspberry Pi edge device talks to a host server running a fine-tuned SSD object detection model for component identification, while a locally hosted LLM answers questions about the repair documentation. The results are projected directly onto the machinery as an augmented reality overlay.
Approach
Built a projection-based AR maintenance environment. The system tracks machine components with computer vision and projects step-by-step assembly instructions onto the workspace itself, while an integrated LLM answers spoken questions hands-free.
Outcome
The prototype showed that spatial AR overlays combined with an LLM assistant work as a hands-free replacement for printed manuals, cutting onboarding time and the constant context-switching between manual and machine.
Architecture
A distributed client-server design. The Raspberry Pi acts as the edge device and handles camera input, calibration, and projector output for the AR overlay. A Dockerized server hosts the computationally heavy SSD object detection model and the LLM logic. Python workflows coordinate the real-time communication: frames go to the server for inference, bounding box coordinates and textual guidance come back to the edge device for immediate visual mapping.
Highlights
Object Detection
Fine-tuned SSD object detection on a custom dataset of maintenance tasks.
Edge Hardware
Raspberry Pi integration for real-time component tracking and visual feedback.
Containerization
Dockerized architecture for consistent deployment across edge and server environments.
Control Workflows
Python workflows coordinating edge camera capture, frame transport, and server-side model inference.
Timeline
Model Optimization
Fine-tuned SSD object detection on a custom dataset of maintenance tasks.
Edge Coordination
Developed the Python backend for both the server and the edge device.
Feature Expansion
Added image validation, camera calibration, network-wide server scanning, and the projector-based AR overlay.
Roadmap and Extension
Evaluated spatial reasoning extensions: bounding box geometric heuristics versus an end-to-end scene graph generation model.