Bachelor Thesis
Automated knowledge graph creation via multi-object detection for an AI maintenance assistant
Overview
This research project addresses the gap between visual object detection and structured knowledge representation. I developed a system that automatically converts bounding box data from images of industrial machines into formal Knowledge Graphs (RDF/OWL). These graphs serve as a structured knowledge base for Large Language Models (LLMs), enabling them to answer spatial and maintenance-related questions in an industrial context without hallucinations.
Tech Stack
The Challenge
Creating Knowledge Graphs manually is time-intensive and error-prone. While text-to-graph methods exist, there is a significant lack of automated approaches that transform visual object detection data into formal, logic-based knowledge structures that LLMs can process efficiently for industrial maintenance tasks.
The Solution
I designed and implemented a modular Python based pipeline (`OntologyGenerator.py`) that ingests CSV output from Multi-Object Detection models. Using geometric algorithms, it automatically derives semantic spatial relations (e.g., 'left_of', 'above', 'inside_of') and instantiates them into an OWL ontology. The system supports multi-camera fusion to minimize the uncertainty and problems of object detection.
System Architecture
The system processes annotated image data (CSV) containing bounding boxes. It calculates the center of each object and applies threshold-based logic to determine spatial relationships. It utilizes `owlready2` to generate standard-compliant RDF triples. The output was evaluated against four LLMs (DeepSeek-R1, DeepSeek-V3, Llama 3.1, Qwen 2.5) using specific metrics for correctness and completeness.
Key Features
Geometric Relation Extraction
Algorithms to automatically determine 'above', 'below', 'left_to', and 'inside_of' relations based on pixel coordinates.
Multi-Camera Fusion
Logic to merge object detection data from multiple angles into a single consistent Knowledge Graph.
Implicit vs. Explicit Modeling
Implemented variants to test graph compactness versus semantic completeness for LLM processing.
LLM Integration
Evaluated different serialization formats (OWL vs. Triples) to optimize LLM reasoning capabilities.
Project Gallery
Development Timeline
Research & Concept
Analyzing state of the art in Semantic Web and Object Detection.
Implementation
Developing the Python generator and relation logic algorithms.
Submission
Final evaluation of LLM performance and thesis submission at TU Dresden.
Defense
Final defense of thesis at TU Dresden. Received a grade of 1.4