Automated knowledge graph creation via multi-object detection for an AI maintenance assistant
Constructing domain-specific semantic graphs manually remains highly inefficient, while standard text-to-graph pipelines ignore spatial coordinate topologies and multi-view visual data critical for industrial automation and maintenance environments.
Designed and developed an automated extraction pipeline that parses pixel coordinates of detected objects, translates them into deterministic spatial relations (such as left of, inside of, or above), and populates formal OWL ontologies using owlready2. Additionally, implemented a multi-view spatial fusion algorithm to consolidate 2D geometric inputs into a cohesive 3D semantic model.
Demonstrated that a deterministic geometric-to-semantic translation pipeline can match or exceed the accuracy of large end-to-end multi-modal models for structured spatial reasoning in industrial contexts. The thesis and its defense received a grade of 1.4, reflecting methodological rigor and structured LLM evaluation.
The system processes annotated image data in tabular format containing bounding boxes. It calculates the center of each object and applies threshold-based logic to determine spatial relationships. It utilizes owlready2 to generate standard-compliant RDF triples. The output was evaluated against four LLMs including DeepSeek-R1, DeepSeek-V3, Llama 3.1, and Qwen 2.5, using specific metrics for correctness and completeness.
Algorithms to automatically determine above, below, left of, and inside of relations based on pixel coordinates.
Logic to merge object detection data from multiple angles into a single consistent Knowledge Graph.
Implemented variants to test graph compactness versus semantic completeness for LLM processing.
Evaluated different serialization formats, such as OWL versus Triples, to optimize LLM reasoning capabilities.
Analyzing state of the art in Semantic Web architectures and Object Detection systems.
Developing the Python-based generator and spatial relationship extraction algorithms.
Finalized quantitative LLM evaluation and submitted the thesis at TU Dresden.
Completed the formal defense of the thesis at TU Dresden, receiving a final grade of 1.4.