BAYONET – A system for detecting threats using an ontology-based Bayesian network and evidence tags

Abstract

This thesis presents the BAYONET system, a system that interprets the contents of images by reasoning over their image annotations. For the use case provided by Capgemini, where this research was conducted, the system was tasked to assign threat levels to images. To achieve this, an ontology was created and used to infer knowledge from image annotations, which were collected with Clarifai. The inferences were used as evidence in a Bayesian network that was derived from the ontology, which enabled predictions to be made about the contents of an image. A problem with such systems is that they are often not re-usable for problem scenarios in multiple domains because their implementation is tied to a specific domain. This thesis aimed to solve this problem by using structural rather than domain-dependent relationships. These relationships were used in the Bayesian network to derive its structure and to translate evidence to node probabilities. For the latter, two distinct methods were created: Method 1, which translates evidence in a certain environment, and Method 2, which translates evidence in an uncertain environment. The results show that a system with a domain-independent implementation is usable for domain-dependent problem scenarios. Performance sometimes lacked, especially in uncertain environments. This was not because of domain-independency, but rather an unbalanced annotation distribution (Clarifai), missing annotations for target objects (Clarifai), and the difficulties of incorporating human perspective (ontology). Further research on the quality of evidence collection is needed to increase the performance of the translation of evidence to node probabilities.