Evaluation

It is important to define a method for evaluating suggestions in oder to be able to evaluate and compare different suggester-engines. One promising approach is outlined below:

First we need to randomly select a test set of entities. Then for every entity we divide the existing property information (snaks) into two parts. One part is taken as input for the suggester-engine that is being tested, while the second part is matched against the output predictions/suggestions. If a suggestion can be matched, it is obviously correct. However the case is more complicated, if there is no match, because a suggestion could still be good/correct even though the proposed property does not already exist for the entity on hand (open world assumption). In these cases outside information could be taken into account and/or manual evalution could be done for a sample of the suggestions (unfortunately this would be more or less subjective). In addition user feedback could also be used systematically for evaluating as well as improving suggestions during later stages of the project.