Improving expert search effectiveness: Comparing ways to rank and present search results

Abstract

Expert search systems help professionals find colleagues with specific expertise. Expert search results can be presented as a list of documents with their associated experts, or as a list of candidate experts with evidence for their expertise based on documents they authored. The type of result may affect search behaviour, and therefore search task performance. Previous work has not considered such effects from the result presentation, focusing instead on how to rank experts or on ways to interact with the search results. We compare the task performance of novice users using either a document-centric interface (where each search result is a document and its associated expert) or a candidate-centric interface (where each search result is a candidate expert and their associated documents). We also compare candidate-centric and document-centric ranking functions per interface. A post-experiment survey indicated that two variables affect which interface participants preferred: the retrieval unit (candidates or documents) and the complexity (number of documents per search result). These variables affected participants' search strategy, and consequently their task performance. A quantitative analysis revealed that 1) using the candidate-centric interface results in a higher rate of correctly completed tasks, as users evaluate candidates more thoroughly, and 2) the document-centric ranking yields faster task completion. Weak evidence of a statistical interaction effect was found that prevents a straightforward combination of the most effective interface type and the most efficient ranking type. Present work resulted in a more effective, albeit less efficient, search engine for expert search at the municipality of Utrecht.