Message Classification with Reject Option in a customer contact system.

Abstract

CaseDriver is a multi-user multi-division customer contact system. Currently, the labelling and distribution of messages in CaseDriver is per- formed manually, i.e. a supervisor has to determine whether a message should be deleted, or which employee and stream the message has to be assigned to. This research focuses on automating this process. Automating this process allows the messages to reach the employees faster and will save time of the employee who has to perform the classification. Three big challenges arise in this research, the first being: how to pre-process the data. The second is finding the best classi- fication algorithm in the huge classification and hyper-parameter search-space. The last challenge is deciding what to do with a prediction; when the confidence of a prediction is low, or not high enough, it is unwise to automatically label the message. The first challenge is addressed with common data-mining techniques. Find- ing an optimal classification algorithm is done with Auto-Weka [14] which uses Sequential Model-based Algorithm Configuration (SMAC [10]). Finally, the last challenge is solved by providing a mean to make a trade-off decision between the proportion that is automatically classified and the accuracy of the classification. The trade-off is generated by considering four different reject-measures (which conclude to multiple types of reject rules), and selecting only the best reject- rules. Selecting the best reject-rules is done by selecting the reject-rules whose performance lies on the Pareto-front. It turns out that three (out of four) reject- measures have a representative amount of reject-rules in the Pareto-front, so all these reject-measure contribute to a better trade-off.