Recall & Precision versus The Bookmaker

Thumbnail Image
Powers, David Martin
Journal Title
Journal ISSN
Volume Title
International Conference on Cognitive Science
In the evaluation of models, theories, information retrieval systems, learning systems and neural networks we must deal with the ubiquitous contingency matrix of decisions versus events. In general this is manifested as the result matrix for a series of experiments aimed at predicting or labeling a series of events. The classical evaluation techniques come from information retrieval, using recall and precision as measures. These are now applied well beyond this field, but unfortunately they have fundamental flaws, are frequently abused, and can prefer substandard models. This paper proposes a well-principled evaluation technique that better takes into account the negative effect of an incorrect result and is directly quantifiable as the probability that an informed decision was made rather than a random guess.
Powers, D.M.W., 2003. Recall & Precision versus The Bookmaker. Presented at the International Conference on Cognitive Science, 13-17 July 2003, Sydney Australia (revised February 2006).