Precision and recall
Contents
About
I always forget which is which for precision and recall in Machine Learning / estimation.
Precision versus Recall
Imagine a machine learning algorithm to detect cat photos.
- Precision -> of the number you thought were positive.... what fraction are actually positive. (think: what fraction of darts hit targets)
True Positives / (True Positives + False Positives)
- Recall -> what fraction of cats were predicted as cats? (think: how many targets were hit)
True Positives / (True Positives + False Negatives)
If you just blindly say everything is a cat we get 100% recall, but really low precision (a tonne of non-cat photos we said were cat). Machine learning papers often show a "precision recall curve" which shows that as you adjust a threshold to guess more "cats".... if we only pick 99.9% confidence of a cat, it's probably almost always right (100% recision), but terrible recall. But as we go down to 80% sure of a cat, we get better recall (we capture a higher fraction of cats), but the trade off is that we're probably selecting more and more non-cats.
Said Another Way
Your boss asks:
- What percent of your predictions were correct? "accuracy"
- What percent of the positive cases did you catch? "recall"
- What percent of positive predictions were correct? "precision"
Table of Confusion
A table of confusion (sometimes also called a confusion matrix), is a table with two rows and two columns that reports the number of false positives, false negatives, true positives, and true negatives.
| True positives (cats correctly classified as cats) |
False negatives (cats incorrectly classified as dogs) |
| False positives (non-cat incorrectly classified as cats) |
True negatives (non-cat, correctly classified as non-cats) |
Accuracy
Accuracy: The fraction of predictions we got right...
