**1. Concepts**

**1.1 Confusion matrix**

An NxN table that summarizes how successful a classification model's predictions were.

It shows the correlation between the label and the model's classification.

N represents the number of classes.

In a confusion matrix, **o****ne axis is the predicted ****label**, and **one axis is the actual label**.

An example when N=2

actual tumors: 19

**1.2. True vs. False and Positive vs. Negative**

Considering the example is based on The Boy Who Cried Wolf story.

Let's make the following definitions:

- "Wolf" is a positive class.
- "No wolf" is a negative class.

**true positive (TP)**is an outcome where the model correctly predicts the positive class.

**true negative (TN)**is an outcome where the model correctly predicts the negative class.

**false positive (FP)**is an outcome where the model incorrectly predicts the positive class.

**false negative (FN)**is an outcome where the model incorrectly predicts the negative class.

**True/False is predicted**

**Positive/Negative is ground-truth**

**1.3 Accuracy**

**Accuracy doesn't tell the full story when you're working with a class-imbalanced data set, where there is a significant disparity between the number of positive and negative labels.**

**1.3 Precision and Recall**

**Precision: What proportion of positive identifications was actually correct?**

**In all positive cases how many positive cases are correctly predicted?**

when it predicts a tumor is malignant, it is correct 50% of the time.

**Recall: What proportion of actual positives was identified correctly?**

when it correctly identifies 11% of all malignant tumors.

**improving precision typically reduces recall and vice versa.**

**Consider Classifying email messages as spam or not spam when changing threshold.**

**1.4 ROC curve**

- True Positive Rate
- False Positive Rate

**evaluate a classification model many times with different classification thresholds**, but this would be inefficient. Fortunately, there's an efficient, sorting-based algorithm that can provide this information for us, called AUC.

## How to use the ROC Curve?

We can generally use ROC curves to decide on a threshold value. The choice of threshold value will also depend on how the classifier is intended to be used. So, if the above curve was for a cancer prediction application, you want to capture the maximum number of positives (i.e., have a high TPR) and you might choose a low value of threshold like 0.16 even when the FPR is pretty high here.

This is because you really don’t want to predict “no cancer” for a person who actually has cancer. In this example, the cost of a false negative is pretty high. You are OK even if a person who doesn’t have cancer tests positive because the cost of false positive is lower than that of a false negative. This is actually what a lot of clinicians and hospitals do for such vital tests and also why a lot of clinicians do the same test for a second time if a person tests positive. (Can you think why doing so helps? Hint: Bayes Rule).

Otherwise, in a case like the criminal classifier from the previous example, we don’t want a high FPR as one of the tenets of the justice system is that we don’t want to capture any innocent people. So, in this case, we might choose our threshold as 0.82, which gives us a good recall or TPR of 0.6. That is, we can capture 60 per cent of criminals.

**1.5 AUC (Area under the ROC Curve)**

- It is threshold invariant i.e. the value of the metric doesn’t depend on a chosen threshold.
- It is scale-invariant i.e. It measures how well predictions are ranked, rather than their absolute values.

In this case the AUC is 0.7, it means there is a 70% chance that the model will be able to distinguish between positive class and negative class.

**When two distributions completely overlap. This is the worst case.**

In this case the AUC is approximately 0.5, the model has no discrimination capacity to distinguish between positive class and negative class.

**The AUC=0**

**In a multi-class model,**we can plot N number of AUC ROC Curves for N number classes using the One vs ALL methodology. So for example, If you have three classes named X, Y, and Z, you will have one ROC for X classified against Y and Z, another ROC for Y classified against X and Z, and the third one of Z classified against Y and X.

## No comments:

## Post a Comment