Imbalanced Datasets

Class imbalance in training sets

X_train.head()

	Time	V1	V2	V3	...	V26	V27	V28	Amount
121775	76314.0	1.505415	-0.546326	-0.518913	...	0.022535	-0.017247	-0.010005	20.00
128746	78823.0	-0.735559	0.459686	2.093094	...	-0.597624	-0.285776	-0.258780	12.99
59776	48998.0	1.217941	0.783337	-0.070014	...	-0.352676	0.041878	0.057699	1.00
282774	171138.0	2.024211	-0.586693	-2.554675	...	0.214823	-0.008147	-0.068130	10.00
268042	163035.0	-0.151161	1.067465	-0.771064	...	-0.209491	0.213933	0.233276	36.26

5 rows × 30 columns

y_train.value_counts('Class')

Class
0    0.998237
1    0.001763
Name: proportion, dtype: float64

A class imbalance typically refers to having many more examples of one class than another in one’s training set.

We’ve seen this in our fraud dataset where our class target column had many more non-fraud than fraud examples.

Real-world data is often imbalanced and can be seen in scenarios such as:

• Ad clicking data (Only around ~0.01% of ads are clicked.)
• Spam classification datasets.

Addressing class imbalance

A very important question to ask yourself:
“Why do I have a class imbalance?”

• Is it because one class is much rarer than the other?

• Is it because of my data collection methods?

But, if you answer “no” to both of these, it may be fine to just ignore the class imbalance.

A very important question to ask yourself: “Why do I have a class imbalance?”

• Is it because one class is much rarer than the other?
  • If it’s just because one is rarer than the other, you need to ask whether you care about False positives or False negatives more than the other.
    
• Is it because of my data collection methods?
  • If it’s the data collection, then that means your test and training data come from different distributions!

But, if you answer “no” to both of these, it may be fine to just ignore the class imbalance.

Handling imbalance

There are two common approaches to this:

1. Changing the training procedure

2. Changing the data (not in this course)

   • Undersampling
   • Oversampling

Can we change the model itself so that it considers the errors that are important to us?

There are two common approaches to this:

1. Changing the training procedure

2. Changing the data (not in this course)

   • Undersampling
   • Oversampling

Changing the training procedure: class_weight

`class_weight: dict or ‘balanced’, default=None`

Set the parameter C of class i to class_weight[i] * C for SVC. 
If not given, all classes are supposed to have weight one. 
The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in 
the input data as n_samples / (n_classes * np.bincount(y))

Most sklearn classifiers have a parameter called class_weight.

This allows you to specify that one class is more important than another.

For example, maybe a false negative is 10x more problematic than a false positive.

So, if you look for example, in the documentation for the SVM classifier, we see class_weight as a parameter.

from sklearn.tree import DecisionTreeClassifier

tree_default= DecisionTreeClassifier(random_state=7)
tree_default.fit(X_train,y_train);

tree_100 = DecisionTreeClassifier(random_state=7, class_weight={1:100})
tree_100.fit(X_train,y_train);

When we made our model before, we can see our confusion matrix on the left.

Now let’s rebuild our pipeline but using the class_weight argument and setting it asclass_weight={1:100}.

This is equivalent to saying “repeat every positive example 100x in the training set”, but repeating data would slow down the code, whereas this doesn’t.

Notice that we now have reduced false negatives and predicted more Fraud this time.

But, as a consequence, we are also increasing false positives.

class_weight=“balanced”

tree_balanced =DecisionTreeClassifier(random_state=7, class_weight="balanced")
tree_balanced.fit(X_train,y_train);

We can also set class_weight="balanced".

This sets the weights so that the classes are “equal”.

We have reduced false negatives but we have many more false positives now!

Are we doing better with class_weight=“balanced”?

tree_default.score(X_valid, y_valid)

0.9989968232737001

tree_balanced.score(X_valid, y_valid)

0.99879618792844

Changing the class weight will generally reduce accuracy.

The original model was trying to maximize accuracy. Now you’re telling it to do something different.

But that’s ok since accuracy isn’t the only metric that matters.

Let’s explain why this happens.

Since there are so many more negative examples than positive, false positives affect accuracy much more than false negatives.

Thus, precision matters a lot more than recall.

So, the default method trades off a lot of recall for a bit of precision.

Stratified Splits

Attribution: Scikit Learn

A similar idea of “balancing” classes can be applied to data splits.

For example, with cross-validation, there is also StratifiedKFold.

From the documentation it says

“This cross-validation object is a variation of KFold that returns stratified folds. The folds are made by preserving the percentage of samples for each class.”

In other words, if we have 10% negative examples in total, then each fold will have 10% negative examples.

Attribution: Scikit Learn

We have the same option in train_test_split with the stratify argument.

Is stratifying a good idea?

Yes and no:

• No longer a random sample.
• It can be especially useful in multi-class situations.

But in general, these are difficult questions to answer.

Well, it’s no longer a random sample, which is probably theoretically bad, but not that big of a deal and If you have many examples, it shouldn’t matter as much.

It can be especially useful in multi-class situations, say if you have one class with very few cases.

In general, these are difficult questions to answer.

Let’s apply what we learned!