Don't "fix" your imbalanced data
Why SMOTE doesn't work for most cases.
Imagine you have data from healthy subjects and patients with a disease. Out of your 10,000 data points, only 50 have the disease (0.5%). Your goal is to classify the disease based on e.g. blood work.
A typical case of imbalanced data.
What do you do?
A commonly suggested “solution” to the imbalanced data problem is to change the data. You can either oversample the minority class (or create synthetic cases) or you can downsample the majority class. Or do both.
I’ve always found that odd, but as a student, I accepted this solution to imbalanced data.
Today I’m telling you: Don’t do it.
“Fixing” imbalanced data by sampling is (mostly) a stupid idea. This advice might be a “The Emperor’s New Clothes” type of situation1, where people doubted this advice privately but didn’t question it publicly. At least that was my impression. But it turns out that the emperor is in his underwear and that sampling approaches don’t work. At least in most cases (more about that later).
Our punching bag for this post will be SMOTE. With over 25k citations, it’s one of the most famous “fixes” for imbalanced data in machine learning.
SMOTE
SMOTE stands for Synthetic Minority Over-sampling Technique. The idea is to oversample the minority class, but instead of CTRL-C CTRL-V the data, SMOTE creates synthetic data for the minority class. SMOTE also recommends undersampling of the majority class. That’s all you need to know.
If your sample represented the population before, SMOTE ensures that it no longer does. But why would you train a machine learning model on a 50% / 50% balance if the class balance in future data is expected to be 95.5% / 0.05%?
SMOTE destroys calibration. Calibration means that e.g. a probability of 30% really means 30%. That is, if you bundle all 30% classifications of a well-calibrated classifier together, you will find that 30% of them, in expectation, matched the ground truth. Not that classifiers are necessarily calibrated by default. However, by sampling to change the distribution of the target classes, you ensure that your model is not calibrated.
Another fun fact. You don’t add new information to your data by over- or under-sampling data. If you under-sample the majority class, you actually reduce the amount of information for your model.
To Smote Or Not To SMOTE?
What about the original SMOTE paper? Did they not evaluate their approach? Did they make a mistake? Or did the machine learning community collectively agree on the wrong take-home message for imbalanced data?
It’s the latter.
Here’s what the abstract says:
This paper also shows that a combination of our method of over-sampling the minority class and under-sampling the majority class can achieve better classifier performance (in ROC space) than varying the loss ratios in Ripper or class priors in Naive Bayes. Our method of over-sampling the minority class involves creating synthetic minority class examples. Experiments are performed using C4.5, Ripper and a Naive Bayes classifier.
What a weird choice of models, right? C4.5, Ripper and Naive Bayes are all so-called weak learners. A weak learner is a machine learning model that won’t make you rich on Kaggle. Examples of strong learners, on the other hand, are xgboost, LightGBM, CNNs, transformers, and random forests.
Could it be that SMOTE actually does work, but only for some cases?
That’s what the authors of the paper To SMOTE, or not to SMOTE? found out: SMOTE works, but only for weak learners, and if you use measures such as AUC and don’t care about calibration. SMOTE didn’t improve performance for strong classifiers.
Don’t use SMOTE if you use state-of-the-art classifiers. Don’t use SMOTE if you care about calibration.
So the original SMOTE paper is actually fine. The authors are not scammers who sell magic clothes to the ML community. It’s just that we as a community have to realize that the scope of model classes and situations where SMOTE works is narrow.
What do you do instead when you have imbalanced data? Well, that’s something for another newsletter issue.
But for now, a good default strategy for imbalanced classification tasks: Do Nothing!
The Emperor’s New Clothes is a fairytale where a scammer sells “clothes” to the emperor that he claims are only visible to clever people. Well, the clothes don’t exist, but everyone goes along with it because no one wants to be seen as dumb.
Agree on the whole article except- do nothing is not the right advice. You use cost sensitive learning or class weighted cost function
I appreciate your perspective on the default strategy for imbalanced classification tasks, but I'd like to share a real-life scenario where "doing nothing" was not a viable option.
In our case, we were dealing with a dataset comprising billions of training examples, making it practically impossible to train a model with all the available data. Additionally, we faced the challenge of having very few positive examples, and achieving a high recall was crucial for our application.
If we had followed the "Do Nothing" approach and sampled the data uniformly, it would have resulted in the loss of most of the positive examples from the training data. This would have severely impacted the model's ability to learn from these crucial instances.
To overcome this challenge, we opted for a different approach. We decided to use all the available positive examples and then sample the negative ones. To ensure that the model didn't become overly biased towards the positive class, we assigned larger weights to the sampled negative examples. This approach allowed us to strike a balance and create a model that was sensitive to both positive and negative instances.
I acknowledge that our situation was unique due to the exceptionally large dataset we had to work with. However, it highlights the importance of considering the specific characteristics of the dataset and problem at hand. In some cases, "doing nothing" may not be an effective strategy, as it can lead to a low recall model, especially when you need to sample the data and positive examples are scarce.