Love this, I am a big fan of permutation importance.
For the first scenario, since the model was overfit, is SHAP more useful with a validation set? This way at least you would see the clear failure of the model for certain examples. Is it worthwhile looking at shap for correct versus incorrect predictions?
(Obviously this doesn’t hold if there is some sort of distribution shift in production, etc.)
I.e. throwing an entire feature set at a fit, then pull out the noisy features? I have an extremely noisy dataset where I've investigated the features and found that relationships of those with highest spearmanR/mutual info with the response can shift drastically, so I've tried to dissect those dependency shifts and find contrast in other features during positive/negative correlation instances. So in essence, there are certainly interaction effects, but the data is so complex that it's not easily investigated. Thanks!
Nice post. I am quite critical with shapley values because I know already all the pitfalls of feature importance and similar but other things seem to be problematic for shapley values.
I tried out information gain for feature selection, but even there are some pitfalls in case of too many factors. In fact the whole topic is still a bit blurry for me and quite hard to handle.
Love this, I am a big fan of permutation importance.
For the first scenario, since the model was overfit, is SHAP more useful with a validation set? This way at least you would see the clear failure of the model for certain examples. Is it worthwhile looking at shap for correct versus incorrect predictions?
(Obviously this doesn’t hold if there is some sort of distribution shift in production, etc.)
Haha 🙈 I totally get your rant. The two quotes speak for themselves...
What are your thoughts on this process for utilizing SHAP for feature selection on a saturated model?
https://towardsdatascience.com/your-features-are-important-it-doesnt-mean-they-are-good-ff468ae2e3d4
I.e. throwing an entire feature set at a fit, then pull out the noisy features? I have an extremely noisy dataset where I've investigated the features and found that relationships of those with highest spearmanR/mutual info with the response can shift drastically, so I've tried to dissect those dependency shifts and find contrast in other features during positive/negative correlation instances. So in essence, there are certainly interaction effects, but the data is so complex that it's not easily investigated. Thanks!
Nice post. I am quite critical with shapley values because I know already all the pitfalls of feature importance and similar but other things seem to be problematic for shapley values.
I tried out information gain for feature selection, but even there are some pitfalls in case of too many factors. In fact the whole topic is still a bit blurry for me and quite hard to handle.
Great post highlithing the need of using and comparing several XAI methods for data analysis.
Here is a blog post with my additional thoughs on this topic : https://medium.com/@jb.excoffier/pitfalls-when-computing-feature-importances-3f5b0e2c198c