Use Interpretability to Justify Your Model
If a machine learning model performs well, it’s justification enough for using that model. Or so some people claim.
But in reality, performance is rarely enough. You need interpretability to justify the model.
Justification is a broad term and can mean anything from verifying that the model follows physical constraints, making your boss trust the model, explaining to a user why a certain prediction was made, or it can mean a formal model audit.
Interpretability is a key tool in the justification toolbox, next to things like performance evaluation, error analysis, and fairness considerations. Compared to the other interpretation goals of model improvement and data insights, justification has a strong social focus: you are justifying the model or its predictions to a person. That someone can be yourself (if you are the modeler) but often it’s someone else: your boss, a colleague, the end user, an external auditor. Ultimately it’s about making people trust the model and its predictions.
When it comes to using interpretability for justification, I have no general advice as to which interpretation approaches to prefer. Sorry. It 100% depends on the context and the person who receives the interpretation output. Just some examples:
If Facebook started showing SHAP plots next to their ads to explain why the user was shown this ad, this probably would lead to confusion.
A research community may use linear regression extensively. If you use explanations that are also in the form of weighted sums (e.g., LIME) within this community, it might be a good fit. However, using that same approach for a different research community might not work so well. Either much more communication might be needed, or it might even be a bad fit.
It might be hard to justify a complex model making high-stakes decisions (e.g. deciding over bail) by just showing the top features based on permutation feature importance. An interpretable model might better meet the requirements of justification.
You already know how to interpret SHAP and use it for studying whether the model aligns with current domain knowledge. For communicating these results to your boss, you might want to integrate other tools as well.
It’s all about understanding the context and audience for model justification.
A great paper to help you think through the people involved in ML justification is Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems. They define different roles involved in the environment of an ML model. I’ll go through them using the example of a churn prediction model:
Creators: Those who build the systems. Further subdivided into:
Owners: Those who own the model. e.g. the company you work for, for example, represented by the product owner or team lead.
Implementers: Those who build the model. e.g. the data scientists training the churn prediction model.
Operators: Those who directly use the system. For example, the model may have been built by the data science team, but the business intelligence unit operates it.
Executors: Those who work with a decision made by the system. For example, the marketing team might ask for certain churn predictions and base a marketing campaign on them.
Decision-subjects: Those affected by the decision. In our example, these are the company’s customers.
Examiners: Those who audit or investigate the system.
For the same model, you might need quite different interpretation techniques to satisfy the people with these different roles.