Roles of Supervised Machine Learning in Science

Can ML models be scientific models?

While I'm finishing the book on conformal prediction, the next book is already in the works: "Supervised Learning for Science" (working title). It's the first book I'm writing not alone, but together with my former colleague and friend Timo Freiesleben.

We are writing about (supervised) machine learning as a new "paradigm" in science. While drafting the book, we discussed the different roles that machine learning models currently play in (applied) research:

1. Study object

2. Scientific tool

3. Scientific model 🔥

4. Intervention

Let’s go through these “levels” one by one.

1) ML Models as Study Objects

The role of ML models as study objects is the currently most common role in science. It’s obvious, at least if you are an ML researcher. It’s the papers that draw their conclusions from comparing the performance of multiple ML models. For more application-centric papers this comparison includes the current state-of-the-art in the field which can also be more mechanistic models (e.g. differential equations) or more traditional statistical modeling (e.g. GLMs with in-sample goodness-of-fit evaluation).

The scientific question is usually in the form of: “How well does a predictive model perform at a certain task?”

In most methodological ML papers, this is an agreed way to progress science: suggest some a new ML algorithm and compare the performance across classic datasets (the infamous Boston housing, Iris, and MNIST).

This way of research has also spilled over to applied research. You will find this style of papers in domain-specific journals as well, with the difference that the question becomes more application-centric. For example:

• Can we use CNNs to predict tornadoes?

• How well can our architecture predict protein structures from the amino sequence?

Seeing ML models as study objects is a conservative way to use ML in science: many others have already studied ML models the same way. And it keeps even the application-centric papers in the language of machine learning research.

Seeing ML models as study objects is convenient: it avoids, or at least reduces, inconvenient questions about causality, interpretability, and uncertainty. Adding those to your model is merely bonus points but not a publication requirement.

Seeing ML models as study objects is also limiting: it blocks, or at least limits, the researcher's ability to make statements about the underlying phenomenon (tornadoes, protein synthesis, …).

2) ML Models as Scientific Tools

Using ML models as scientific tools goes a step further than merely seeing them as study objects: it requires trusting the predictions enough to use them in your research.

An example: Use word embeddings on articles and study the embeddings to better understand satire articles.

In cases like this one, machine learning becomes an accepted part of the scientific toolkit, just like surveys, visualizations, and data imputation. The researchers use ML as a tool and then “throw away” or rather ignore the model and only work with the model outputs. The model isn’t at the center of the scientific question, but it’s a necessary part of the analytic pipeline.

The more the scientific question is embedded within a (supervised) ML model, the more we move to level 3.

3) ML Models as Scientific Models

ML models in this role become a central part of how you answer a scientific question.

Some examples of questions:

• “How does fertilizer use affect almond yields?”

• “How are elephants distributed in Africa?”

What these questions have in common is that they are translated into a question of prediction (predicting yield, predicting occurrence). But what distinguishes the use of ML as a scientific model from that of a mere object of study is the dual use of the model:

• Make predictions with the model

• Learn through the model about the studied phenomenon

The world in a bottle. Made with Stable Diffusion.

The predictive model encodes your scientific question and allows you to get answers about the phenomenon that you are studying. But this doesn’t come for free: You have to put in more assumptions and domain knowledge and do more post-hoc work with the model.

There is an immense amount of (implicit) domain knowledge going into the model:

• Which data to use

• How concepts are operationalized as variables

• Selecting features to use (e.g. deciding on confounders)

• How the model is evaluated

After training the model, you don’t suddenly get out “insights” but you have to work with the model and put in assumptions:

• Interpret the model: For example, what were the most important features?

• Explain predictions: How can a prediction be attributed to the feature values?

• Study the residuals: For which data did the model perform poorly?

• Quantifying the uncertainty of the model: How certain are individual predictions?

• Understand causes and effects: May we interpret a feature effect causally?

Prediction and explanation are often presented as a dichotomy. I disagree. How could I agree to such a dichotomy after writing Interpretable Machine Learning?

Superficially, there are lots of shortcomings that render supervised machine learning useless as a scientific model: lack of interpretability, a conflict between causality and predictive performance, lack of domain knowledge, and lack of uncertainty quantification.

But the ML landscape is changing. We have tools such as model-agnostic interpretation, conformal prediction for uncertainty quantification, a lot of insights about statistical learning, and much more. But these puzzle pieces are scattered all over the place. Timo and I are very excited to put it all together in our new book.

The tools are only one building block in the acceptance of supervised machine learning as (part of) scientific models. The other is cultural and philosophical. We think there’s a strong case that can be made in favor of a supervised learning mindset: The better a model predicts reality (given constraints, of course), the better it “explains” reality. Or if we inverse the thought: Why should we explain the world with models that make miserable predictions? We discuss justifications in favor of supervised learning in our new book as well.

4) ML Models as Intervention

The last role is a bit different from the other three and won’t be the focus of the book, but it’s an important point to mention for completeness.

Machine learning models aren’t just an artifact in the world, but, once deployed, they might shape their environments. The deployment is an intervention in the real world. Such an intervention creates an opportunity and needs to study its impact.

That’s a wrap for today.

What are your thoughts on the role of machine learning in science? Feel free to leave a comment below this post.