Roles of Supervised Machine Learning in Science
Can ML models be scientific models?
While I'm finishing the book on conformal prediction, the next book is already in the works: "Supervised Learning for Science" (working title). It's the first book I'm writing not alone, but together with my former colleague and friend Timo Freiesleben.
We are writing about (supervised) machine learning as a new "paradigm" in science. While drafting the book, we discussed the different roles that machine learning models currently play in (applied) research:
Study object
Scientific tool
Scientific model đ„
Intervention
Letâs go through these âlevelsâ one by one.
1) ML Models as Study Objects
The role of ML models as study objects is the currently most common role in science. Itâs obvious, at least if you are an ML researcher. Itâs the papers that draw their conclusions from comparing the performance of multiple ML models. For more application-centric papers this comparison includes the current state-of-the-art in the field which can also be more mechanistic models (e.g. differential equations) or more traditional statistical modeling (e.g. GLMs with in-sample goodness-of-fit evaluation).
The scientific question is usually in the form of: âHow well does a predictive model perform at a certain task?â
In most methodological ML papers, this is an agreed way to progress science: suggest some a new ML algorithm and compare the performance across classic datasets (the infamous Boston housing, Iris, and MNIST).
This way of research has also spilled over to applied research. You will find this style of papers in domain-specific journals as well, with the difference that the question becomes more application-centric. For example:
Can we use CNNs to predict tornadoes?
How well can our architecture predict protein structures from the amino sequence?
Seeing ML models as study objects is a conservative way to use ML in science: many others have already studied ML models the same way. And it keeps even the application-centric papers in the language of machine learning research.
Seeing ML models as study objects is convenient: it avoids, or at least reduces, inconvenient questions about causality, interpretability, and uncertainty. Adding those to your model is merely bonus points but not a publication requirement.
Seeing ML models as study objects is also limiting: it blocks, or at least limits, the researcher's ability to make statements about the underlying phenomenon (tornadoes, protein synthesis, âŠ).
2) ML Models as Scientific Tools
Using ML models as scientific tools goes a step further than merely seeing them as study objects: it requires trusting the predictions enough to use them in your research.
An example: Use word embeddings on articles and study the embeddings to better understand satire articles.
In cases like this one, machine learning becomes an accepted part of the scientific toolkit, just like surveys, visualizations, and data imputation. The researchers use ML as a tool and then âthrow awayâ or rather ignore the model and only work with the model outputs. The model isnât at the center of the scientific question, but itâs a necessary part of the analytic pipeline.
The more the scientific question is embedded within a (supervised) ML model, the more we move to level 3.
3) ML Models as Scientific Models
ML models in this role become a central part of how you answer a scientific question.
Some examples of questions:
What these questions have in common is that they are translated into a question of prediction (predicting yield, predicting occurrence). But what distinguishes the use of ML as a scientific model from that of a mere object of study is the dual use of the model:
Make predictions with the model
Learn through the model about the studied phenomenon
The predictive model encodes your scientific question and allows you to get answers about the phenomenon that you are studying. But this doesnât come for free: You have to put in more assumptions and domain knowledge and do more post-hoc work with the model.
There is an immense amount of (implicit) domain knowledge going into the model:
Which data to use
How concepts are operationalized as variables
Selecting features to use (e.g. deciding on confounders)
How the model is evaluated
After training the model, you donât suddenly get out âinsightsâ but you have to work with the model and put in assumptions:
Interpret the model: For example, what were the most important features?
Explain predictions: How can a prediction be attributed to the feature values?
Study the residuals: For which data did the model perform poorly?
Quantifying the uncertainty of the model: How certain are individual predictions?
Understand causes and effects: May we interpret a feature effect causally?
Prediction and explanation are often presented as a dichotomy. I disagree. How could I agree to such a dichotomy after writing Interpretable Machine Learning?
Superficially, there are lots of shortcomings that render supervised machine learning useless as a scientific model: lack of interpretability, a conflict between causality and predictive performance, lack of domain knowledge, and lack of uncertainty quantification.
But the ML landscape is changing. We have tools such as model-agnostic interpretation, conformal prediction for uncertainty quantification, a lot of insights about statistical learning, and much more. But these puzzle pieces are scattered all over the place. Timo and I are very excited to put it all together in our new book.
The tools are only one building block in the acceptance of supervised machine learning as (part of) scientific models. The other is cultural and philosophical. We think thereâs a strong case that can be made in favor of a supervised learning mindset: The better a model predicts reality (given constraints, of course), the better it âexplainsâ reality. Or if we inverse the thought: Why should we explain the world with models that make miserable predictions? We discuss justifications in favor of supervised learning in our new book as well.
4) ML Models as Intervention
The last role is a bit different from the other three and wonât be the focus of the book, but itâs an important point to mention for completeness.
Machine learning models arenât just an artifact in the world, but, once deployed, they might shape their environments. The deployment is an intervention in the real world. Such an intervention creates an opportunity and needs to study its impact.
Thatâs a wrap for today.
What are your thoughts on the role of machine learning in science? Feel free to leave a comment below this post.
Great take!
Here is a question: do you think ML models will ever replace scientific theory, especially those areas of science where the human brain may be inherently limited to explore? In other words, instead of descriptive equations showing how dark energy works, could ML take its place without the need for us to explain how dark energy exactly works, especially if we can make accurate predictions using that ML model about dark energy?
Predictive models teach us a lot about the system we are observing. My fair value bitcoin model does a poor job of predicting price, but it teaches us a lot about the human emotions behind market prices.