Inductive biases - a better way to think about machine learning?

part 7 of the inductive bias series

We have covered much ground in the inductive bias series: Inductive biases underpin the “learning” in machine learning. Without inductive assumptions, we can’t predict at all. Inductive biases have tangible consequences for the model's robustness, interpretability, and domain alignment. The random forest, for example, inherits many biases from the underlying decision trees and introduces new ones through randomness and ensembling. We can even frame classic statistical modeling in terms of inductive biases. The better we are at leveraging inductive biases, the better modelers we become.

After writing this mini-series about inductive biases, I came to think of them very differently. My image of inductive biases changed from “arcane academic knowledge” to a way of thinking about machine learning.

However, it is difficult to speak about inductive biases, because how they shape our models depends not only on the ML algorithm but also on the hyperparameter configuration and the data. Especially preferential inductive biases strongly interact with the data. Think of LASSO (sparse linear models): If all features strongly affect the target, all of the coefficients may still be non-zero.

Here is what I take away from writing this mini-series:

• ML algorithms aren’t the only category to think in: Hyperparameter settings can change the inductive biases so extremely that the same algorithm or implementation can produce vastly different types of models with different inductive biases.

• We can think of an ML algorithm as a “set of inductive biases” that shape the model.

• Inductive biases are key to understanding robustness, interpretability, and plausibility.

• Understanding how the ML algorithm works is required to understand the inductive biases. We can’t abstract away the algorithm and model. For example, knowing that a model is tree-based tells us that extrapolating one of the features beyond the feature range doesn’t lead to further changes in the prediction.

The inductive bias series is also the inspiration for a new book I’m working on called “Reconstructing Machine Learning”. The book first takes machine learning apart into its atomic elements and then “reconstructs” it. Sign up for updates here:

I have yet to decide what the book cover mascot will be. Maybe a monkey.

This quote by Mitchell (1980) is a good conclusion to this series:

If biases and initial knowledge are at the heart of the ability to generalize beyond observed data, then efforts to study machine learning must focus on the combined use prior knowledge, biases, and observation in guiding the learning process. It would be wise to make the biases and their use in controlling learning just as explicit as past research has made the observations and their use.

I hope you enjoyed it!