From bare-bones to holistic machine learning

Machine learning lives on a scale from bare-bones to holistic.

Think of bare-bones machine learning as a casting mold to produce puzzle pieces:

This casting mold represents all the choices that have been made for the modeling process: which features to use, what the target is, what type of task, how the model will be used, how it will be evaluated, and so on.

---

Tomorrow is the last chance to get the SHAP book with an early bird coupon. Get your copy today!

---

Training the model is like molding a puzzle piece using this cast.

You decide whether you want to interpret the yellow color as liquid gold or as slime mold. Both work for this analogy.

How well can the gold/slime fit the casting mold? The equivalent in the case of machine learning is the evaluation metric that tells us how well the model “fits the cast”.

Bare-bones machine learning is the part of modeling where the task is already perfectly defined and it’s mostly about model comparison and model tuning.

Bare-bones ML invites the modeler to put on blindfolds, chug an energy drink and minimize L(Y, f(X)). No questions asked.

There’s a platform called numerai where you can compete in machine learning competitions. Each row in the data corresponds to a stock at a given time, but you don’t get to know what the features stand for, they are just named “feature1”, “feature2”, … And as in most machine learning competitions, everything has been set for the player: the evaluation metric, which data to use, etc. This competition is as bare-bones as it gets in terms of modeling. Other examples of bare-bones machine learning are educational toy examples which you find in introductory courses, benchmarks on platforms such as OpenML, and automated machine learning (AutoML).

While bare-bones territory may feel like a playground, it’s not kindergarten: If you have ever competed in a Kaggle challenge with ambitions to climb the leaderboard or even win, you know the struggle.

That’s the strength and the weakness of supervised machine learning: It’s the modeler’s goal to quickly define the task at hand so that it becomes a bare-bones ML task and you can just let the algorithms loose. If the cast is set correctly, machine learning excels at producing great models. But it also distances the modeler from the data and the larger puzzle.

Holistic Machine Learning

If you are an experienced modeler, you know that life rarely gives us bare-bone ML tasks. It‘s tedious to translate a problem into a prediction task and get the right data.

The reason why we molded a puzzle piece in the first place was because it becomes part of a bigger puzzle. While bare-bones ML is the fitting process of creating the puzzle piece, holistic machine learning is also about designing the cast so that machine learning produces a puzzle piece that fits:

Your model needs to fit with your business and systems in a technical sense. But beyond that, you often need many other puzzle pieces to connect your model and get a full picture. Holistic ML is also about producing puzzle pieces to enhance the model:

• You might need uncertainty quantification so that predictions can be used with confidence.

• You might need interpretability so that you can convince another department to actually use your model.

• You need a plan for what happens with distribution shifts.

My writing is about collecting and teaching puzzle pieces to enable holistic machine learning. Interpretable machine learning is such a puzzle piece, which is especially easy to dock onto your model when you use model-agnostic interpretability tools such as SHAP values. Conformal prediction is another puzzle piece that allows you to bring uncertainty quantification to any machine learning model.

What were the puzzle pieces in your machine learning projects that helped fit the model into a larger context?

P.S.: Timo and I write a book about bringing those puzzle pieces together for machine learning in science. Sign up on Leanpub for updates.