Mindful Modeler

Share this post

From bare-bones to holistic machine learning

mindfulmodeler.substack.com

Discover more from Mindful Modeler

Better machine learning by thinking like a statistician. About model interpretation, paying attention to data, and always staying critical.
Over 8,000 subscribers
Continue reading
Sign in

From bare-bones to holistic machine learning

Christoph Molnar
Aug 8, 2023
8
Share this post

From bare-bones to holistic machine learning

mindfulmodeler.substack.com
Share

Machine learning lives on a scale from bare-bones to holistic.

Think of bare-bones machine learning as a casting mold to produce puzzle pieces:

This casting mold represents all the choices that have been made for the modeling process: which features to use, what the target is, what type of task, how the model will be used, how it will be evaluated, and so on.


Tomorrow is the last chance to get the SHAP book with an early bird coupon. Get your copy today!

Get SHAP book


Training the model is like molding a puzzle piece using this cast.

You decide whether you want to interpret the yellow color as liquid gold or as slime mold. Both work for this analogy.

How well can the gold/slime fit the casting mold? The equivalent in the case of machine learning is the evaluation metric that tells us how well the model “fits the cast”.

Bare-bones machine learning is the part of modeling where the task is already perfectly defined and it’s mostly about model comparison and model tuning.

Bare-bones ML invites the modeler to put on blindfolds, chug an energy drink and minimize L(Y, f(X)). No questions asked.

There’s a platform called numerai where you can compete in machine learning competitions. Each row in the data corresponds to a stock at a given time, but you don’t get to know what the features stand for, they are just named “feature1”, “feature2”, … And as in most machine learning competitions, everything has been set for the player: the evaluation metric, which data to use, etc. This competition is as bare-bones as it gets in terms of modeling. Other examples of bare-bones machine learning are educational toy examples which you find in introductory courses, benchmarks on platforms such as OpenML, and automated machine learning (AutoML).

While bare-bones territory may feel like a playground, it’s not kindergarten: If you have ever competed in a Kaggle challenge with ambitions to climb the leaderboard or even win, you know the struggle.

That’s the strength and the weakness of supervised machine learning: It’s the modeler’s goal to quickly define the task at hand so that it becomes a bare-bones ML task and you can just let the algorithms loose. If the cast is set correctly, machine learning excels at producing great models. But it also distances the modeler from the data and the larger puzzle.

Holistic Machine Learning

If you are an experienced modeler, you know that life rarely gives us bare-bone ML tasks. It‘s tedious to translate a problem into a prediction task and get the right data.

The reason why we molded a puzzle piece in the first place was because it becomes part of a bigger puzzle. While bare-bones ML is the fitting process of creating the puzzle piece, holistic machine learning is also about designing the cast so that machine learning produces a puzzle piece that fits:

Your model needs to fit with your business and systems in a technical sense. But beyond that, you often need many other puzzle pieces to connect your model and get a full picture. Holistic ML is also about producing puzzle pieces to enhance the model:

  • You might need uncertainty quantification so that predictions can be used with confidence.

  • You might need interpretability so that you can convince another department to actually use your model.

  • You need a plan for what happens with distribution shifts.

My writing is about collecting and teaching puzzle pieces to enable holistic machine learning. Interpretable machine learning is such a puzzle piece, which is especially easy to dock onto your model when you use model-agnostic interpretability tools such as SHAP values. Conformal prediction is another puzzle piece that allows you to bring uncertainty quantification to any machine learning model.

What were the puzzle pieces in your machine learning projects that helped fit the model into a larger context?

P.S.: Timo and I write a book about bringing those puzzle pieces together for machine learning in science. Sign up on Leanpub for updates.

8
Share this post

From bare-bones to holistic machine learning

mindfulmodeler.substack.com
Share
Comments
Top
New
Community

No posts

Ready for more?

© 2023 Christoph Molnar
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing