Can an office game outperform machine learning?

Wisdom of the crowds, prediction markets, and more fun in the work place.

I was recently talking to a friend who works for an insurance company. Every quarter, insurance companies must report a couple of numbers for regulatory reasons. Most importantly, they have to report their solvency ratio, which is their eligible funds divided by their solvency capital requirement. This ratio has to be 100% or higher to make regulators happy. To understand this post, you don’t really need to understand what those numbers mean. Because I certainly don’t.

It takes the insurance company weeks to calculate the solvency ratio. But at the same time, it’s an important factor in many insurance business decisions. Ideally, they could have estimates of those numbers all the time, and — even better — they could simulate what happens to those numbers when they make certain business decisions. So this insurance company turned to machine learning to estimate these numbers. First, they hired a consulting firm to build the model. Great! Now, all year long, they could get an estimate for the next quarter and make decisions based on those estimates.

Introducing a fun game

As it happens, there’s another way to estimate these numbers: Some of the employees play a little prediction game. Before every quarter, they predict the solvency ratio. After the quarterly numbers come out, they have a leaderboard to see who has the best prediction. No prizes involved. Just a little bit of fame and kudos. So there’s no big incentive to invest time and effort, it’s just a guess.

Here’s the fun part: You can also place the machine learning model on this leaderboard. How does it perform compared to the humans’ predictions? As it turns out often it’s just some place in the middle. Think about it: There is a team of highly paid machine learning experts who are being outperformed by a couple of people making guesses. On one side there is an expensive machine learning model. Probably with a high upfront cost, and probably not cheap to maintain. As I understood a couple of people are involved in the responsible machine learning team, more or less full-time. All that for a mediocre prediction. On the other side, we have a couple of people predicting the solvency ratio FOR FUN, and many of them end up better than the machine learning model.¹

This makes me wonder: Could they try using the wisdom of the crowds (e.g. an ensemble) and compare it with machine learning? This is a very cheap way to estimate the target value. The cost is just a small amount of time spent by the players. And that is very little compared to having a machine learning team. One disadvantage of crowdsourced predictions is that they are slower. With a machine learning model, we can get very quick estimates. That also makes it more suitable for conditional scenarios, where you want to predict the solvency rate under certain conditions. But there is a version where you could leverage the wisdom of the crowd to get continuously adjusted predictions and even conditional predictions: prediction markets.

Prediction Markets

In a prediction market, you buy “shares” for your desired prediction. The shares associated with the correct outcome later receive a payout. There is a market for these shares, and their prices can be translated into probabilities. This has several advantages: You get an estimate over time, reflected in fluctuating share prices. In theory, you can also introduce conditional prediction markets (e.g., predicting the solvency ratio if the company introduces product X). However, this is more complex and requires many participants to ensure sufficient market liquidity.

The idea behind prediction markets is that everyone in the company has valuable information relevant to the solvency ratio. The market then acts as an information aggregation mechanism, leveraging the wisdom of the crowds. Averaging the predictions of many participants can, under certain assumptions—such as partial independence—outperform individual predictions in expectation. However, you would still have to show that the prediction market consistently outperforms the machine learning model.

Why are prediction markets not more common? We can see a rise in online prediction markets like Polymarket, but it doesn’t seem like they are getting established in companies (yet?). Funnily enough, many German companies have tip games for events like the World Soccer Cup. Why not simply establish such prediction games for company-relevant predictions? Like the quarterly numbers. Or to estimate things that matter, like how long a project will take to finish, or whether or not a product will flop or be a hit.

I believe we lack the cultural acceptance of such prediction markets: A prediction market would introduce tension with the machine learning department, which also might serve the narrative of “doing AI”. Predictions of this kind are mostly associated with sports, like betting on the World Soccer Cup, and might be considered unserious or have an association with gambling.

However, if I were an employee, I’d appreciate being involved in such prediction markets. It would be a great way to make the job more playful in a good and useful way, it might even come with a potential bonus payment if you predict well, and I think it could align employees more with company goals.

If you are interested in such topics, I’m working on the book “Prediction Mindset” which covers many aspects of how to make good predictions, including prediction markets. You can sign up for book updates here:

Also, if you want to diver further into prediction markets, here are some resources.

• Paper on the implementation of a prediction market at Hewlett-Packard for sales forecasting

• Podcast on prediction markets

• Blog posts by Robin Hanson who is a strong advocate of prediction markets.

1. As mentioned in the comments: We need a way to pick a good prediction before the numbers are released. This would mean that we would have to show that some people consistently outperform the model, or showing that an ensemble of the prediction is consistently better. Only then could we know for sure whether this prediction game is better than the machine learning model.