demand forecasts: what they model and why they get it wrong

This morning, the CEO of a major UK retail brand told me, “Success in retail is understanding and predicting your customers' behaviour.”

Ask most people who work in retail what they do and “predict human behaviour” won't be their answer. And yet, this is exactly what almost every decision is based on.

Pricing is a prediction of which shoppers will pay at a certain price and which ones will walk away. Buyers make a prediction of which products people will want, and how many of these products they'll want. Allocation (deciding which products, and how many of those products, go where) is a prediction about how the behaviour of the populations around stores differs.

Marketing is a prediction of how customers will respond to different messages, replenishment a prediction of whether the behaviour observed last week will continue this week, markdowns a prediction of the extent to which customers will respond to discounts.

Almost every function of a retailer involves asking the same question: what will this group of people do?

However, the tool used by the industry for answering this was never designed to look at behaviour at all. Retailers use demand forecasts designed to identify trends and patterns and extrapolate them forward.

This approach can follow any level of sophistication, from looking at last year's data and assuming it will be the same this year, to running the data through complex machine-learning-based models. Regardless of the approach, what they are doing is fundamentally the same. They identify patterns in sales from the past, and assume they will continue in the future.

The key insight is that the patterns they identify and assume will continue in the future are just claims about human behaviour. If you assume that the sales this year will be the same as last year, then you're effectively just making the assumption that next season's shopper will behave exactly like last season's shopper. If you run a more complex model which learns that sales for your winter jackets increase the colder the weather is, then you're just assuming that this same behaviour will emerge next season and taking the likely conditions into account to predict demand.

You are indirectly modelling human behaviour, and making the assumption that it will continue without fully understanding why. Notably, this isn't necessarily a problem. Chances are that the weather being colder will lead to more winter jacket sales next year. Unknowingly making the assumption that this behaviour will continue, without realising it, isn't necessarily a bad thing. You'll still make the right decision provided this assumption was right.

Be that as it may, the problem with demand forecasts is that they miss vast amounts of the human behaviours that matter. They face three fundamental problems which will always limit their utility and ability to accurately model human behaviour: data scarcity, non-stationarity, and endogeneity.

data scarcity

Data scarcity is the claim that there's not enough data to learn all the behaviours that matter.

This might seem a bold claim to make about a retailer which has hundreds of stores and decades of sales data. However, when you look at the decisions a retailer needs to make and the granularity of those decisions, the problem becomes evident.

Take Fellside, a fictional outdoors retailer with 200 stores across the UK. They need to choose nine months in advance how many of their Helvellyn-style rain jackets to order for next season. They sold 7,890 units last season, so they could just order 8,000 units and assume demand will be the same. But this year was an especially rainy season, so they probably sold more because of that. On top of that, the economy was in a recession, and an influencer posted a photo of themselves wearing the jacket.

They only have ten years of sales data, so maybe a demand forecast can learn that you sell 10% more when it's a wet season, but what about when it's rainy and there's a recession and the product appeared on 100,000 people's social media feeds? That's never happened before, so a traditional demand forecast will always miss some of the behaviours that clearly affect the demand.

Even worse, perhaps the extra sales this year were due to the influencer. The poor data scientist is likely blissfully unaware of this influencer even posting, and their demand forecast mistakes the spike in demand this year for an underlying higher demand for rain jackets. The demand forecast predicts even higher sales for next year, and Fellside lose thousands, having to mark down stock that didn't sell because the forecast didn't understand the behaviour that actually led to the sales.

Similarly, on a finer scale, the allocators at Fellside need to know what the demand will be for the Helvellyn jacket in each store across the country. They need to know how many of each size of jacket they're going to sell in a given week in a given store so that they can ensure that product is available for the customer to buy.

The problem is that a given store only sells a handful of items each week. The data on the sale of a product looks something like this:

No demand forecast model can learn why they sold 0 one week, then 6 the next week. The statistical models have no context about any data stream you put in, so they can't distinguish between Keswick selling more one week because there was a local event on, or because that was the one week of summer that was overcast, or it was a bank holiday.

When there's not enough data, it can't tell if there's a real pattern or just noise.

The solution the demand forecasts use is to pool across similar stores or similar products. If everywhere sold more products on a bank holiday weekend, then it learns that a bank holiday is associated with more sales. The problem is that again this misses the behaviour that matters. In Keswick, where it's rainy, there are more sales of rain jackets on a bank holiday weekend, but in Newquay, where it's sunny, there are more sales of t-shirts. The model doesn't learn any bank holiday effect because it's not the same everywhere. Fellside's stores understock and they lose sales they would've otherwise made.

All of these trace back to the fundamental problem in the traditional approaches to demand forecasting. You're giving a machine various streams of data with no context on what each stream is and asking it to identify the human behaviour which leads to your sales. The machine has no concept of why it being a bank holiday and rainy might mean you make more rain jacket sales; it can only learn it from identifying patterns in the past. Most of the patterns that are commonsensical to humans have never happened in the data, so a statistical demand forecast doesn't stand a chance.

non-stationarity

Non-stationarity simply means the thing you're trying to predict is always changing.

This is something known to retailers everywhere. Styles and trends change from year to year, and in the case of viral trends, even from week to week.

The problem with non-stationarity is that it exaggerates the data scarcity problem. Suppose Fellside's demand model had seen a similarly rainy August bank holiday weekend four years ago; maybe the model would be able to recognise this and predict a similar effect. However, four years ago lightweight raincoats weren't “in”; people carried umbrellas instead. The demand the model learned has changed.

Now the model predicts you'll sell more umbrellas this year, so you send more to Keswick and end up overstocked on umbrellas and understocked on the raincoats. Again, Fellside lose thousands.

The model is trying to learn the complexity of human behaviour, but by the time it's learned it, the behaviour has changed. Learning behaviour indirectly, by modelling demand, fails and leads to the wrong decisions.

endogeneity

Endogeneity means that the actions you take affect the outcome. The most obvious example is how allocation at a given store changes the demand for each item.

Suppose Fellside are deciding whether or not to introduce the Helvellyn jacket to a store it's never been in before. This store typically sells 100 of a cheaper version, the Rambler jacket, each month.

Your traditional demand forecast might look at similar products in this store and forecast the Helvellyn to have similar sales. But if I start offering the Helvellyn, the demand for the Rambler is likely to change. Maybe everyone currently buying the Rambler would prefer a premium option if it was available, and introducing the Helvellyn would wipe out the demand for the Rambler. Maybe you make good sales at the moment because there are a lot of people who forget an umbrella and are willing to pay to not get wet, so offering a premium option wouldn't capture any of them and demand for the Helvellyn would actually be zero. Maybe there's a subset of people who want a jacket but leave the store as the Rambler isn't premium enough for them, so introducing the Helvellyn would capture this demand as well.

Just like how marketing a product changes the demand, your choice of store allocation changes the demand for each item. People browse in stores and rarely know exactly what they're looking for when shopping. If you make the choice of what to allocate assuming the demand is unrelated to the decision you make, you'll make a wildly wrong decision. You need to understand the behaviour causing these sales to allocate the products in the right place.

so why does anyone use a demand forecast?

Evidently, demand forecasts struggle to capture much of the human behaviour that matters. Individual retailers are losing millions per year.

$1.7 trillionlost to overstocks and out-of-stocks every year — 6.4% of global retail sales (IHL)

IHL's global study of inventory distortion puts into numbers the cost of getting only the question of “how much will we sell” wrong. They estimate that overstocks and out-of-stocks cost retailers $1.7 trillion each year. That's 6.4% of global retail sales that are lost due to demand forecasts struggling to understand customer behaviour. The failures in understanding the customer behaviour that leads to the decisions of “what products?” and “what price should we sell it for?” are harder to quantify, yet undoubtedly just as pivotal.

So why do people still use them?

The answer is that there historically hasn't been a better way. The decisions need to be made by retailers every year as to how much to buy, where to put it, what price to sell it at, when to mark it down. The only competing method of predicting customer behaviour was to rely on the “gut feel” of experts who work in buying, merchandising, allocating and other roles. In practice, most retailers follow a blend of these approaches. Their experts are guided by a demand forecast, overruling it whenever they believe they know better.

At Clinchr, we believe there is a better way. Frontier AI's capabilities and knowledge have opened up a new way of predicting customer behaviour; one that models and understands the behaviour directly, rather than inferring it from patterns in the sales. If you want to hear more about it, get in touch with us at tom@clinchr.ai, or subscribe to be notified of the next instalment of our blog series.