Several papers I reviewed claim that demand can be either intermittent or lumpy. They then mention the Syntetos-Boylan-Croston (SBC) classification and use the thresholds from Syntetos et al. (2005: ) to do some things with ML methods. Sounds reasonable?

No! And here’s why.

Actually, I’ve already explained this in a previous post, but let me summarise the main points again.

First, intermittent demand is the demand that happens at irregular frequency. That’s the definition John Boylan and I came up with in our paper (this one). But even before that, the literature generally agreed: if you observe naturally occurring zeroes (e.g., no one wants to buy a product), then the demand is intermittent – even if there’s only one zero in the data.

Now, Syntetos et al. (2005) specifically studied intermittent demand and proposed a classification to help choose between Croston’s method and SBA. Their classification includes four types (see image in the post):

1. Erratic but not very intermittent
2. Smooth
3. Lumpy
4. Intermittent but not very erratic

The thresholds they used (ADI=1.32 and CV²=0.49) were only intended to guide the choice between Croston and SBA. And “lumpy”, as you can see, is just a special case of intermittent demand!

Yes, you can classify intermittent demand into “lumpy” and “smooth”, but this separation is not well-defined. Use a different classification (e.g., this paper) and you’ll get different results. In fact, practically speaking, your ML approach likely doesn’t need this classification at all.

So, here are a two things you should NOT DO:

1. Saying that demand can be “intermittent” or “lumpy” – the latter is a subset of the former.
2. Use ADI=1.32 and/or CV²=0.49 to categorise demand, unless you’re selecting between Croston and SBA. And let’s be honest, you’re probably not doing that. So forget about it!

And honestly, stop overusing SBC! Lately, I’ve seen more harm than good from it. If you really want to use it, make sure you’ve read carefully and understood the original paper.

But if you don’t know what you are doing, SBC is not for you!

Message SBC is not for you! first appeared on OpenForecast.

What is this paper about?

Intermittent demand is a common challenge in sectors like supply chain and retail. But the key issue is that zeroes in sales can happen for two fundamentally different reasons (see one of my previous posts):

• Nobody wanted to buy the product (naturally occurring zeroes),
• Nobody could buy the product (artificially occurring due to stockouts, etc).

However, forecasting methods are typically unaware of this distinction and treat both types equally. This can lead to inaccurate forecasts and poor decisions. On top of that, existing classification schemes for intermittent demand (such as SBC) use arbitrary thresholds and rely on choosing between forecasting methods like Croston and SBA. There’s a clear need for smarter, more flexible tools that can distinguish between types of demand and make classifications practical.

In this paper, we introduce a two-stage, model-based framework called “Automatic Identification of Demand” (AID), designed to bring more clarity and accuracy to demand classification. The first stage uses a data-driven approach to detect artificially occurring zeroes. Once those are accounted for, the second stage classifies the demand into one of six categories based on key characteristics: whether the demand is regular or intermittent, whether it consists of count or fractional values, and whether intermittent demand is smooth or lumpy in nature. AID detects stockouts by analysing demand intervals using the Geometric distribution, then flags the demand as one of those six types based on several simple statistical models.

We applied AID to a retailer dataset covering over 31,000 products with weekly sales across three stores. Based on that, we generated several features and tested multiple approaches (local level, pooled regression, and LightGBM) to see whether their accuracy improved. We found that:

1. Correcting for stockouts significantly improved the accuracy of all approaches;
2. Using a mixture approach (separating demand into sizes and occurrences) yielded large gains in accuracy, regardless of the forecasting method used;
3. Further splitting the data by demand categories (e.g., regular vs. intermittent, smooth vs. lumpy) provided additional, though more modest, benefits.

We argue that these three principles are universally valuable for forecasting, no matter what approach you use. If you face intermittent demand, at a minimum, consider detecting stockouts and then using the mixture approach.

Hope you find this paper useful. Let me know what you think in the comments.

Message Svetunkov & Sroginis (2025) – Model Based Demand Classification first appeared on OpenForecast.

The mistake? Using error measures based on the Mean Absolute Error (MAE). This is a crime against the humanity when working with intermittent demand. I’ve explained this issue multiple times before (here, here, and here), but it appears that this idea needs to be repeated over and over again. Let me explain.

MAE is minimised by the median. In the case of intermittent demand, the median can often be zero. If you use MAE (or scaled measures like MASE or sMAE) to evaluate forecasts and compare, for example, Croston, TSB, ETS, and an Artificial Neural Network (ANN), you may find the ANN outperforming the others. However, this could simply mean that the ANN produces forecasts closer to zero than the alternatives. This is not what you want for intermittent demand! The goal is to capture the structure correctly and produce conditional mean forecasts (typically). Instead, by relying on MAE, you might conclude: “We won’t sell anything in the next two weeks”, implying that there’s no need to stock products. This is apparently wrong and unhelpful.

Attached to this post is a figure showing three forecasts for an intermittent demand series:

• The blue line represents the mean of the data;
• The green line is a forecast from an Artificial Neural Network;
• The red line is the zero forecast.

In the figure’s legend, you’ll see error measures indicating that the zero forecast performs best in terms of MAE, followed by the ANN, and lastly, the mean forecast. Based on MAE, the conclusion would be: “We won’t sell anything, so don’t bother stocking the product”. But this outcome occurs solely because 12 out of 20 values in the holdout are zeros, making the median zero as well.

On the other hand, RMSE provides a more reasonable evaluation, showing that the mean of the data is more informative and preferable to the other methods.

The brief summary of this post is: *Don’t use MAE-based error measures for intermittent demand!* (Insert as many exclamation marks as you’d like!)

P.S. Actually, as a general rule, avoid using MAE for evaluating methods that produce mean forecasts. For more details, check out this post.

P.P.S What frustrates me a lot is that the reviewers of those papers did nothing to fix this issue, which means that they are clueless about that as well.

Message Don’t use MAE-based error measures for intermittent demand! first appeared on OpenForecast.

The featured image in this post shows two forecasting approaches: one that tries to predict when demand happens (the yellow line), and the other one that tries capturing the structure of the demand and extrapolates it (the blue line). The green line shows the values in the holdout, and the RMSE indicates the error of the two approaches. Apparently, the straight line is better in this example. Let’s discuss why.

Just a reminder, intermittent demand is the demand that happens at irregular frequency. By definition, we cannot know when a person will come to our store and buy the product. We operate with probabilities in this case, and can say sometimes that the probability of purchase goes up or down due to some factors (seasonality, holidays, promotion etc). When a spherical ML expert in vacuum hears about probability, the first thing that pops to their mind is the “decision boundary” for classification task. Why not set some threshold and say that if the probability is higher than that, the product will be bought and in the other case it won’t?

Well, while this works in classification, it typically doesn’t make sense in demand forecasting.

First, there’s not much structure to capture in intermittent demand besides the basic level, external factors, such as promotions and calendar effects, and occasional trend. Yes, some of them might change the probability of occurrence, and, for example, show that a product will be bought on Monday with 90% probability. This still does not mean that the product will be indeed bought. Saying that it will is just informed guessing, not forecasting.

Second, point forecast is supposed to capture the structure and filter out the noise (see this post). In case of intermittent demand, the structure consists of two parts: expected occurrence (probability) and demand sizes. If we substitute the probability with zeroes and ones based on some threshold, we’ll end up overfitting the noise, but on a different level than usually: the future is uncertain and we can never say for sure what will happen and when, yet we would be playing a guessing game, hoping to be correct. It is like tossing a coin, trying to guess how it will land next time. If you want to have an expectation in that experiment, you should have probability, not a sequence of zeroes and ones.

Third and most important, working with intermittent demand, we typically want to solve a specific problem. The classical example is inventory management, in which case we don’t care whether customers will come and buy our product on Monday, instead of Tuesday. We care about having enough product on shelves to satisfy customers throughout a period of time, while our product is being delivered (lead time). So, the goal in this case is to identify the appropriate safety stock level based on the current stock and thus get an estimate of the demand over lead time, not to predict when people come and how much they will buy. Focusing on the point forecast in this setting is a futile task.

So, when working with intermittent demand, don’t waste your time on trying to forecast when the demand will happen. Focus instead on getting the structure correctly and then understanding what is needed by decision makers and how it will be used.

Message Intermittent demand: don’t try to predict WHEN it will happen first appeared on OpenForecast.

Naive is a forecasting method that uses the last available observation as a forecast for the next ones. It does not have any parameters to estimate, it does not require training, it can be applied to the sample of any data (even if you only have one observation). When you deal with a regular demand, it makes perfect sense to use Naive as a benchmark, because it costs nothing in terms of computational time, you get a forecast of demand, and if you cannot beat it, you should rethink your forecasting process.

However, in case of intermittent demand, the demand itself does not happen on every observation. As a result, when the Naive copies the last available value, it can either reproduce either a proper non-zero demand, or just the absence of demand. The latter implies that nobody bought our product today, and nobody will do in the next week or whatever the forecasting horizon we use. In the following image, Naive will be the most accurate forecasting method, because in the training set, the final observation was zero, and in the test set we did not have any sales:

But is this useful? To answer this question, we need to understand what specifically we are forecasting when we deal with demand with zeroes.

As discussed in a previous post, zeroes can occur for different reasons: some of them happen because nobody came to buy the product (naturally occurring zeroes), while the others appear because there was some sort of disruption (e.g. a stockout) or a product was discontinued (artificially occurring zeroes). The two situations are fundamentally different, but if we work with the sales data exclusively (no stock information), it can be hard to tell the difference between them. Naive might work perfectly in both cases, forecasting no sales for the next few observations, and it can be 100% right in some cases. But the problem is that this is not useful. If we indeed cannot beat Naive on the data with zeroes, it does not mean that we should use it, because there is a chance that we have stockouts in the holdout period. If that’s the case, we might be doing something fundamentally wrong. After all, “we will not sell anything” is in general a simple statement, but not ordering products based on that could be a mistake, because “no sales” is not the same as “no demand”. In fact, if Naive indeed performs very well on your series with zeroes, this might indicate that your evaluation is wrong and you need to clean the data, removing the discontinued and out of stock items from the evaluation.

There are three lessons here:

1. we should forecast demand, not sales;
2. we should measure accuracy on the data with naturally occurring zeroes – do data cleaning before setting up your evaluation;
3. it’s better to use a benchmark that tries capturing demand, not the one that reproduces sales.

Arguably, a more helpful benchmark forecast would be the one in the following image:

The forecast above was generated using the Simple Moving Average, and it tells us that there is a demand for the product over the next 13 days. Yes, it is less accurate than Naive, but it gives an estimate of the expected demand, not the expected sales.

Message Why Naive is not a good benchmark for intermittent demand first appeared on OpenForecast.

First, a quick note: it’s a mistake to define intermittent demand simply as “demand with zeroes”. That definition is incomplete and can be misleading. As some of you know, the definition I prefer is that intermittent demand occurs at irregular frequencies. This means the zeroes in such demand are unpredictable and happen simply because nobody wanted to buy the product on a specific day. Unless you know precisely who will buy and how much, you can’t predict if there will be demand that day. These zeroes can be considered “naturally occurring”.

But zeroes can also happen for other reasons. People might want to buy a product, but it may be unavailable. This typically happens due to stockouts, caused by either incorrect safety stock levels, supply chain disruptions (e.g., a container ship running aground), or a product being discontinued by the company. Sometimes, zero sales occur because a store was closed for a holiday, a gas leak, a flood, or another unexpected event. These types of zeroes are explainable and sometimes even predictable, so we can call them “artificially occurring”.

Furthermore, zeroes may appear at the start of a time series if a product was recently introduced and lacks a sales history. These zeroes don’t provide useful information for forecasting.

Some zeroes might also occur seasonally, for example, for Christmas-related products. These too can be classified as artificially occurring because, while there may be a small demand for such items, it’s usually unreasonable to sell them just to satisfy a handful of customers.

Finally, errors in the system can result in zeroes. For example, sales might not have been recorded correctly, leading to either zeroes or missing values (sometimes treated as zeroes). These can also be categorized as “artificially occurring”.

Demand with only artificially occurring zeroes isn’t intermittent; it is regular demand with issues.

Having said that, the reality is often more complex. You can easily have intermittent demand with stockouts, and distinguishing between naturally and artificially occurring zeroes in such cases can be challenging.

But why bother?

If your goal is to forecast demand (not just sales), you need to address artificially occurring zeroes in your data. When applying models, you should indicate which observations should either be ignored or treated differently. Similarly, when measuring the performance of your models (e.g., forecasting accuracy), you should evaluate them on data without artificially occurring zeroes. Otherwise, you’ll end up testing which model forecasts stockouts or system failures better, rather than actual demand. This ties into the well-known principle: “You should forecast demand, not sales”. In the case of intermittent demand, this is not only difficult but also extremely important.

Here’s an example of a time series N27364 from the M5 competition (Makridakis et al., 2022):

This isn’t a unique case, most time series in the M5 dataset have stockouts. In this specific example, gaps in sales are apparent and likely caused artificially. If we train a model on this data, it might those zeroes into account and produce inaccurate demand forecasts, e.g. lower point forecasts than necessary. The problem worsens if the test set also contains stockouts, as the selected model would be the one that forecasts artificially occurring zeroes better. Using such a model in decision-making could be harmful, leading to erroneous decisions like discontinuing products that actually sell well.

As a final note, Stephan Kolassa has given excellent presentations on the challenges of forecasting in retail. He has shared insightful examples of the complexities of tracking sales and stock. For instance, he discussed this topic in one of the CMAF webinars and in this short video.

Message Why zeroes happen first appeared on OpenForecast.

Intermittent demand comes in different flavours: sometimes zeroes occur frequently with low demand volumes, while other times the volumes are high with occasional zeroes. Demand patterns can also change over time, with demand either becoming obsolete (more zeroes) or building up (fewer zeroes). How can we classify these different types of demand? Well, there is a paper on that (academic Rule 34)!

Syntetos, Boylan & Croston (2005) developed a categorization scheme using the Average Demand Interval (ADI) and Coefficient of Variation (CV). They compared MSE performance of Croston (1972) and SBA (Syntetos & Boylan, 2005) forecasting methods, creating four categories of intermittent demand with ADI=1.32 and CV²=0.49 as cut-off values:

1. Erratic but not very intermittent
2. Smooth
3. Lumpy
4. Intermittent but not very erratic

These are distinct categories of INTERMITTENT demand, though the names of the first and last are sometimes shortened to “Erratic” and “Intermittent,” causing confusion (intermittent demand can be intermittent?). The authors recommended using Croston for (1) and SBA for the other three. The image below illustrates these categories, with ADI increasing from left to right and CV increasing from bottom to top.

But that’s not all! Kostenko & Hyndman (2006) found that the split between Croston and SBA does not form four distinct areas – the cut-off should be non-linear. While mathematically correct, this classification has not gained as much popularity as SBC because it is more complicated. There is also a reply from Syntetos, Boylan & Croston to Kostenko & Hyndman, where the authors of the original classification agree with the new cut-off but also point out that their classification is practical, while not necessarily as accurate as KH.

Furthermore, Petropoulos & Kourentzes (2015) extended the KH classification by adding Simple Exponential Smoothing for regular demand, where the average inter-demand interval equals to one.

So, we have at least 3 popular techniques. So what?

These classifications were designed for conventional intermittent demand (e.g., spare parts) assuming stable ADI and CV over time. But what if demand builds up (fewer zeroes, higher volume) or slows down? In such cases, SBC, KH, and PK would be inappropriate. Moreover, classification should serve a purpose. SBC’s original purpose was to help choosing between Croston and SBA. So, the threshold between “lumpy” and “erratic” is based on these methods’ MSE performance. Are you using these methods in your case? If not, why bother with SBC/KH/PK classifications?

In the 2024, we have more advanced models and methods, and, for example, using SBC to decide between XGBoost and Poisson regression would be unwise. You need a different classification! Or maybe you don’t need one at all, just apply competing approaches and select the most appropriate one based on the holdout performance.

So, next time you work with intermittent demand, stop for a second and think what you plan to do. SBC is useful, but don’t use it just because you don’t know what to do!

Message Intermittent demand classifications: is that what you need? first appeared on OpenForecast.

Intermittent demand is the demand that happens at irregular frequency (Svetunkov & Boylan, 2023). This means you cannot predict when demand occurs: you might have some sales of a lipstick on Monday but then nothing on Tuesday, and there is no pattern in purchases. This additional element of randomness, not only about how much but also when people buy, creates additional challenges. Intermittent demand can be split into two parts: demand sizes (how much is needed) and demand occurrence (when it is needed) or demand intervals (the time between purchases).

If your demand has zeroes for some specific reasons, such as certain times of day, it may be non-intermittent. Zeroes can also result from shortages or recording errors, which should be treated differently. In those cases, the demand itself maybe non-zero, but sales would be zero due to technical reasons. This means that not every series that has zeroes should be considered and treated as intermittent.

An important thing to note is that intermittent demand is not the same as count demand, but it is a wider term. Zero value in case of count data is just a possible outcome, e.g. nobody came to the hospital at this hour because there was no demand, and thus we recorded a zero value. In fact, it is possible for count demand not to have any zeroes at all, while for the intermittent it is a requirement. Furthermore, zero has a slightly different meaning in the intermittent demand: there might be a potential non-zero demand for paracetamol at our shop at this hour, but we just did not observe it because a customer didn’t make it to the shop. The difference between the two situations is intricate, but fundamental. However it is worth noting that intermittent demand can be either count or non-count. For example, demand for jet engines is count, while demand for EV charging is non-count.

The fundamental difference between count demand and intermittent demand lies in their treatment. In intermittent demand, you may need separate models for demand occurrence and demand sizes. In count demand, using an appropriate count distribution model (e.g., Poisson or Negative Binomial) or an ML method is sufficient. The split into demand sizes and demand occurrence (or demand intervals) gives additional flexibility and allows capturing more complex patterns in the data. For example, the frequency of purchase might increase if the product becomes more popular, implying that the probability of sale goes up, but the demand sizes might stay on a similar level. Another situation would be when product becomes obsolete, where both demand sizes and demand occurrence decline over time. Finally, there is also a possibility of demand having the same level of occurrence, not changing substantially over time (e.g. sales of large machines). All of these cases can be captured using models that split demand into two aforementioned parts.

The following image provides several examples of intermittent demand, which we will discuss in future posts:

Finally, intermittent demand can appear at higher recording frequencies. For instance, while daily demand may seem regular, hourly data may reveal zeroes. This increases forecasting complexity, so when selecting an aggregation level, we should consider whether the added complexity aligns with the decision-making based on the forecasts.

Additional resources to read: a post about measuring accuracy of intermittent demand, a monograph on intermittent demand forecasting by Boylan & Syntetos.

Message Introduction to intermittent demand first appeared on OpenForecast.

• Introduction to intermittent demand
• Do you really need SBC?
• Lumpy is a type of intermittent demand and the usefulness of the classification scheme;

And there is more, check the Intermittent Demand category on the website.

In one of the previous posts, we have discussed how to measure the accuracy of forecasting methods on the continuous data. All these MAE, RMSE, MASE, RMSSE, rMAE, rRMSE and other measures can give you an information about the mean or median performance of forecasting methods. We have also discussed how to measure the performance of models in terms of prediction intervals, and should now be comfortable with such measures as Range, Coverage, pinball, MIS. But all of this might become irrelevant when we face the intermittent demand – it can really make a bright day dark and screw with you and your measures. So, care is needed if you have data with randomly occurring zeroes.

We have already discussed aspects of intermittent demand in a post about the intermittent exponential smoothing model some time ago, and we have seen some time series examples in one of the previous posts, so I will not repeat myself, but I still think that there are a couple of important notes worth making about it.

Intermittent time series is the series that has non-zero values occurring at irregular intervals. This implies that there might be some periods of time, when we observe zeroes (e.g. no one buys our product). An example of such time series is shown below:

As you can see, there are two sources of randomness in this data: there is a randomness about the non-zero value itself (e.g. demand size) and there is also a randomness about the occurrence of the non-zero value (e.g. demand occurrence). So, a very general model for intermittent demand can be represented as:
\begin{equation} \label{eq:general}
y_t = o_t z_t ,
\end{equation}
where \(y_t\) is the observed value at time \(t\), \(o_t\) is the binary occurrence variable and \(z_t\) is the demand sizes variable. While there are some statistical models that don’t do this distinction, the intermittent demand forecasting methods that are most popular in practice do it either directly or indirectly.

Although we can create very fancy forecasting models (there are even tries to use neural networks for this), we usually cannot accurately predict, when specifically the product will be bought and how many units we will sell. All we usually can do is to produce the mean value and / or prediction intervals for the demand itself. And even if we do that, the next thing to ask ourselves is: “What do we do with that?”

In a supply chain context, the typical decision would be how many units of product to order or to produce, given the amount that we already have. In this case we are usually talking about the safety stock – how many units of the product we should hold, so that we satisfy the demand and don’t have empty shelves. Typically, this is determined based on calculation of quantiles of a distribution. In many cases, in practice, the Normal distribution is used, and I cannot describe on how many levels this is wrong (the most obvious thing to point out is that usually we cannot have negative amounts of product). But let’s not get too much distracted.

What does the safety stock typically depend on? In many cases this is not just the demand of a product, but specifically the demand over the lead time – for example, how many products we will sell over the next week. Why? Because the deliveries tend to happen once over a time period, e.g. once a week, and we tend to order products once in a period of time – it does not make sense to order the product each time we run out of it. So, we do periodic reviews of the inventory and decide, how much we should order, so that we don’t run out of stock until the next replenishment.

Good. So, how do we get the information about the demand over the lead time? Here comes our forecasting model! But all of this implies that we should focus not just on the point and interval forecasts for each specific horizon, but on the values aggregated over the lead time. This is the first important difference between conventional forecasting and forecasting for inventory purposes (see for example, Kourentzes et al., 2019).

As you can see, the relation between the actual demand forecast and the decision of how much to order is complex. And this means that all those nice error measures, discussed in the previous posts, might not give us the important information of how our model performs in terms of orders. The model can produce very accurate forecasts, but this does not necessarily translate directly to correct ordering policy.

In order to align the model with the specific decisions, one can revert to simulations, so that it becomes more apparent how the model performs in terms of achieved service level, lost sales and excess inventory (which all are sort of proxies to the object of real interest in practice – the cost). This is one of the common approaches in the modern supply chain and inventory literature. Unfortunately, this approach is computationally expensive and may vary from one situation to another, so you would need to set up the simulation for each separate company and potentially for different products.

An alternative approach would be to at least asses the performance of models over the lead time, not on each observation. This means that we need to refer to cumulative values over the period of time:
\begin{equation} \label{eq:demandOverTheLeadTime}
Y_{t+h} = \sum_{j=1}^h {y}_{t+j} .
\end{equation}
Then we can measure, for example, how our models perform in terms of working stock (aka “cycle stock”). In this case we are only interested in the mean demand over the lead time, which for the additive models is relatively easy to deal with because the expectation of the sum of the demands is equal to the sum of the expectations:
\begin{equation}
\begin{aligned}
\text{E} \left(\sum_{j=1}^h {y}_{t+j} \right) = & \text{E}\left(\sum_{j=1}^h (\hat{y}_{t+j}+e_{t+j})\right) =\\
&\sum_{j=1}^h \text{E}(\hat{y}_{t+j}) + \sum_{j=1}^h \text{E}(e_{t+j}) = \sum_{j=1}^h \text{E}(\hat{y}_{t+j}).
\end{aligned} \label{eq:workingStock}
\end{equation}
where \(\hat{y}_{t+j}\) is the point forecast of a model. This means that we can produce the \(h\) steps ahead forecast and aggregate it over the lead time \(h\) and then compare it with the actual values for the same period. However, if we deal with the additive models in this context, then we are assuming that the demand can be negative – which is usually unrealistic, especially for intermittent demand. This means that a different model is needed, and \eqref{eq:workingStock} might not hold any more. So, for example, in case of ETS(M,N,N) simulations are necessary in order to produce the correct cumulative conditional expectation over the lead time.

Now, let’s say that we have managed to produce meaningful cumulative point forecast. What’s the next step? We need to measure its accuracy. The good news is, RMSE-based measures can be used in order to assess the performance of models in terms of the working stock (I hope you still remember, why MAE- and MAPE-based measures are not appropriate for intermittent demand). So, we can evaluate the Squared Cumulative Error and compare models based on it:
\begin{equation} \label{eq:workingStockRMSCE}
\text{SCE} = \left( \sum_{j=1}^h y_{t+j} -\sum_{j=1}^h \hat{y}_{t+j} \right)^2 .
\end{equation}
We can use relative or scaled measures, if we want to compare performance of models across products – all the things discussed in the previous post are applicable here. The main limitation here is that, given that we deal with intermittent demand, Naive method might cause problems if used as a benchmark. An example is a situation, when it produces zero forecasts for the holdout sample (because the last observation was zero), and the holdout contains only zeroes. This might happen by chance, but it will mess your beautiful relative error measures. A simple average of the whole series might be more appropriate as a benchmark for the intermittent demand.

There is a measure, which is ideologically close to the SCE. It is called “Periods-In-Stock” – PIS (Walstrom, 2010):
\begin{equation} \label{eq:workingStockPIS}
\text{PIS} = \sum_{j=1}^h \hat{y}_{t+j} -\sum_{j=1}^h y_{t+j} .
\end{equation}
Have you noticed that it is calculated as forecast minus the actual values? This was done by the author intentionally, so that the sign of the measure aligns with the inventory decisions. If the value is negative, then this means that we missed sales. If it is positive, then this means that we had too much stock and did not sell as much as we should have. Petropoulos & Kourentzes (2015) have proposed several modifications of PIS, which allow aggregating this measure over several products. So, have a look at the paper if you are interested, but I will not expand here.

By looking at those error measures, we can evaluate the performance in terms of working stock. But, as it was mentioned before, this does not necessarily translate to safety stock performance. So we need to figure out how to align our error measures with it. It seems that the closest we can get to it is by producing the specific quantiles for the values accumulated over the lead time. We typically don’t care about the lower bound, because in case of intermittent demand it is usually just zero, plus, the lower bound is not helpful for inventory decisions (e.g., knowing what will be the level of sales in 2.5% of cases is not helpful). So, we are more interested in the upper bound. In a realistic situation we would need to simulate the data from our model (something like a 1000 runs for the possible future outcomes), sum it up over the lead time and then take the upper quantile (for example, 95% or 99%). We have to do it this way, because we might not have expressions for the quantiles for the values accumulated over the lead time. But this way we will get quite close to the safety stock.

How do we measure performance of models in terms of this upper bound? I would be inclined to recommend pinball function, keeping in mind its limitations, discussed in the previous post. We can also analyse the coverage and range, and if we really need to use something like Mean Interval Score (MIS), then it makes sense to use Quantile Score instead (Gneiting, 2007), because it only takes the upper bound into account. However it needs to be modified in order to reflect the idea of cumulative values:
\begin{equation} \label{QS}
\text{QS} = \left(Y_{t+h} – U_{t+h} \right) \left(\mathbb{1} \left\{ Y_{t+h} \leq U_{t+h} \right\} -\alpha \right) ,
\end{equation}
where \(\alpha\) is the confidence level (in our example 95% or 99%), \(U_{t+h}\) is the value of the quantile and \(\mathbb{1}(\cdot)\) is the indicator function, returning one, when the condition is true and zero otherwise. The QS has similar interpretation and working principle as MIS: if the values lie below the specified quantile, then the measure is penalised less than in the case, when they lie above: in the former case the indicator function returns 1, so the weight becomes \(1-0.95=0.05\), while in the latter case it is equal to zero, so the weight is \(0 -0.05 = -0.05\). Note also that in that latter case the difference between the actual value and the bound is negative. This means that QS will always be positive, and in the ideal situation, when all the future values lie on the bound, the QS is equal to zero. The difference between QS and pinball function is in what specifically they measure. The QS measures the performance in terms of coverage and range for the bound, while the pinball tries to measure, how accurately we hit the specific quantile. So, arguably, QS is closer to what we would be interested in real life: we want to have the coverage closer to nominal (so that we achieve the nominal service level), but with the smallest range possible (so that we do not order too much and do not have huge holding costs). Based on QS, we can calculate scaled or relative measures in order to compare performance of forecasting models across the products.

So, what about the other intermittent demand contexts? In some of them the situation might be quite different and in some sense easier than in supply chain. For example, in case of the forecasting the patients flow in a hospital, we often do not need to deal with values over the lead time, which simplifies things substantially. In addition, we might not care much about the average number of patients for a specific day, but be more interested in the prediction interval. This way we can assess how many patients to expect in the “worst” / “best” case scenarios for the selected confidence level. Based on these values, we then decide, how many nurses, doctors and beds we should have for a specific day in order to service, let’s say, 95% of the patients in the next 4 hours. Note that yet again the prediction interval is not directly related to this goal, but at least it can be considered as an approximation to it, if we know how many nurse and doctor hours we typically need in order to process a specific type of patient. This also implies that we should evaluate the accuracy of the prediction intervals, not the accuracy of the mean. And, when doing all of that in case of intermittent demand, once again, we usually don’t care about the lower bound, because we can typically say even without any model that in the worst (best?) case we won’t have any patients at all. Instead, we should concentrate on the upper bound of the prediction interval, and we can use the measures, discussed in the previous context in this post.

Having said all this, the general recommendation for dealing with intermittent demand – avoid it at all costs. Why? Because intermittent demand is messy to identify and to deal with. Besides, are you sure that you really need to produce forecasts on that level of sales? What specific decisions do you make based on those forecasts? Maybe you don’t need to know how many watermellons you will sell in each separate shop every day. Maybe you should switch to weekly data, where the intermittency disappears, or aggregate sales of products over larger volumes and over several shops in the chain… So, before diving into the problem and trying to evaluate models, think about the decisions you usually make based on those models.

Message What about all those zeroes? Measuring performance of models on intermittent demand first appeared on OpenForecast.