A year has 365.25/7 ≈ 52.18 weeks on average. This is typically resolved by having 53 weeks in some years (roughly once every five to six years). If you just use a seasonal model with 52 weeks, the pattern drifts by about a day every year, and after five or six years, your Christmas peak is a whole week off.

And there is a twist. Many tools do not use ISO weeks. Some count seven-day blocks from the 1st of January (e.g. lubridate::week() in R), so the last one or two days of the year form a “week 53”. Others follow the weekdays but cut the weeks at the start of the year (e.g. WEEKNUM() in Excel by default), so the first and last weeks may contain only a few days. If weekly demand is the sum of daily sales, these short weeks show sharp drops, although nothing has happened to demand. A rough fix is to model the average daily demand per week and scale the forecast back. A better one is to aggregate daily data into ISO weeks in the first place (e.g. with lubridate::isoweek() in R or ISOWEEKNUM() in Excel), which are always full seven-day weeks.

This leaves us with the 52/53 weeks, for which there are several options, each with its own assumptions:

1. Shift the seasonal lag from 52 to 53 in years with the extra week. No additional parameters, but it assumes that week 53 behaves like week 52, which is not true if Christmas falls into it.
2. Use categorical variables for weeks of the year. This avoids that assumption, but you need to estimate the effect of week 53, which you observe once every five to six years, if at all.
3. Use fractional seasonality, e.g. TBATS (De Livera, Hyndman & Snyder, 2011: https://doi.org/10.1198/jasa.2011.tm09771), which models seasonality with harmonics and accepts the period of 52.18. This handles the drift elegantly, but you need to select the number of harmonics, and the harmonics might miss sharp peaks, like Black Friday.
4. Pre-process the data, e.g. merge week 53 with week 52. This assumes that the merged week does not matter, while it often contains the peak you care about the most.

So, which one to use? There is no universal answer. In practice, it often depends on your software and its restrictions, e.g. if it does not support categorical variables, the second option is gone. But whatever you choose, be aware of the assumptions behind it.

In the next post, I’ll tell you about the new TBATS implementation in the smooth package, which handles fractional seasonality and has some features that other implementations don’t. Stay tuned!

Message The problem with weekly seasonality first appeared on OpenForecast.

But first, why bother? Because a method just gives you a number, while a model gives you a likelihood, and with it flexibility: you can select components and features via information criteria, get predictive distributions, and extend the model however you like. So, if you derive a model underlying a method, you bring that method to another level.

A quick recap. Croston split intermittent demand into two parts: how often the product sells (demand intervals) and how much it sells when it does (demand sizes), and applied Simple Exponential Smoothing (SES) to each of them. In my previous post, I showed the simplest model for the “how often” part, which assumes that the chance of a sale does not change over time.

A more complicated statistical model was proposed by John Boylan and me in our iETS paper. It assumes that the occurrence of sales follows a Bernoulli distribution with a time-varying probability. In the “inverse-odds-ratio” version of the model, we track the expected number of periods without sales: if the gap between sales is 4 periods then the probability of a sale is 1/4. Every new zero or one is converted into an error term, and this expected number is updated via ETS(M,N,N), the model underlying SES. When the probability does not change much over time, this model can be estimated with Croston’s method, which is why it can be considered as the model underlying it.

In Python, the basic model is done like this:

from smooth import OM, ADAM

occ = OM(model="MNN", occurrence="inverse-odds-ratio")
occ.fit(y)
model = ADAM(model="MNN")
model.fit(y, occurrence=occ)
model.predict(h=10, interval="prediction")

In the paper, we also showed that this model is especially suitable for new products with demand building up. This means that the gaps between sales become shorter and shorter, i.e. the demand transforms from intermittent to regular. If you expect this to continue in the holdout period, you can use the trend model, and here how to do that in Python:

occ = OM(model="MMN", occurrence="inverse-odds-ratio")

This way, the expected gap between sales would decline in the holdout period, and the probability of a sale would grow. The image in this post shows how this works on an artificial example.

And if you know what drives sales (e.g. promotions or price changes), you can include this information in the occurrence model as well.

So, yes, there is a model underlying Croston’s method, and it allows doing things that the method itself cannot do.

Try smooth for Python: pip install smooth
Occurrence model on wiki

Message smooth in Python: iETS and the Croston’s Method first appeared on OpenForecast.

This idea goes back to Croston, who suggested splitting intermittent demand into demand occurrence and demand sizes. The main image in the post shows a typical intermittent demand series, and the one below shows its occurrence part: “1” when someone bought the product, and “0” otherwise.

We now have a series that can be modelled on its own, and there are many ways to do that, which come down to your preferences and experience: ETS, ARIMA, logistic regression, GBM, Neural Network – you name it!

One thing to avoid, though, is trying to predict the exact periods when sales will happen: when demand is truly intermittent, you can’t. What you can forecast is the probability. I had a post on that topic.

In the smooth package, there is a separate function for this, OM, which allows using ETS, ARIMA and regression elements to model demand occurrence. It supports several types of occurrence, including the odds-ratio, the inverse-odds-ratio (related to Croston’s method) and the direct one (similar to TSB). But the simplest of all is the fixed probability, which assumes that the probability of sale does not change over time. It simply takes the share of periods with sales: if the product sold in 30 out of 100 weeks, the probability is 0.3. If you read my previous post, you might notice that this is essentially 1/ADI turned into a forecast, with the same limitation of ignoring how demand evolves. Still, it is a good benchmark and a starting point for intermittent demand forecasting.

In Python, this is done like this:

from smooth import OM, ADAM

model = OM(occurrence="fixed")
model.fit(y)
fc = model.predict(h=10)

The resulting probability forecast is shown in the following image:

We can also use this directly in ADAM, which splits the demand into two parts on its own. Here is the model with ETS(M,N,N) for demand sizes and the fixed probability of occurrence, iETS(M,N,N)_F:

model = ADAM(model="MNN", occurrence="fixed")
model.fit(y)
model.predict(h=10)

The last command takes the probability from the occurrence model and multiplies it by the forecast of demand sizes. The result is the mean demand per period, just like from any other forecasting method.

Next time, I’ll tell you about the more advanced occurrence models, where the probability changes over time. So, stay tuned!

Try smooth for Python: pip install smooth
Occurrence model on wiki.

Message smooth in Python: fixed probability model for demand occurrence first appeared on OpenForecast.

I don’t know when specifically ADI was proposed, but it was popularised by Syntetos, Boylan & Croston (2005), yes, the same Croston from this post. A demand interval is the number of periods between consecutive non-zero demands, and ADI is just their mean. It lies between one (demand in every period) and, theoretically, infinity (no demand at all). But why would anyone need it?

Originally, it was used together with the coefficient of variation \(CV^2 \) to classify demand into four types (the SBC classification) and to choose between Croston’s method and SBA. The authors compared the theoretical MSE of the methods and found cut-off points: Croston does better only when \(ADI < 1.32\) and \(CV^2 < 0.49\). Otherwise, SBA wins.

My interpretation of the \(ADI < 1.32\) (this is not in the paper): SBA fixes the bias of Croston’s method, which matters most for very slow-moving products. But it does that at a cost. SBA multiplies Croston’s forecast by \( (1 − \alpha /2) \), which just lowers probability of occurrence. If demand becomes less sporadic, SBA underestimates it (e.g. 0.85 instead of 0.9 with \(\alpha = 0.1\)).

But why ADI in the first place? Because historically the main academic work on intermittent demand was carried out in spare parts and similar domains, where sales are rare even at the monthly level (the 2005 paper used 3000 series from the automotive industry). When the probability of occurrence is low, demand does not evolve much over time, so averaging the demand intervals is acceptable.

Nowadays, we face intermittent demand everywhere: in retail, energy, healthcare and other areas. The data is often collected at a more granular level (weekly or daily), and intermittent demand comes in more complex shapes than in spare parts. Averaging the intervals then hides important information. For example, if your product becomes obsolete halfway through the sample and the probability of occurrence drops from 0.5 to 0.1, ADI will be around 3.3. That corresponds to a probability of 0.3: neither the 0.5 you had nor the 0.1 you have now. It is like judging a hospital by the average temperature of all its patients.

So, ADI is a fine measure for its original purpose. But time has passed, use cases have changed, and nowadays instead of averaging the intervals, I recommend looking at how the probability of occurrence evolves over time, for example, by smoothing it (see the image in the post) or with a model that has a time-varying probability (like iETS). A plot of demand occurrence over time tells you more than ADI ever will.

We have a separate session on intermittent demand in our Demand Forecasting Principles course, where we cover this and other topics in detail.

Message What the Average Demand Interval (ADI) hides first appeared on OpenForecast.

Back in 1972, John Croston, who worked on stock control systems relying on Simple Exponential Smoothing (SES), noticed that SES forecasts were biased exactly when ordering decisions were made: right after a demand occurred. The following image shows why.

Intermittent demand consists mostly of periods with no sales, and SES treats each of them as a sale of size zero rather than as the absence of a sale. So, its forecast goes down with every zero, as if people wanted the product less and less. Then a customer buys 5 units, and SES immediately becomes optimistic. But in intermittent demand, sales rarely happen in consecutive periods, so this optimism leads to wrong ordering decisions.

The problem is not SES itself, but that it does not distinguish demand occurrence from demand sizes. Croston split the series into two parts (see the main image of the post): demand intervals (the number of periods between non-zero demands) and demand sizes (the non-zero values). He applied SES to each of them separately, and defined the final forecast as the ratio of the smoothed sizes to the smoothed intervals, i.e. the expected demand per period. Both parts are updated only when a demand happens, so the method does not change its mind with every zero, giving more stable forecasts and more reasonable ordering decisions.

Nowadays, we have many great forecasting approaches, from simple ones like TSB to more advanced ones like LightGBM. Croston’s method itself turned out to be biased (Syntetos & Boylan proposed a correction for that). Still, it works fine for manufacturers of large expensive items, with low volumes and long gaps between orders, where a stable forecast matters more than a quick reaction. I am not a big fan of the method, but I really like Croston’s original idea: split intermittent demand into demand occurrence and demand sizes. More than half a century later, this idea still works. My iETS paper used it, outperforming other simple forecasting methods on the M5 data. Türkmen et al. (2021) used the same idea with deep neural networks. The paper with Anna Sroginis showed that it also works with LightGBM, leading to substantial accuracy improvements that translate into inventory savings. So, even if you dislike Croston’s method, I bet you can improve your approach by using his original idea.

We will cover intermittent demand and the methods built on Croston’s idea in our Demand Forecasting Principles course in November.

Message The main idea behind the Croston’s method first appeared on OpenForecast.

First, why would we want non-equal weights in the first place? Demand evolves over time, and recent sales might reflect its current level better than the sales we had several years ago. So, it is only natural to give higher weights to recent observations and lower ones to older ones. Can we do that in a simple and principled way? Yes, and there is a forecasting method that does exactly that!

I’m talking about Simple Exponential Smoothing (SES), proposed by Robert Goodell Brown back in 1956 and independently by Charles Holt in 1957. It is yet another simple forecasting method, suitable for data without trend, seasonality or features. Mathematically, it is written like this:

\begin{equation}
F_{t+1} = \alpha A_{t} + (1−\alpha) F_{t}
\end{equation}

where \(F_t\) is the one-step-ahead forecast, \(A_t\) is the actual value, \(t\) is the time index, and \( \alpha \) is the smoothing parameter. Roughly speaking, \( \alpha \) defines how the weight is split between the most recent actual value and the previous forecast. If \( \alpha=0 \), the forecast ignores all new information and stays at its starting value, which could be, for example, the global average. With \( \alpha=1 \), it ignores all previous forecasts and becomes Naïve. The nice thing about SES is that \( \alpha \) also regulates how the weights are distributed over time, because the method can be rewritten as (see derivations here):

\begin{equation}
F_{t+1} = \alpha A_{t} + \alpha (1−\alpha) A_{t−1} + \alpha (1−\alpha)^2 A_{t−2} + \alpha (1−\alpha)^3 A_{t−3} + …
\end{equation}

Take \(\alpha=0.5\). The most recent observation gets the weight of 0.5, the one before it 0.5 × 0.5 = 0.25, the one before that 0.5 × 0.5² = 0.125, and so on. The weights decay exponentially. So, instead of 12 parameters, we define just one, and it determines how fast the weights decay. The chart in this post compares the equal weights of SMA(12) with SES for \( \alpha=0.2 \) and \( \alpha=0.5 \): the closer α is to zero, the more evenly the weights are spread; the closer it is to one, the more weight goes to the most recent observation, and the faster the older ones are forgotten.

There are tons of papers discussing SES, its modifications and extensions. Gardner (1985 and 2006) is the best review on the topic. And finally, SES is a predecessor of ETS, and its mechanism is used in TBATS.

How do you choose α in practice, and what do you do when SES is not enough? This is what we discuss in our Demand Forecasting Principles course.

Message Simple Exponential Smoothing: one parameter instead of many weights first appeared on OpenForecast.

One of the widely used approaches was developed by Hyndman & Khandakar (2008). It is implemented in auto.arima in R (and in StatsForecast for Python). It mixes two frameworks: stationarity tests (KPSS or ADF) to select the order of differencing d, and then information criteria for p and q. The tests are needed because, in the conventional formulation, models with different d are estimated on different sample sizes, so their information criteria cannot be compared (I explained this here). Although the algorithm works well and showed good performance in several competitions, it mixes hypothesis testing with information theory. The former tests the differences between samples, but assumes that you can make a mistake (based on your significance level), while the latter simply selects the most plausible model given data.

AutoMSARIMA does the selection differently. Because it is formulated in state space, the sample stays the same for any d, so the algorithm can compare all orders directly via information criteria, including the differencing. This follows the approach of Svetunkov & Boylan (2020), which treats order selection as a component selection problem rather than hypothesis testing. No ADF test, no KPSS test, no pre-screening for stationarity. The IC decides everything. And because nothing in this logic is tied to a specific seasonal period, it extends naturally to multiple seasonalities: each seasonal lag simply adds its own components to the model, and the same criterion decides which of them to keep, with the seasonality remaining stochastic rather than fixed.

The search itself is stepwise: the algorithm inspects the ACF/PACF of the residuals to propose a candidate order, adds it, and keeps it only if the information criterion decreases. When done, it also tries several simple SARIMA models to check whether any of them beats the selected one. The algorithm is explained in detail here.

In Python:

from fcompdata import AirPassengers
from smooth import AutoMSARIMA

# Monthly data, automatic order selection up to SARIMA(3,2,3)(3,1,3)[12]
model = AutoMSARIMA(lags=[1, 12])
model.fit(AirPassengers.y)
print(model)

The same algorithm works for the multiple seasonal ARIMA. The only thing to specify explicitly is the maximum orders to check:

from fcompdata import taylor

model = AutoMSARIMA(
lags=[1, 48, 336],
orders={"ar": [3, 2, 2],
"i": [2, 1, 1],
"ma": [3, 2, 2]}
)
model.fit(taylor.y)

The selected model returns the same attributes and supports the same methods as ADAM.

Does it actually work? I benchmarked AutoMSARIMA against statsforecast AutoARIMA, skforecast and R’s auto.arima on 5,315 series from the M1, M3 and Tourism datasets. The honest summary: point accuracy is a tie – RMSSE differs in the third decimal place across all implementations. But smooth produces the best-calibrated prediction intervals of the pack and is 3–5 times faster than the alternatives. Here is the full notebook.

Try smooth now: pip install smooth
smooth wiki.

Message smooth for Python: Automatic order selection for MSARIMA first appeared on OpenForecast.

The idea behind the Simple Moving Average (SMA) is to take the average of the last few observations and use it as a forecast for the next several steps ahead. Very crude and very simple. In fact, it has Naive (from this post) as a special case if you take the average of one most recent observation. On the other hand, if you increase the order to include all the observations, you will end up with the Global Mean. And this simple method works quite well if you have level data, i.e. no apparent strong trend, no obvious seasonality, and no other important elements of structure.

The only thing that makes it a bit harder to use in practice is the choice of the order, i.e. the number of observations to average over. Unfortunately, there is no universal answer here. But people report that the order of 12 or 13 is fine for weekly data, although it’s not completely clear why. In the academic literature, Aris Syntetos & John Boylan (2005) found that SMA(13) performed quite well on intermittent demand data, which was unexpected given the nature of the data (lots of zeroes). And almost 10 years ago, Fotios Petropoulos and I proposed a model underlying SMA with automatic order selection. We showed that it outperforms other simple benchmarks on supply chain data.

There is also some evidence from the VN2 inventory competition by Nicolas Vandeput. The benchmark there was built around a 13-week moving average with a simple seasonal adjustment, and only 25 out of 180+ participants managed to beat it. Many sophisticated ML pipelines lost to a method that predates computers.

So, if you work, for example, in retail or in supply chain, SMA is a method to consider for your sanity-check pool of models. But don’t expect miracles from it! It is still a simple method that works for level time series. Use it as a stepping stone to find a better model that has more features.

Anyone else found SMA to be a strong contender? Leave a comment – it would be interesting to see how many of you have had the same experience.

And yes, we discuss it in our “Demand Forecasting Principles” training in more detail. The next one will be held online in November, with live sessions from 2pm to 4pm UK time. We still have a few places left, so register here.

Message The Menace of ML: Simple Moving Average first appeared on OpenForecast.

The standard practice when ETS is not enough to capture the dynamics of real demand is to either switch to ARIMA or apply it to the residuals of the model. But it is possible to join the two models in one by using the Single Source of Error (SSOE) state space framework and to estimate the parameters jointly. This reduces the bias in the parameter estimates, letting the two models work as a team rather than one cleaning up after the other.

In the SSOE form, ETS and ARIMA states are stacked one after another in a large vector. For example, an ETS(A,N,A)+AR(2) model has four states: level, seasonal, and two AR components — each updated independently but responding to the same shock. Structurally, this resembles fitting ETS and then modelling its residuals with ARIMA, but with a critical difference: the parameters are estimated jointly, removing the bias that accumulates from sequential fitting.

But this flexibility comes with constraints. Some ETS and ARIMA combinations are actually redundant — they describe the same process from two angles, producing infinitely many parameter combinations with identical fit. For example, ETS(A,N,N) and ARIMA(0,1,1) are equivalent models, and combining them creates an unidentifiable model with no stable solution. So, I have come up with a few practical guidelines, which you can read about in Section 9.4 of ADAM.

Practically speaking, this is all handled under the hood of the ADAM function in Python, so you don’t need to worry about it. Here is an example:

from fcompdata import AirPassengers
from smooth import ADAM

# ETS(A,A,N) + SARIMA(0,0,0)(1,1,1)[12]
model = ADAM(
    model="AAN",
    ar_order=[0, 1],
    i_order=[0, 1],
    ma_order=[0, 1],
    lags=[1, 12],
    h=12, holdout=True
)
model.fit(AirPassengers.y)

Note how the orders are specified: per lag, with the first element referring to lag 1 and the second to lag 12. The code above fits ETS(A,A,N) + SARIMA(0,0,0)(1,1,1)[12], delegating the seasonality to the ARIMA part. The advantage of doing this via ADAM is that the result can be compared with pure ETS or ARIMA directly using information criteria. And if you don’t want to choose the orders yourself, AutoADAM will select them for you.

But do you even need the full combination? Gardner (1985) and Taylor (2010) argued that adding AR(1) tends to improve the accuracy of ETS. So, maybe you can get away with the simple ETS+AR(1) model, which is easy to apply using ADAM.

When to use ETS vs ARIMA.
How to combine them.

Try it yourself: pip install smooth

Message smooth for Python: ETS+ARIMA first appeared on OpenForecast.

One of the definitions I see on the internet: “A measure of the degree to which something may be forecast with accuracy”. This definition is so disturbing that I cannot resist ranting about it. The thing that annoys me the most is the “with accuracy”, which is really arbitrary and can mean anything. Your Naive method produces a relative RMSE of 1.0. Is that accurate enough? Does this make the time series forecastable? What if ETS produces 1.2, while LightGBM delivers 0.8? Is the data forecastable now? And when do you say “this is fine”?

But let me breathe and take a step back. The thing is, any method of measuring forecastability has assumptions behind it. Here is my list of five (did I miss any?):

1. The models you use: each time series has its own characteristics, so you can only say that the data is “forecastable” given the set of models you have. The data with trend might look unforecastable for the model that only has the level component.
2. The features/indicators that are available: if you do not know when promotions happen, some observations will look unforecastable. It’s a similar argument to (1), but more about what you include in your model. If you spend more time on feature generation and transformation, then the series that looked very hard might become quite easily forecastable.
3. The statistic you focus on: this could be the conditional mean, the median, some quantile, or the whole predictive distribution. A model can do great in terms of point forecasts but very poorly when it comes to prediction intervals. Does that make the time series unforecastable? Which one do you care more about?
4. The forecast horizon: the longer it is, the less accurate the forecasts become. One-step-ahead forecasts are easy, multiple steps ahead are hard. Which one do you use to measure forecastability?
5. The error measures: I’ll just leave this here: https://openforecast.org/category/forecasting-theory/forecast-evaluation/

Do I hear someone mentioning the coefficient of variation (CoV) as a good measure of forecastability? This is, by the way, what is used in the conventional XYZ classification. Well, let’s check the five points above against it and see what it assumes: (1) the global mean is suitable and the variance is constant; (2) no features are available; (3) conditional mean as the statistic of interest; (4) 1-step-ahead point forecast; (5) Root Mean Squared Error (the core of the CoV). So, it is suitable for a very small set of time series. If you use it universally, you might decide that some time series with a very clear structure are not forecastable. And the same exercise works for the more sophisticated measures, entropy-based ones included: run them through the five questions and see what they silently assume.

And then we come to the final point: so what? Let’s say you split your data into the “forecastable/not” categories. What are you going to do with that? The intention behind this is legitimate — you cannot babysit ten thousand SKUs equally, so you want to know where to spend your time. But the label “unforecastable” answers the wrong question. The right question is why the series is hard to forecast: missing promotion information? Wrong model? Genuinely random demand? Each of these implies a different action. It might well be that your “unforecastable” time series just require more time and effort to become forecastable again.

We don’t teach “forecastability” in our Demand Forecasting Principles course — because, as you can see, I don’t believe in it. What we do teach is everything in the list above: models, their assumptions, features, horizons, and how to evaluate forecasts properly. See details about the next course here.

This is my personal view. Happy to hear what others have to say.

Message Five assumptions behind “forecastability” first appeared on OpenForecast.