One of the widely used approaches was developed by Hyndman & Khandakar (2008). It is implemented in auto.arima in R (and in StatsForecast for Python). It mixes two frameworks: stationarity tests (KPSS or ADF) to select the order of differencing d, and then information criteria for p and q. The tests are needed because, in the conventional formulation, models with different d are estimated on different sample sizes, so their information criteria cannot be compared (I explained this here). Although the algorithm works well and showed good performance in several competitions, it mixes hypothesis testing with information theory. The former tests the differences between samples, but assumes that you can make a mistake (based on your significance level), while the latter simply selects the most plausible model given data.

AutoMSARIMA does the selection differently. Because it is formulated in state space, the sample stays the same for any d, so the algorithm can compare all orders directly via information criteria, including the differencing. This follows the approach of Svetunkov & Boylan (2020), which treats order selection as a component selection problem rather than hypothesis testing. No ADF test, no KPSS test, no pre-screening for stationarity. The IC decides everything. And because nothing in this logic is tied to a specific seasonal period, it extends naturally to multiple seasonalities: each seasonal lag simply adds its own components to the model, and the same criterion decides which of them to keep, with the seasonality remaining stochastic rather than fixed.

The search itself is stepwise: the algorithm inspects the ACF/PACF of the residuals to propose a candidate order, adds it, and keeps it only if the information criterion decreases. When done, it also tries several simple SARIMA models to check whether any of them beats the selected one. The algorithm is explained in detail here.

In Python:

from fcompdata import AirPassengers
from smooth import AutoMSARIMA

# Monthly data, automatic order selection up to SARIMA(3,2,3)(3,1,3)[12]
model = AutoMSARIMA(lags=[1, 12])
model.fit(AirPassengers.y)
print(model)

The same algorithm works for the multiple seasonal ARIMA. The only thing to specify explicitly is the maximum orders to check:

from fcompdata import taylor

model = AutoMSARIMA(
lags=[1, 48, 336],
orders={"ar": [3, 2, 2],
"i": [2, 1, 1],
"ma": [3, 2, 2]}
)
model.fit(taylor.y)

The selected model returns the same attributes and supports the same methods as ADAM.

Does it actually work? I benchmarked AutoMSARIMA against statsforecast AutoARIMA, skforecast and R’s auto.arima on 5,315 series from the M1, M3 and Tourism datasets. The honest summary: point accuracy is a tie – RMSSE differs in the third decimal place across all implementations. But smooth produces the best-calibrated prediction intervals of the pack and is 3–5 times faster than the alternatives. Here is the full notebook.

Try smooth now: pip install smooth
smooth wiki.

Message smooth for Python: Automatic order selection for MSARIMA first appeared on OpenForecast.

The standard practice when ETS is not enough to capture the dynamics of real demand is to either switch to ARIMA or apply it to the residuals of the model. But it is possible to join the two models in one by using the Single Source of Error (SSOE) state space framework and to estimate the parameters jointly. This reduces the bias in the parameter estimates, letting the two models work as a team rather than one cleaning up after the other.

In the SSOE form, ETS and ARIMA states are stacked one after another in a large vector. For example, an ETS(A,N,A)+AR(2) model has four states: level, seasonal, and two AR components — each updated independently but responding to the same shock. Structurally, this resembles fitting ETS and then modelling its residuals with ARIMA, but with a critical difference: the parameters are estimated jointly, removing the bias that accumulates from sequential fitting.

But this flexibility comes with constraints. Some ETS and ARIMA combinations are actually redundant — they describe the same process from two angles, producing infinitely many parameter combinations with identical fit. For example, ETS(A,N,N) and ARIMA(0,1,1) are equivalent models, and combining them creates an unidentifiable model with no stable solution. So, I have come up with a few practical guidelines, which you can read about in Section 9.4 of ADAM.

Practically speaking, this is all handled under the hood of the ADAM function in Python, so you don’t need to worry about it. Here is an example:

from fcompdata import AirPassengers
from smooth import ADAM

# ETS(A,A,N) + SARIMA(0,0,0)(1,1,1)[12]
model = ADAM(
    model="AAN",
    ar_order=[0, 1],
    i_order=[0, 1],
    ma_order=[0, 1],
    lags=[1, 12],
    h=12, holdout=True
)
model.fit(AirPassengers.y)

Note how the orders are specified: per lag, with the first element referring to lag 1 and the second to lag 12. The code above fits ETS(A,A,N) + SARIMA(0,0,0)(1,1,1)[12], delegating the seasonality to the ARIMA part. The advantage of doing this via ADAM is that the result can be compared with pure ETS or ARIMA directly using information criteria. And if you don’t want to choose the orders yourself, AutoADAM will select them for you.

But do you even need the full combination? Gardner (1985) and Taylor (2010) argued that adding AR(1) tends to improve the accuracy of ETS. So, maybe you can get away with the simple ETS+AR(1) model, which is easy to apply using ADAM.

When to use ETS vs ARIMA.
How to combine them.

Try it yourself: pip install smooth

Message smooth for Python: ETS+ARIMA first appeared on OpenForecast.

In the conventional ARIMA, taking differences is treated as a pre-processing step, where we switch from, for example, the sales of a product to the sales increase/decrease. If you had a sample of 100 observations, after first differences you are left with 99 – the first observation has nothing to be subtracted from. So yes, with the conventional approach, ARIMA(1,0,1) and ARIMA(1,1,1) are estimated on different sample sizes, and their information criteria are not directly comparable.

But! MSARIMA (and ADAM in general) is a state space model, and this pre-processing step is not needed. The differences are embedded in the model as additional components: ARIMA(0,1,1) has one, ARIMA(0,2,1) has two. The sample stays the same; what changes is the number of estimated parameters – each new component needs its initial value, and the information criterion penalises that. So the comparison stays fair: same sample, same likelihood basis, different number of parameters. I have several examples of how ARIMA is formulated in the ADAM SSOE framework.

What this implies: any ARIMA model of any orders (non-seasonal, seasonal, or multiseasonal) can be compared with other ARIMA models directly via information criteria, on the same sample of data.

This is not a new idea, by the way. I used this approach in the ARIMA algorithm I developed for the DemandWorks company (now part of Netstock) more than ten years ago, and John Boylan and I later built the State Space ARIMA on the same principle (the paper).

So, yes: if you use the conventional ARIMA (auto_arima from pmdarima, ARIMA from aeon or statsforecast in Python; stats or forecast implementations in R), you NEED to make sure that you compare models with the same order of differencing. But if you use the smooth implementation, you don’t need to worry – everything is already taken care of for you. This comes as a bonus from the specific formulation of the model.

Message On differencing of ARIMA in state space first appeared on OpenForecast.

The conventional implementations (statsmodels or pmdarima in Python, stats in R) do their job well: they estimate SARIMA via likelihood, select orders automatically, and produce sensible forecasts for series with trend and one seasonality. Monthly or quarterly demand data is their home ground. The problems start when the data has more than one cycle: the classical SARIMA formulation has a slot for exactly one seasonal lag, so for the half-hourly electricity demand you would have to pick either the hour-of-day or the day-of-week cycle and discard the other.

MSARIMA (Multiple Seasonal ARIMA) solves this by reformulating ARIMA in the Single Source of Error (SSOE) state space form. Each AR, I, and MA element becomes a state in the model, so nothing restricts you to one seasonal lag: you can have as many as you want. In the smooth package, orders are specified per lag, matched to a lags list. Here is an example on the classical taylor series (half-hourly electricity demand in England and Wales), with two seasonal cycles:

from fcompdata import taylor
from smooth import MSARIMA

# MSARIMA(3,0,1)(0,1,1)[48](0,1,1)[336] model = MSARIMA(
orders={"ar": [3, 0, 0], "i": [0, 1, 1], "ma": [1, 1, 1]},
lags=[1, 48, 336],
h=336, holdout=True
)
model.fit(taylor.y)
model.predict(h=336, interval="prediction", level=0.95)

In Python, it takes only 1.5 seconds for the function to fit the model to the data and estimate its parameters, which is fast for a double-seasonal model fit to roughly four thousand observations:

But the smooth implementation brings several additional practical benefits.

First, model.fit() accepts a matrix of external regressors via xreg, estimated jointly with the ARIMA components – no separate pre-filtering step. The logic and the code are identical to ETSX, which I covered earlier.

Second, you can swap the loss: loss=”GTMSE” or “MSEh” optimises the model directly on multistep errors, with the same shrinkage mechanism I discussed in the post on multistep losses.

Third, you don’t need to stick with the Gaussian distribution – you can choose other ones if you think that, for example, Laplace is more suitable for the data (this post).

And there is one more thing. Because MSARIMA now lives in the same state space framework as ETS, the two models can be combined into one and compared with each other directly via information criteria. But that deserves a post of its own, so stay tuned.

Read Chapter 9 on ADAM ARIMA.
Or check out the documentation in smooth wiki.

An why not try it yourself? pip install smooth

Message smooth in python: Multiple Seasonal ARIMA first appeared on OpenForecast.

When Box and Jenkins wrote their book back in the 1960s, it was a very different era: computers were massive, and people worked with punch cards. To make their approach viable, Box and Jenkins developed a methodology for selecting the appropriate orders of AR and MA based on the values of the autocorrelation and partial autocorrelation functions (ACF and PACF, respectively). Their idea was that if an ARMA process generates a specific ACF/PACF pattern, then it could be identified by analysing those functions in the data. At the time, it wasn’t feasible to do cross-validation or rolling origin evaluation, and even using information criteria for model selection was a challenge. So, the Box–Jenkins approach was a sensible option, producing adequate results with limited computational resources, and was considered state of the art.

Unfortunately, as the M1 competition later showed (see my earlier post), the methodology didn’t work well in practice. Simpler methods that didn’t rely on rigorous model selection actually performed better. But in fact, the winning model in the competition was ARARMA by Emanuel Parzen (https://doi.org/10.1002/for.3980010108). His idea was to make the series stationary by applying a low-order, non-stationary AR to the data, then extract residuals and select appropriate ARMA orders using AIC. Parzen ignored the Box–Jenkins methodology entirely – he didn’t analyse ACF or PACF and instead relied fully on automated selection. And it worked!

So why didn’t the Box–Jenkins methodology perform as expected? In my monograph Forecasting and Analytics with ADAM, I use the following example to explain the main issue: “All birds have wings. Sarah has wings. Thus, Sarah is a bird.” But Sarah, as shown in the image attached to this post, is a butterfly.

The fundamental issue with the Box–Jenkins methodology lies in its logic: if a process generates a specific ACF/PACF, that doesn’t mean that an observed ACF/PACF must come from that process. Many ARMA and even non-ARMA processes can generate exactly the same autocorrelation structure.

Further developments in ARIMA modelling have shown that ACF and PACF can only be used as general guidelines for order selection. To assess model performance properly, we need other tools. All modern approaches rely on information criteria for ARIMA order selection, and they consistently perform well in forecasting competitions. For example, Hyndman & Khandakar (2008) use AIC for ARMA order selection, while Svetunkov & Boylan (2020) apply AIC after reformulating ARIMA in a state space form. The former is implemented in the forecast package in R and the StatsForecast library in Python (thanks to Nixtla and Azul Garza); the latter is available in the smooth package in R. I also discuss another ARIMA order selection approach in Section 15.2 of my book.

Long story short: don’t use the Box–Jenkins methodology for order selection. Use more modern tools, such as information criteria.

P.S. See also my early post on ARIMA, discussing what is wrong with it.

Message Fundamental Flaw of the Box-Jenkins Methodology first appeared on OpenForecast.

One of the important steps in forecasting with statistical models is identifying the existing structure. In the case of ETS, it comes to selecting trend/seasonal components, while for ARIMA, it’s about order selection. In R, several functions automatically handle this based on information criteria (Hyndman & Khandakar, 2006; Svetunkov & Boylan (2017); Chapter 15 of ADAM). I decided to investigate how this mechanism works.

I generated data from the Normal distribution with a fixed mean of 5000 and a standard deviation of 50. Then, I asked ETS and ARIMA (from the forecast package in R) to automatically select the appropriate model for each of 1000 time series. Here is the R code for this simple experiment:

# Set random seed for reproducibility
set.seed(41, kind="L'Ecuyer-CMRG")
# Number of iterations
nsim <- 1000
# Number of observations
obsAll <- 120
# Generate data from N(5000, 50)
rnorm(nsim*obsAll, 5000, 50) |>
  matrix(obsAll, nsim) |>
  ts(frequency=12) -> x

# Load forecast package
library(forecast)
# Load doMC for parallel calculations
# doMC is only available on Linux and Max
# Use library(doParallel) on Windows
library(doMC)
registerDoMC(detectCores())

# A loop for ARIMA, recording the orders
matArima <- foreach(i=1:nsim, .combine=cbind, .packages=c("forecast")) %dopar% {
    testModel <- auto.arima(x[,i])
    # The element number 5 is just m, period of seasonality
    return(c(testModel$arma[-5],(!is.na(testModel$coef["drift"]))*1))
}
rownames(matArima) <- c("AR","MA","SAR","SMA","I","SI","Drift")

# A loop for ETS, recording the model types
matEts <- foreach(i=1:nsim, .combine=cbind, .packages=c("forecast")) %dopar% {
    testModel <- ets(x[,i], allow.multiplicative.trend=TRUE)
    return(testModel[13]$method)
}

The findings of this experiment are summarised using the following chunk of the R code:

#### Auto ARIMA ####
# Non-seasonal ARIMA elements
mean(apply(matArima[c("AR","MA","I","Drift"),]!=0, 2, any))
# Seasonal ARIMA elements
mean(apply(matArima[c("SAR","SMA","SI"),]!=0, 2, any))

#### ETS ####
# Trend in ETS
mean(substr(matEts,7,7)!="N")
# Seasonality in ETS
mean(substr(matEts,nchar(matEts)-1,nchar(matEts)-1)!="N")

I summarised them in the following table:

So, ARIMA detected some structure (had non-zero orders) in almost 40% of all time series, even though the data was designed to have no structure (just white noise). It also captured non-seasonal orders in a quarter of the series and identified seasonality in 18% of them. ETS performed better (only 0.2% of seasonal models identified on the white noise), but still captured trends in 2.3% of cases.

Does this simple experiment suggest that ARIMA is a bad model and ETS is a good one? No, it does not. It simply demonstrates that ARIMA tends to overfit the data if allowed to select whatever it wants. How can we fix that?

My solution: restrict the pool of ARIMA models to check, preventing it from going crazy. My personal pool includes ARIMA(0,1,1), (1,1,2), (0,2,2), along with the seasonal orders of (0,1,1), (1,1,2), and (0,2,2), and combinations between them. This approach is motivated by the connection between ARIMA and ETS. Additionally, we can check whether the addition of AR/MA orders detected by ACF/PACF analysis of the best model reduces the AICc. If not, they shouldn't be included.

This algorithm can be written in the following simple function that uses msarima() function from the smooth package in R (note that the reason why this function is used is because all ARIMA models implemented in the function are directly comparable via information criteria):

arimaCompact <- function(y, lags=c(1,frequency(y)), ic=c("AICc","AIC","BIC","BICc"), ...){

    # Start measuring the time of calculations
    startTime <- Sys.time();

    # If there are no lags for the basic components, correct this.
    if(sum(lags==1)==0){
        lags <- c(1,lags);
    }

    orderLength <- length(lags);
    ic <- match.arg(ic);
    IC <- switch(ic,
                 "AIC"=AIC,
                 "AICc"=AICc,
                 "BIC"=BIC,
                 "BICc"=BICc);

    # We consider the following list of models:
    # ARIMA(0,1,1), (1,1,2), (0,2,2),
    # ARIMA(0,0,0)+c, ARIMA(0,1,1)+c,
    # seasonal orders (0,1,1), (1,1,2), (0,2,2)
    # And all combinations between seasonal and non-seasonal parts
    # 
    # Encode all non-seasonal parts
    nNonSeasonal <- 5
    arimaNonSeasonal <- matrix(c(0,1,1,0, 1,1,2,0, 0,2,2,0, 0,0,0,1, 0,1,1,1), nNonSeasonal,4,
                               dimnames=list(NULL, c("ar","i","ma","const")), byrow=TRUE)
    # Encode all seasonal parts ()
    nSeasonal <- 4
    arimaSeasonal <- matrix(c(0,0,0, 0,1,1, 1,1,2, 0,2,2), nSeasonal,3,
                               dimnames=list(NULL, c("sar","si","sma")), byrow=TRUE)

    # Check all the models in the pool
    testModels <- vector("list", nSeasonal*nNonSeasonal);
    m <- 1;
    for(i in 1:nSeasonal){
        for(j in 1:nNonSeasonal){
            testModels[[m]] <- msarima(y, orders=list(ar=c(arimaNonSeasonal[j,1],arimaSeasonal[i,1]),
                                                      i=c(arimaNonSeasonal[j,2],arimaSeasonal[i,2]),
                                                      ma=c(arimaNonSeasonal[j,3],arimaSeasonal[i,3])),
                                       constant=arimaNonSeasonal[j,4]==1, lags=lags, ...);
            m[] <- m+1;
        }
    }

    # Find the best one
    m <- which.min(sapply(testModels, IC));
    # Amend computational time
    testModels[[m]]$timeElapsed <- Sys.time()-startTime;

    return(testModels[[m]]);
}

Additionally, we can check whether the addition of AR/MA orders detected by ACF/PACF analysis of the best model reduces the AICc. If not, they shouldn't be included. I have not added that part in the code above. Still, this algorithm brings some improvements:

#### Load the smooth package
library(smooth)

# A loop for the compact ARIMA, recording the orders
matArimaCompact <- foreach(i=1:nsim, .packages=c("smooth")) %dopar% {
    testModel <- arimaCompact(x[,i])
    return(orders(testModel))
}

#### Auto MSARIMA from smooth ####
# Non-seasonal ARIMA elements
mean(sapply(sapply(matArimaCompact, "[[", "ar"), function(x){x[1]!=0}) |
  sapply(sapply(matArimaCompact, "[[", "i"), function(x){x[1]!=0}) |
  sapply(sapply(matArimaCompact, "[[", "ma"), function(x){x[1]!=0}))

# Seasonal ARIMA elements
mean(sapply(sapply(matArimaSmooth, "[[", "ar"), function(x){length(x)==2 && (x[2]!=0)}) |
  sapply(sapply(matArimaSmooth, "[[", "i"), function(x){length(x)==2 && (x[2]!=0)}) |
  sapply(sapply(matArimaSmooth, "[[", "ma"), function(x){length(x)==2 && (x[2]!=0)}))

In my case, it resulted in the following:

As we see, when we impose restrictions on order selection in ARIMA, it avoids fitting seasonal models to non-seasonal data. While it still makes minor mistakes in terms of non-seasonal structure, it's nothing compared to the conventional approach. What about accuracy? I don't know. I'll have to write another post on this :).

Note that the models were applied to samples of 120 observations, which is considered "small" in statistics, while in real life is sometimes a luxury to have...

Message Detecting patterns in white noise first appeared on OpenForecast.

ARIMA has been a standard forecasting model in statistics for ages. It gained popularity with the famous Box & Jenkins (1970) book and was considered the best forecasting model by statisticians for a couple of decades without any strong evidence to support this.

It represents one of the two fundamental approaches to time series modeling (the second being the state space approach): it captures the relation between the variable and itself in the past. This has a great rationale in technical areas. For example, the quantity of CO2 in a furnace at this moment in time will depend on the quantity of CO2 five minutes ago. Such processes can be efficiently modeled and then forecasted using ARIMA. In demand forecasting, making sense of ARIMA is more challenging: it is hard to argue that the demand for shoes on Monday can impact the demand on Tuesday. So, when we apply ARIMA to such data, we sort of rely on a spurious relation. Still, demand data often exhibits autocorrelations, and ARIMA has been used efficiently in that context.

Over the years, ARIMA did not perform well in different competitions (see my post about that), but this was mainly due to the wrong assumptions in Box-Jenkins methodology, not because the model has been fundamentally bad. After Hyndman & Khandakar (2006) implemented their version with automatic order selection based on information criteria, ARIMA has started producing much more accurate forecasts.

But if I were to summarize what the problem with the model is, I would outline these points:

1. It is hard to explain ARIMA to people who are not comfortable with statistics. Here is an example of how seasonal ARIMA(1,0,1)(1,0,1)_4 is written mathematically:
   \begin{equation*}
   y_t (1 -\phi_{4,1} B^4)(1 -\phi_{1} B) = \epsilon_t (1 + \theta_{4,1} B^4) (1 + \theta_{1} B).
   \end{equation*}
   Good luck explaining this to a demand planner who does not know mathematics.
2. It is hard to estimate, especially for models with seasonality. It is typically estimated using some numeric optimisation, and reaching the maximum likelihood (or a global minimum of a loss function) is not guaranteed.
3. It is hard to select the appropriate order of the model, as there can be thousands of potential models to choose from. Yes, there are heuristic approaches that allow simplifying the problem and selecting a reasonable model (e.g. Hyndman & Khandakar, 2006; or Svetunkov & Boylan, 2017), but they do not guarantee that you will get the best possible model.

Nonetheless, ARIMA is a strong contender that can outperform other models if implemented well. Furthermore, it has become one of the standard forecasting benchmarks in forecasting-related experiments. So, if you are a data scientist comfortable with mathematics and want to see how your machine learning approach performs, you should consider including ARIMA as a benchmark.

P.S. Check out a post by Nicolas Vandeput on LinkedIn – he had a discussion about ARIMA and raised good points as well.

Message What’s wrong with ARIMA? first appeared on OpenForecast.

Journal: Computational Statistics

Abstract: Many modern statistical models are used for both insight and prediction when applied to data. When models are used for prediction one should optimise parameters through a prediction error loss function. Estimation methods based on multiple steps ahead forecast errors have been shown to lead to more robust and less biased estimates of parameters. However, a plausible explanation of why this is the case is lacking. In this paper, we provide this explanation, showing that the main benefit of these estimators is in a shrinkage effect, happening in univariate models naturally. However, this can introduce a series of limitations, due to overly aggressive shrinkage. We discuss the predictive likelihoods related to the multistep estimators and demonstrate what their usage implies to time series models. To overcome the limitations of the existing multiple steps estimators, we propose the Geometric Trace Mean Squared Error, demonstrating its advantages. We conduct a simulation experiment showing how the estimators behave with different sample sizes and forecast horizons. Finally, we carry out an empirical evaluation on real data, demonstrating the performance and advantages of the estimators. Given that the underlying process to be modelled is often unknown, we conclude that the shrinkage achieved by the GTMSE is a competitive alternative to conventional ones.

DOI: 10.1007/s00180-023-01377-x.

Working paper.

About the paper

DISCLAIMER 1: To better understand what I am talking about in this section, I would recommend you to have a look at the ADAM monograph, and specifically at the Chapter 11. In fact, Section 11.3 is based on this paper.

DISCLAIMER 2: All the discussions in the paper only apply to pure additive models. If you are interested in multiplicative or mixed ETS models, you’ll have to wait another seven years for another paper on this topic to get written and published.

Introduction

There are lots of ways how dynamic models can be estimated. Some analysts prefer likelihood, some would stick with Least Squares (i.e. minimising MSE), while others would use advanced estimators like Huber’s loss or M-estimators. And sometimes, statisticians or machine learning experts would use multiple steps ahead estimators. For example, they would use a so-called “direct forecast” by fitting a model to the data, producing h-steps ahead in-sample point forecasts from the very first to the very last observation, then calculating the respective h-steps ahead forecast errors and (based on them) Mean Squared Error. Mathematically, this can be written as:

\begin{equation} \label{eq:hstepsMSE}
\mathrm{MSE}_h = \frac{1}{T-h} \sum_{t=1}^{T-h} e_{t+h|t}^2 ,
\end{equation}
where \(e_{t+h|t}\) is the h-steps ahead error for the point forecast produced from the observation \(t\), and \(T\) is the sample size.

In my final year of PhD, I have decided to analyse how different multistep loss functions work, to understand what happens with dynamic models, when these losses are minimised, and how this can help in efficient model estimation. Doing the literature review, I noticed that the claims about the multistep estimators are sometimes contradictory: some authors say that they are more efficient (i.e. estimates of parameters have lower variances) than the conventional estimators, some say that they are less efficient; some claim that they improve accuracy, while the others do not find any substantial improvements. Finally, I could not find a proper explanation of what happens with the dynamic models when the estimators are used. So, I’ve started my own investigation, together with Nikos Kourentzes and Rebecca Killick (who was my internal examiner and joined our team after my graduation).

Our investigation started with the single source of error model, then led us to predictive likelihoods and, after that – to the development of a couple of non-conventional estimators. As a result, the paper grew and became less focused than initially intended. In the end, it became 42 pages long and discussed several aspects of models estimation (making it a bit of a hodgepodge):

1. How multistep estimators regularise parameters of dynamic models;
2. That multistep forecast errors are always correlated when the models’ parameters are not zero;
3. What predictive likelihoods align with the multistep estimators (this is useful for a discussion of their statistical properties);
4. How General Predictive Likelihood encompasses all popular multistep estimators;
5. And that there is another estimator (namely GTMSE – Geometric Trace Mean Squared Error), which has good properties and has not been discussed in the literature before.

Because of the size of the paper and the spread of the topics throughout it, many reviewers ignored (1) – (4), focusing on (5) and thus rejecting the paper on the grounds that we propose a new estimator, but instead spend too much time discussing irrelevant topics. These types of comments were given to us by the editor of the Journal of the Royal Statistical Society: B and reviewers of Computational Statistics and Data Analysis. While we tried addressing this issue several times, given the size of the paper, we failed to fix it fully. The paper was rejected from both of these journals and ended up in Computational Statistics, where the editor gave us a chance to respond to the comments. We explained what the paper was really about and changed its focus to satisfy the reviewers, after which the paper was accepted.

So, what are the main findings of this paper?

How multistep estimators regularise parameters of dynamic models

Given that any dynamic model (such as ETS or ARIMA) can be represented in the Single Source of Error state space form, we showed that the application of multistep estimators leads to the inclusion of parameters of models in the loss function, leading to the regularisation. In ETS, this means that the smoothing parameters are shrunk to zero, with the shrinkage becoming stronger with the increase of the forecasting horizon relative to the sample size. This makes the models less stochastic and more conservative. Mathematically this becomes apparent if we express the conditional multistep variance in terms of smoothing parameters and one-step-ahead error variance. For example, for ETS(A,N,N) we have:

\begin{equation} \label{eq:hstepsMSEVariance}
\mathrm{MSE}_h \propto \hat{\sigma}_1^2 \left(1 +(h-1) \hat{\alpha} \right),
\end{equation}
where \( \hat{\alpha} \) is the smoothing parameter and \(\hat{\sigma}_1^2 \) is the one-step-ahead error variance. From the formula \eqref{eq:hstepsMSEVariance}, it becomes apparent that when we minimise MSE\(_h\), the estimated variance and the smoothing parameters will be minimised as well. This is how the shrinkage effect appears: we force \( \hat{\alpha} \) to become as close to zero as possible, and the strength of shrinkage is regulated by the forecasting horizon \( h \).

In the paper itself, we discuss this effect for several multistep estimators (the specific effect would be different between them) and several ETS and ARIMA models. While for ETS, it is easy to show how shrinkage works, for ARIMA, the situation is more complicated because the direction of shrinkage would change with the ARIMA orders. Still, what can be said clearly for any dynamic model is that the multistep estimators make them less stochastic and more conservative.

Multistep forecast errors are always correlated

This is a small finding, done in bypassing. It means that, for example, the forecast error two steps ahead is always correlated with the three steps ahead one. This does not depend on the autocorrelation of residuals or any violation of assumptions of the model but rather only on whether the parameters of the model are zero or not. This effect arises from the model rather than from the data. The only situation when the forecast errors will not be correlated is when the model is deterministic (e.g. linear trend). This has important practical implications because some forecasting techniques make explicit and unrealistic assumptions that these correlations are zero, which would impact the final forecasts.

Predictive likelihoods aligning with the multistep estimators

We showed that if a model assumes the Normal distribution, in the case of MSEh and MSCE (Mean Squared Cumulative Error), the distribution of the future values follows Normal as well. This means that there are predictive likelihood functions for these models, the maximum of which is achieved with the same set of parameters as the minimum of the multistep estimators. This has two implications:

1. These multistep estimators should be consistent and efficient, especially when the smoothing parameters are close to zero;
2. The predictive likelihoods can be used in the model selection via information criteria.

The first point also explains the contradiction in the literature: if the smoothing parameter in the population is close to zero, then the multistep estimators will give more efficient estimates than the conventional estimators; in the other case, it might be less efficient. We have not used the second point above, but it would be useful when the best model needs to be selected for the data, and an analyst wants to use information criteria. This is one of the potential ways for future research.

How General Predictive Likelihood (GPL) encompasses all popular multistep estimators

GPL arises when the joint distribution of 1 to h steps ahead forecast errors is considered. It will be Multivariate Normal if the model assumes normality. In the paper, we showed that the maximum of GPL coincides with the minimum of the so-called “Generalised Variance” – the determinant of the covariance matrix of forecast errors. This minimisation reduces variances for all the forecast errors (from 1 to h) and increases the covariances between them, making the multistep forecast errors look more similar. In the perfect case, when the model is correctly specified (no omitted or redundant variables, homoscedastic residuals etc), the maximum of GPL will coincide with the maximum of the conventional likelihood of the Normal distribution (see Section 11.1 of the ADAM monograph).

Accidentally, it can be shown that the existing estimators are just special cases of the GPL, but with some restrictions on the covariance matrix. I do not intend to show it here, the reader is encouraged to either read the paper or see the brief discussion in Subsection 11.3.5 of the ADAM monograph.

GTMSE – Geometric Trace Mean Squared Error

Finally, looking at the special cases of GPL, we have noticed that there is one which has not been discussed in the literature. We called it Geometric Trace Mean Squared Error (GTMSE) because of the logarithms in the formula:
\begin{equation} \label{eq:GTMSE}
\mathrm{GTMSE} = \sum_{j=1}^h \log \frac{1}{T-j} \sum_{t=1}^{T-j} e_{t+j|t}^2 .
\end{equation}
GTMSE imposes shrinkage on parameters similar to other estimators but does it more mildly because of the logarithms in the formula. In fact, what the logarithms do is make variances of all forecast errors similar to each other. As a result, when used, GTMSE does not focus on the larger variances as other methods do but minimises all of them simultaneously similarly.

Examples in R

The estimators discussed in the paper are all implemented in the functions of the smooth package in R, including adam(), es(), ssarima(), msarima() and ces(). In the example below, we will see how the shrinkage works for the ETS on the example of Box-Jenkins sales data (this is the example taken from ADAM, Subsection 11.3.7):

library(smooth)

adamETSAANBJ <- vector("list",6)
names(adamETSAANBJ) <- c("MSE","MSEh","TMSE","GTMSE","MSCE","GPL")
for(i in 1:length(adamETSAANBJ)){
    adamETSAANBJ[[i]] <- adam(BJsales, "AAN", h=10, holdout=TRUE,
                              loss=names(adamETSAANBJ)[i])
}

The ETS(A,A,N) model, applied to this data, has different estimates of smoothing parameters:

sapply(adamETSAANBJ,"[[","persistence") |>
	round(5)

          MSE MSEh TMSE   GTMSE MSCE GPL
alpha 1.00000    1    1 1.00000    1   1
beta  0.23915    0    0 0.14617    0   0

We can see how shrinkage shows itself in the case of the smoothing parameter \(\beta\), which is shrunk to zero by MSEh, TMSE, MSCE and GPL but left intact by MSE and shrunk a little bit in the case of GTMSE. These different estimates of parameters lead to different forecasting trajectories and prediction intervals, as can be shown visually:

par(mfcol=c(3,2), mar=c(2,2,4,1))
# Produce forecasts
lapply(adamETSAANBJ, forecast, h=10, interval="prediction") |>
# Plot forecasts
    lapply(function(x, ...) plot(x, ylim=c(200,280), main=x$model$loss))

This should result in the following plots:

Analysing the figure, it looks like the shrinkage of the smoothing parameter \(\beta\) is useful for this time series: the forecasts from ETS(A,A,N) estimated using MSEh, TMSE, MSCE and GPL look closer to the actual values than the ones from MSE and GTMSE. To assess their performance more precisely, we can extract error measures from the models:

sapply(adamETSAANBJ,"[[","accuracy") |>
	round(5)[c("ME","MSE"),]

         MSE    MSEh    TMSE    GTMSE    MSCE     GPL
ME   3.22900 1.06479 1.05233  3.44962 1.04604 0.95515
MSE 14.41862 2.89067 2.85880 16.26344 2.84288 2.62394

Alternatively, we can calculate error measures based on the produced forecasts and the measures() function from the greybox package:

lapply(adamETSAANBJ, forecast, h=10) |>
    sapply(function(x, ...) measures(holdout=x$model$holdout,
                                     forecast=x$mean,
                                     actual=actuals(x$model)))

A thing to note about the multistep estimators is that they are slower than the conventional ones because they require producing 1 to \( h \) steps ahead forecasts from every observation in-sample. In the case of the smooth functions, the time elapsed can be extracted from the models in the following way:

sapply(adamETSAANBJ, "[[", "timeElapsed")

In summary, the multistep estimators are potentially useful in forecasting and can produce models with more accurate forecasts. This happens because they impose shrinkage on the estimates of parameters, making models less stochastic and more inert. But their performance depends on each specific situation and the available data, so I would not recommend using them universally.

Message Multi-step Estimators and Shrinkage Effect in Time Series Models first appeared on OpenForecast.

New engines for ETS, MSARIMA and SMA

The first and one of the most important changes is the new engine for the ETS (Error-Trend-Seasonal exponential smoothing model), MSARIMA (Multiple Seasonal ARIMA) and SMA (Simple Moving Average), implemented respectively in es(), msarima() and sma() functions. The new engine was developed for adam() and the three models above can be considered as special cases of it. You can read more about ETS in ADAM monograph, starting from Chapter 4; MSARIMA is discussed in Chapter 9, while SMA is briefly discussed in Subsection 3.3.3.

The es() function now implements the ETS close to the conventional one, assuming that the error term follows normal distribution. It still supports explanatory variables (discussed in Chapter 10 of ADAM monograph) and advanced estimators (Chapter 11), and it has the same syntax as the previous version of the function had, but now acts as a wrapper for adam(). This means that it is now faster, more accurate and requires less memory than it used to. msarima() being a wrapper of adam() as well, is now also faster and more accurate than it used to be. But in addition to that both functions now support the methods that were developed for adam(), including vcov(), confint(), summary(), rmultistep(), reapply(), plot() and others. So, now you can do more thorough analysis and improve the models using all these advanced instruments (see, for example, Chapter 14 of ADAM).

The main reason why I moved the functions to the new engine was to clean up the code and remove the old chunks that were developed when I only started learning C++. A side effect, as you see, is that the functions have now been improved in a variety of ways.

And to be on the safe side, the old versions of the functions are still available in smooth under the names es_old(), msarima_old() and sma_old(). They will be removed from the package if it ever reaches the v.4.0.0.

New methods for ADAM

There are two new methods for adam() that can be used in a variety of cases. The first one is simulate(), which will generate data based on the estimated ADAM, whatever the original model is (e.g. mixture of ETS, ARIMA and regression on the data with multiple frequencies). Here is how it can be used:

adam(BJsales, "AAdN") |>
     simulate() |>
     plot()

which will produce a plot similar to the following:

This can be used for research, when a more controlled environment is needed. If you want to fine tune the parameters of ADAM before simulating the data, you can save the output in an object and amend its parameters. For example:

testModel <- adam(BJsales, "AAdN")
testModel$persistence <- c(0.5, 0.2)
simulate(testModel)

The second new method is the xtable() from the respective xtable package. It produces LaTeX version of the table from the summary of ADAM. Here is an example of a summary from ADAM ETS:

adam(BJsales, "AAdN") |>
     summary()

Model estimated using adam() function: ETS(AAdN)
Response variable: BJsales
Distribution used in the estimation: Normal
Loss function type: likelihood; Loss function value: 256.1516
Coefficients:
      Estimate Std. Error Lower 2.5% Upper 97.5%  
alpha   0.9514     0.1292     0.6960      1.0000 *
beta    0.3328     0.2040     0.0000      0.7358  
phi     0.8560     0.1671     0.5258      1.0000 *
level 203.2835     5.9968   191.4304    215.1289 *
trend  -2.6793     4.7705   -12.1084      6.7437  

Error standard deviation: 1.3623
Sample size: 150
Number of estimated parameters: 6
Number of degrees of freedom: 144
Information criteria:
     AIC     AICc      BIC     BICc 
524.3032 524.8907 542.3670 543.8387

As you can see in the output above, the function generates the confidence intervals for the parameters of the model, including the smoothing parameters, dampening parameter and the initial states. This summary can then be used to generate the LaTeX code for the main part of the table:

adam(BJsales, "AAdN") |>
     xtable()

which will looks something like this:

Other improvements

First, one of the major changes in smooth functions is the new backcasting mechanism for adam(), es() and msarima() (this is discussed in Section 11.4 of ADAM monograph). The main difference with the old one is that now it does not backcast the parameters for the explanatory variables and estimates them separately via optimisation. This feature appeared to be important for some of users who wanted to try MSARIMAX/ETSX (a model with explanatory variables) but wanted to use backcasting as the initialisation. These users then wanted to get a summary, analysing the uncertainty around the estimates of parameters for exogenous variables, but could not because the previous implementation would not estimate them explicitly. This is now available. Here is an example:

cbind(BJsales, BJsales.lead) |>
    adam(model="AAdN", initial="backcasting") |>
    summary()

Model estimated using adam() function: ETSX(AAdN)
Response variable: BJsales
Distribution used in the estimation: Normal
Loss function type: likelihood; Loss function value: 255.1935
Coefficients:
             Estimate Std. Error Lower 2.5% Upper 97.5%  
alpha          0.9724     0.1108     0.7534      1.0000 *
beta           0.2904     0.1368     0.0199      0.5607 *
phi            0.8798     0.0925     0.6970      1.0000 *
BJsales.lead   0.1662     0.2336    -0.2955      0.6276  

Error standard deviation: 1.3489
Sample size: 150
Number of estimated parameters: 5
Number of degrees of freedom: 145
Information criteria:
     AIC     AICc      BIC     BICc 
520.3870 520.8037 535.4402 536.4841

As you can see in the output above, the initial level and trend of the model are not reported, because they were estimated via backcasting. However, we get the value of the parameter BJsales.lead and the uncertainty around it. The old backcasting approach is now called "complete", implying that all values of the state vector are produce via backcasting.

Second, forecast.adam() now has a parameter scenarios, which when TRUE will return the simulated paths from the model. This only works when interval="simulated" and can be used for the analysis of possible forecast trajectories.

Third, the plot() method now can also produce ACF/PACF for the squared residuals for all smooth functions. This becomes useful if you suspect that your data has ARCH elements and want to see if they need to be modelled separately. This can also be done using adam() and sm() and is discussed in Chapter 17 of the monograph.

Finally, the sma() function now has the fast parameter, which when true will use a modified Ternary search for the best order based on information criteria. It might not give the global minimum, but it works much faster than the exhaustive search.

Conclusions

These are the main new features in the package. I feel that the main job in smooth is already done, and all I can do now is just tune the functions and improve the existing code. I want to move all the functions to the new engine and ditch the old one, but this requires much more time than I have. So, I don't expect to finish this any time soon, but I hope I'll get there someday. On the other hand, I'm not sure that spending much time on developing an R package is a wise idea, given that nowadays people tend to use Python. I would develop Python analogue of the smooth package, but currently I don't have the necessary expertise and time to do that. Besides, there already exist great libraries, such as tsforecast from nixtla and sktime. I am not sure that another library, implementing ETS and ARIMA is needed in Python. What do you think?

Message smooth v3.2.0: what’s new? first appeared on OpenForecast.

This talk was based on Chapter 9 of ADAM monograph, and more specifically on Section 9.4.

The slides of the presentation are available here.

Message ISF2022: How to make ETS work with ARIMA first appeared on OpenForecast.