There’s a large and fascinating area in time series analysis called “changepoint detection”. I hadn’t worked in this area before, but thanks to Rebecca Killick and Thomas Grundy, I contributed to the paper “Online Detection of Forecast Model Inadequacies Using Forecast Errors“, which has just been published in the Journal of Time Series Analysis.

DISCLAIMER: the image in the post is taken from the paper, Figure 6, showing the proportion of GRP A&E admissions, the forecast errors and two detectors.

Here’s a brief summary of what it’s about:

One of the common issues in forecasting is that there might be some serious changes in the data due to external factors (e.g. changes in consumer preferences). These changes are not always captured by the model, which can lead to reduced accuracy, increased variance, and ultimately to losses. The changepoint detection literature addresses this by trying to automatically identify such structural changes and alert analysts when intervention might be needed. This becomes especially useful when managing large numbers of time series, where visual inspection isn’t feasible.

However, most existing approaches either work directly on raw data or rely on a model, which makes their usefulness limited.

Tom Grundy and Rebecca Killick came up with a better idea: analysing forecast errors instead. They kindly invited me to join as a co-author (since I know a thing or two about forecasting). The result is an online changepoint detection mechanism that is more universal and can be applied to classical statistical forecasting models and potentially to machine learning approaches.

The paper is quite technical and includes theoretical derivations, showing that the proposed method substantially reduces detection delay compared to some conventional approaches. We also evaluated its performance with ARIMA and ETS models on simulated data and provided several examples with real time series, demonstrating how it works.

The final version of the paper is available here, while the pre-print is here.