Apparently, we need to talk about scaling of error measures because this is not as obvious as it seems. In forecasting literature, since early days of the area, there has been a general consensus that the forecast errors from the individual time series should not be analysed and aggregated as is. This is because you … Read more
What do you use for model selection? Do you select the best model based on its cross-validated performance, or do you use in-sample measures like AIC? If so, there is a way to improve your selection process further. JORS recently published the paper of Nikos Kourentzes and I based on a simple but powerful idea: … Read more
If someone tells you that method X solves all problems and is the best one ever, they are either lying intentionally or do not fully understand what they are talking about. There is no such thing as “the best approach for everything”. Let me explain. Consider two products sold by retailers: ice cream and bread. … Read more
I’m currently doing a literature review for one of my papers on intermittent demand forecasting with machine learning, and I’ve noticed a recurring fundamental mistake in several recently published papers, even in respectable peer-reviewed journals. The mistake? Using error measures based on the Mean Absolute Error (MAE). This is a crime against the humanity when … Read more
Another question my students sometimes ask is how to define the sizes for the training and test sets in a forecasting experiment. If you’ve done data mining or machine learning, you’re likely familiar with this concept. But when it comes to forecasting, there are a few nuances. Let’s discuss. First and foremost, in forecasting, the … Read more
Look at the image above. Which forecast seems more appropriate: the red straight line (1) or the purple wavy line (2)? Many demand planners might choose option 2, thinking it better captures the ups and downs. But, in many cases, the straight line is just fine. Here’s why. In a previous post on Structure vs. … Read more
I have summarised several posts on point forecasts evaluation in an article for the Foresight journal. Mike Gilliland, being the Editor-in-Chief of the journal, contributed to the paper a lot, making it read much smoother, but preferred not to be included as the co-author. This article was recently published in the issue 74 for Q3:2024. … Read more
So far, we’ve discussed forecasts evaluation, focusing on the precision of point forecasts. However, there are many other dimensions in the evaluation that can provide useful information about your model’s performance. One of them is bias, which we’ll explore today. Introduction But before that, why should we bother with bias? Research suggests that bias is … Read more
One question I received from my LinkedIn followers was how to evaluate forecast accuracy in practice. MAPE is wrong, but it is easy to use. In practice, we want something simple, informative and straightforward, but not all error measures are easy to calculate and interpret. What should we do? Here is my subjective view. Step … Read more
Frankly speaking, I didn’t see the point in discussing MAPE when I wrote recent posts on error measures. However, I’ve received several comments and messages from data scientists and demand planners asking for clarification. So, here it is. TL;DR: Avoid using MAPE! MAPE, or Mean Absolute Percentage Error, is a still-very-popular-in-practice error measure, which is … Read more