How to measure the performance of forecasting models
“My amazing forecasting method has a lower MASE than any other method!” You’ve probably seen claims like this on social media or in papers. But have you ever thought about what it really means? Many forecasting experiments come to applying several approaches to a dataset, calculating error measures for each method per time series and … Read more
In order to help master students of Lancaster University Managemen Science department, I have developed a flow chart, that acts as a basic guide on what error measures to use in different circumstances. This is not a complete and far from perfect flow chart, and it assumes that the decision maker knows what intermittent demand … Read more
Previously we discussed how to measure accuracy of point forecasts and performance of prediction intervals in different cases. Now we look into the question how to tell the difference between competing forecasting approaches. Let’s imagine the situation, when we have four forecasting methods applied to 100 time series with accuracy measured in terms of RMSSE: … Read more
UPDATE: Read more about intermittent demand in newer posts: Introduction to intermittent demand Do you really need SBC? Lumpy is a type of intermittent demand and the usefulness of the classification scheme; And there is more, check the Intermittent Demand category on the website. In one of the previous posts, we have discussed how to … Read more
Introduction Some people think that the main idea of forecasting is in predicting the future as accurately as possible. I have bad news for them. The main idea of forecasting is in decreasing the uncertainty. Think about it: any event that we want to predict has some systematic components \(\mu_t\), which could potentially be captured … Read more
Two years ago I have written a post “Naughty APEs and the quest for the holy grail“, where I have discussed why percentage-based error measures (such as MPE, MAPE, sMAPE) are not good for the task of forecasting performance evaluation. However, it seems to me that I did not explain the topic to the full … Read more
Today I want to tell you a story of naughty APEs and the quest for the holy grail in forecasting. The topic has already been known for a while in academia, but is widely ignored by practitioners. APE stands for Absolute Percentage Error and is one of the simplest error measures, which is supposed to … Read more