In order to help master students of Lancaster University Managemen Science department, I have developed a flow chart, that acts as a basic guide on what error measures to use in different circumstances. This is not a complete and far from perfect flow chart, and it assumes that the decision maker knows what intermittent demand … Read more
Previously we discussed how to measure accuracy of point forecasts and performance of prediction intervals in different cases. Now we look into the question how to tell the difference between competing forecasting approaches. Let’s imagine the situation, when we have four forecasting methods applied to 100 time series with accuracy measured in terms of RMSSE: … Read more
If you work in the field of statistics, analytics, data science or forecasting, then you probably have already noticed that some of the instruments that are used in your field are called “methods”, while the others are called “models”. The issue here is that the people, using these terms, usually know the distinction between them, … Read more
UPDATE: I have also written a short post on “The role of M competitions in forecasting“, which gives historical perspective and a brief overview of the main findings of the previous competitions. Some of you might have noticed that the guidelines for the M5 competition have finally been released. Those of you who have previously … Read more
UPDATE: Read more about intermittent demand in newer posts: Introduction to intermittent demand Do you really need SBC? Lumpy is a type of intermittent demand and the usefulness of the classification scheme; And there is more, check the Intermittent Demand category on the website. In one of the previous posts, we have discussed how to … Read more
Introduction Some people think that the main idea of forecasting is in predicting the future as accurately as possible. I have bad news for them. The main idea of forecasting is in decreasing the uncertainty. Think about it: any event that we want to predict has some systematic components \(\mu_t\), which could potentially be captured … Read more
Two years ago I have written a post “Naughty APEs and the quest for the holy grail“, where I have discussed why percentage-based error measures (such as MPE, MAPE, sMAPE) are not good for the task of forecasting performance evaluation. However, it seems to me that I did not explain the topic to the full … Read more
One of the basic things the students are taught in statistics classes is that the comparison of models using information criteria can only be done when the models have the same response variable. This means, for example, that when you have \(\log(y_t)\) and calculate AIC, then this value is not comparable with AIC from a … Read more
Today I want to tell you a story of naughty APEs and the quest for the holy grail in forecasting. The topic has already been known for a while in academia, but is widely ignored by practitioners. APE stands for Absolute Percentage Error and is one of the simplest error measures, which is supposed to … Read more
In the modern statistical literature there is a notion of “true model”, by which people usually mean some abstract mathematical model, presumably lying in the core of observed process. Roughly saying, it is implied that data we have has been generated by some big guy with a white beard sitting in mathematical clouds using some … Read more