2.3 How to choose appropriate error measure
While in general the selection of error measure should be dictated by the specific problem at hand, there are some guidelines that might be helpful in the process. I have summarised them in the flowchart in Figure 2.1.
The flowchart does not provide the excessive options, and is a simplification of the possible process. It does not discuss the quantile and interval measures in detail, as there are many more of them, and the idea is to list the most important ones. The aim of the flowchart is to provide a basic idea that the selection can be done based on:
- Number of time series under consideration. If there is several of them and you need to aggregate the error measure, then you need to use either scaled or relative ones. In case of just one time series, you do not need to scale the error measure;
- What specifically you want to measure: point forecasts, quantiles, prediction interval or something else;
- Whether the interpretability of the error measure is important or not. If not, then scaled measures similar to Hyndman and Koehler (2006) can be used. If yes, then the choice is between relative and scaled using mean measures;
- Whether the data is stationary or not. If it is then it is safe to use scaled measures similar to Petropoulos and Kourentzes (2015), because the division by in-sample mean would be meaningful. Otherwise you should either use Hyndman and Koehler (2006) scaling or relative measures;
- Whether the data is intermittent or not. If it is and you are interested in point forecasts, then you should use RMSE based measures - other measures might recommend zero forecast as the best one;
- Symmetry of distribution. If it is symmetric (which does not happen very often), then median will coincide with mean and geometric mean, and it would not be important, whether to use RMSE-, MAE- or RMSLE- based measure. In that case, just use MAE-based one;
- What you need (denoted as “What do you like?” in the flowchart). If you are interested in mean performance then use RMSE based measures. MAE is minimised by median, and RMSLE is minimised by geometric mean.
You can also download this flowchart in pdf format via this link.