One of the core ideas in statistics, which extends to many other fields including forecasting, is the concept of structure versus noise. You’ve probably heard of it, but it’s often overlooked by those without a strong quantitative background. So, let’s discuss.
The core of the idea is that any data consists of two fundamental parts:
- Structure, which can take various forms, and might include trend, seasonality, calendar effects, and the influence of external factors on demand (e.g., price changes, promotions etc).
- Noise, which is inherently unpredictable.
Structure can be captured using models or methods, and this is what produces the fitted values or point forecasts. Noise, on the other hand, is unpredictable – like not knowing exactly who will visit a store and when they’ll make a purchase.
For example, consider a local Lancaster pub that has a nice selection of beers. Their sales likely follow a pattern, such as higher sales on weekends or during special events like football matches. These patterns are the structure we can capture and forecast. However, the pub can’t anticipate when my friend Yves will visit me, and when we’ll go out for drinks. This element of uncertainty forms the noise – while it’s explainable from my perspective, it’s a mystery to the pub owner.
But as I said, the idea of structure vs. noise isn’t just relevant in demand forecasting; it applies in many other areas too. Take classification, for instance. When identifying mushrooms, you might not be able to tell for sure whether you’re looking at a Rosy Brittlegill or The Sickener without a microscope. While certain characteristics (like stem shape or cap colour) make up the structure, there’s always some randomness that can make one mushroom look like another. So, in classification, you can only say that it’s more likely that we have one type of mushroom rather than the other, and you need to consider the uncertainty around this choice (the modern approach to this is to use conformal prediction).
Furthermore, we as humans are very good at finding patterns in the noise. If you look at clouds and see a mushroom, it’s not a real mushroom, just a random arrangement of vapour. So when you work with the data, remember this feature and don’t fall into the trap of finding patterns that don’t actually exist. Be critical and avoid overfitting the noise.
As you can see, the concept of structure versus noise is fundamental and shows up in many contexts. In forecasting, our job all is to capture the structure somehow, filter out the noise so that we can then produce point forecasts (future structure) and prediction intervals (representing the size of uncertainty) to be able to make adequate decisions.