NHS has a helpful page with a set of questions you can ask your GP to ensure you receive the right treatment for your illness. Surprisingly, these questions can be applied in other fields as well. Here’s an example in applied forecasting, working with companies.
I’m not going to go through all of them, I picked several most useful. These are questions you can ask a consultant who is helping you (as a client). But they can also help you (as a consultant) deliver a better product for your client.
- What are the alternative approaches you could try? (Are there other ways to treat my condition?)
- What are the drawbacks of the proposed approach? (Are there any side effects or risks? If so, what are they?)
- What if we don’t change the process? (What will happen if I don’t have any treatment?)
- How will the new approach improve our processes? (How effective is this treatment?)
- Are there parts of the process that should be avoided? (Is there anything I should stop or avoid doing?)
- What else should we do? (Is there anything I can do to help myself?)
If a consultant presents only one approach and says it’s the best, they may have found something that works for them but not necessarily for you. Ask whether they’ve tried other approaches on your data. A fancy neural network might do a great job, but a gradient boosting method, or even a simple statistical one, could perform similarly. After all, there is no such thing as the best approach for everything. If a consultant says there are no alternatives, find a new one.
Every approach relies on assumptions. If they are violated, the approach may fail. Understanding when and why this might happen is crucial. For example, the method might assume an inappropriate distribution (e.g. Normal for intermittent demand), or require more data than the company can afford to store. It could be too time-consuming to use in practice, or assume that the end users know what they are doing. There are always assumptions and trade-offs. If a consultant says, “There are no drawbacks”, they’re either lying or don’t understand what they’re doing.
One of the main responsibilities of a professional consultant is to benchmark new approaches against the current one. This allows them to show exactly what would change and what improvements would it would bring. If they don’t do this, consider it a red flag.
At the very least, a consultant should demonstrate an improvement in accuracy. Even better, they should show how this translates into better decisions and costs reduction (e.g. via inventory simulations). Reducing the workload for demand planners can also be a significant benefit. In any case, ask – they should know.
A good consultant should know what can cause issues. e.g. if external information (promotions?) is used in the model, judgmentally adjusting forecasts may do more harm than good. And a good consultant should be able to explain what not to do.
It’s not enough to just deploy a new approach. End users must understand how to use it. It’s great if demand planners are familiar with forecasting, statistics, and feature engineering, but if not, they might abandon an accurate approach in favour of a simpler, less effective alternative. A consultant should be able to explain what the company can do to make the new approach practical and ensure it delivers value (e.g. train the team).
Any other questions you would ask?