Retail Demand Forecasting Software: How to Judge One

The accuracy claim is quoted at the grain that flatters it

Every vendor in this category will tell you their forecast is 90 something percent accurate. Most of them are telling the truth and the number is still useless.

Forecast accuracy is entirely a function of grain. Aggregate far enough and errors cancel. Chain-level weekly forecasts routinely hit 90 percent and cannot be used to order anything, because you do not place orders at chain level.

The grain that drives replenishment is store by SKU by day. At that grain, WMAPE of 30 to 60 percent is normal for slow-moving items and is not a sign of a bad system. It is a sign that forecasting a product that sells 0.4 units a day in one store is genuinely hard.

So the first question for any vendor is not how accurate. It is: at what grain, with what error metric, over what horizon.

Why WMAPE and not MAPE

Plain MAPE averages the percentage error of every item equally. A SKU that sold one unit and was forecast at two contributes a 100 percent error with the same weight as your best seller missing by 3 percent.

In retail, where most of the catalog is slow moving, that makes MAPE close to meaningless and easy to game in both directions.

WMAPE is total absolute error divided by total actual volume. It weights by the units actually at stake, so the number tracks the inventory and the dollars rather than the item count.

Ask for WMAPE. Ask for bias alongside it, because a forecast can have acceptable WMAPE and be systematically 8 percent high, which fills your back rooms while looking fine on an error metric.

How to run a bake-off a vendor cannot game

Hold out a period the vendor never sees. This is the whole design.

• Give every vendor identical history ending on the same date.
• Ask for forecasts for the following 13 weeks at store by SKU by day.
• Score them yourself against what actually happened, using WMAPE and bias.
• Enter your current process as one of the competitors.

Include a promotional period and a holiday in the holdout window. Those are where forecasting systems separate, and a holdout of quiet weeks separates nobody.

If a vendor asks to tune on the holdout, you have your answer about what their published accuracy numbers mean.

Entering your current process matters more than it sounds. A system that beats a planner's spreadsheet by four points on WMAPE is a different purchase from one that beats it by twenty, and plenty of evaluations never find out which one they are looking at.

The four cases that separate vendors

Ask specifically about these. General accuracy figures hide all of them.

New items. Every catalog has them constantly and they have no history. The answer should involve attribute-based analogues, and the vendor should be able to show you how the analogue is chosen and how quickly the system stops leaning on it.

Stockouts. If the system reads zero sales during an out-of-stock as genuine zero demand, it will learn that your best seller does not sell in that store and forecast accordingly. Ask how censored demand is handled. This is the single most important technical question in the evaluation.

Promotions. Lift, cannibalization of substitutes, and the post-promotion dip from pantry loading. A system that forecasts promotional lift without modeling the dip will overstock the two weeks after every event.

Fresh and short shelf life. Daily ordering cycles, waste cost on one side and lost sales on the other, and an optimal service level that is not 98 percent. Systems built for ambient center store often handle this poorly, and it is not visible in an aggregate accuracy number.

How much history you need

Two full years is the working answer, because it gives two observations of each seasonal peak and each holiday. One year is workable with caveats and effectively no ability to separate trend from seasonality.

What the history contains matters as much as how long it is. Three fields decide whether more years help:

• Which weeks were promoted, and what the mechanic was. Without it, promotional weeks pollute the base demand estimate permanently.
• Which days the item was out of stock. Without it, censored demand is indistinguishable from low demand.
• What price was actually charged. Without it, price response cannot be estimated.

If those three are missing, five years of history is not better than two. Check them before you scope anything.

Why forecasting projects fail

Rarely because the forecast was bad.

Nothing downstream changed. A better forecast feeding an ordering process with a manual override at every step produces the same orders you were already placing. If the plan does not include changing the ordering process, the forecast improvement has nowhere to go.

Inventory accuracy. A forecast is an input to a replenishment decision that also needs to know what is on hand. Improving one while the other is 90 percent accurate at SKU level moves very little.

Nobody owns the exceptions. Every forecasting system produces a list of items it is unsure about. If no named person works that list weekly, the system is running open loop and its accuracy will decay.

Ask every vendor what their customers changed operationally, not just what accuracy they achieved. If the answer is only a number, the reference call is worth making yourself.