Measuring forecast accuracy without fooling yourself

Why measuring forecast accuracy is harder than it looks

Every forecasting team reports an accuracy number. Very few can say what decision it should change. The gap between those two facts is where most of the value in this topic sits, and it is not a modelling problem: it is a measurement-design problem.

The trap is that forecast accuracy is not one quantity. It is at least three, they fail in different directions, and a process that reports only one will be blind to the failure it does not measure. A forecast can be highly accurate and still bankrupt a category. A forecast can look mediocre and be entirely fit for purpose. Both situations are common and neither is visible from a single percentage.

This page covers the metrics that matter, what each one hides, the grain question that quietly determines your score, and how to build a review that changes a decision rather than producing a number.

The three questions, and one metric each

Measuring forecast accuracy well starts with separating three distinct questions that get collapsed into one.

MAPE: the default, and its two failure modes

Mean absolute percentage error is the standard. For each item and period, take the absolute difference between forecast and actual, divide by actual, and average across items.

It has two well-known defects and both bite in retail.

It is undefined when actual is zero, and in store-and-item-level retail data a large share of cells are zero. A slow item at a small store genuinely sells nothing many weeks. Implementations handle this by dropping those rows, which silently excludes the part of the range where forecasting is hardest, or by substituting a small number, which produces enormous percentage errors that swamp the average.

It treats a unit as a unit regardless of size. Missing by 5 units on an item selling 10 is a 50% error. Missing by 500 on an item selling 5,000 is a 10% error. The second miss is a hundred times larger in units and looks better. Average those two and you get 30%, a number describing neither.

WMAPE: usually the right default

Weighted MAPE fixes the second problem by dividing total absolute error by total actual volume, rather than averaging percentages:

WMAPE = sum of absolute errors / sum of actuals

Applied to the example above: total absolute error is 505 units, total actual is 5,010 units, so WMAPE is about 10.1%. That figure describes the portfolio honestly, because it is dominated by the items that dominate the volume. It also sidesteps the divide-by-zero problem entirely, since the denominator is a sum across items rather than a per-item value.

For most retail and CPG applications, WMAPE is the better default and MAPE is the one people report out of habit.

Bias: the metric that actually predicts inventory

Bias is the signed version, and it is the one most reviews omit. Take the mean percentage error without the absolute value, so overs and unders cancel.

The distinction matters enormously and is easiest to see side by side. Two forecasters, twelve weeks, same MAPE:

Forecaster B has less than half the error of A on the headline metric and is the one causing the problem. A consistent 5% over-forecast never trips an exception, because no individual week looks wrong. It just adds cover relentlessly until someone notices the category is carrying eight weeks of stock and starts an overstock project that will not find the cause, because the cause is not in the overstock data. It is in the forecast, and it was visible the whole time to anyone measuring bias.

If you take one thing from this page: a small consistent bias is more expensive than a large random error.

The grain question determines your score

The single largest lever on a reported accuracy number is not the model. It is the level at which error is measured, and this is where accuracy reporting is most often, if unintentionally, self-serving.

Errors cancel as you aggregate. The same forecast measured at four levels:

None of these numbers is wrong and none is comparable to another. The forecast did not change between rows. A team quoting the top row while the replenishment system consumes the bottom row is describing a different forecast than the one actually in use.

The rule that keeps this honest: measure accuracy at the grain the decision is made at. If the output drives store-level ordering, a national figure is decoration. Two consequences follow. Store-item accuracy will look alarming compared to what leadership is used to hearing, and that is the correct number. And a forecast can be entirely adequate for one decision and useless for another, which means one accuracy target across the business is the wrong shape.

What a good score still hides

Three failures survive a strong accuracy number.

Error concentration. A 12% WMAPE across 4,000 items can mean every item is near 12%, or that 3,700 items are near 4% and 300 are near 60%. Those need completely different responses, and the average cannot distinguish them. Report the distribution, or at minimum the tail: what share of items, and what share of volume, sits above twice the portfolio error.

Promotional versus baseline error. Promoted weeks are where forecasting is hardest and where being wrong is most expensive, because the volumes are large and the inventory was bought specially. Blending promoted and non-promoted weeks into one number hides both. Score them separately, always. The first reorder after a promotion is a related and separately dangerous case, discussed in replenishment planning.

Error against a naive baseline. A 15% WMAPE means nothing in isolation. The question is 15% against what alternative. Compare every forecast against the naive benchmark of last period's actual, or last year's same period for seasonal items. A sophisticated model that does not beat "same as last week" has told you something important about either the model or the demand pattern, and plenty of production models quietly fail this test.

Accuracy is not the objective

The uncomfortable conclusion of everything above is that forecast accuracy is an intermediate measure, not a goal. Nobody is paid to be accurate. They are paid for service level at an acceptable inventory cost, and accuracy is one input to that.

This matters because the two can move in opposite directions. A team that improves WMAPE from 22% to 19% by getting better at high-volume items may have changed nothing about service, because high-volume items were never the ones going out of stock. A team that leaves WMAPE untouched but removes a 4% positive bias will reduce inventory measurably without the headline metric moving at all.

The translation from accuracy to money runs through two channels:

Those costs are asymmetric and the asymmetry differs by item. On a high-margin staple with a long shelf life, under-forecasting is much more expensive than over-forecasting, and the right safety stock is generous. On a perishable or a seasonal item with a hard end date, the asymmetry reverses hard. A single service-level target applied across the range ignores this and is wrong in both directions simultaneously.

The practical consequence for measurement: weight your error reporting by something economic. Ranking the error tail by units is better than ranking by percentage. Ranking it by margin dollars at risk is better still, and it usually reorders the list substantially, which is the point.

A review that changes a decision

The practical shape of a monthly forecast review that is worth holding:

1. WMAPE at decision grain, with last month for comparison.
2. Bias at the same grain, called out separately, with any persistent direction flagged by category rather than averaged away.
3. The tail: items above twice portfolio error, ranked by volume, not by error percentage. This is the worklist.
4. Promoted versus baseline split, never blended.
5. Versus naive, so the model is earning its keep.
6. One decision per review. Which parameter changes, which items move to a different method, which supplier's lead-time assumption gets remeasured.

The sixth item is what separates a review from a report. An accuracy meeting that ends without a changed parameter has measured something and improved nothing.

Three arguments you will have

Measuring forecast accuracy honestly tends to produce the same three objections, and each has a short answer worth having ready.

"That number is too high to be right." It is high because it is measured at store-item grain, and the number people are used to hearing was measured national-monthly. Both describe the same forecast. Show the aggregation table and the argument usually ends there, because nobody wants to defend planning store orders off a national figure once it is written down.

"We beat the target every month." Then the target is set at the wrong grain, or against no benchmark. Ask what the naive forecast scores on the same data. If the model beats last-week-repeated by two points, the target is measuring whether demand is stable rather than whether the model is good.

"Bias is small, so it does not matter." Small and persistent is the expensive combination, and it is the one that never triggers an exception. Multiply the bias by the volume and the cover period, and it becomes a number of units sitting in the estate. That is usually enough to end the discussion.

Where Scout fits

Scout scores forecasts against actuals at store and item grain, from the same POS and syndicated feeds the forecast was built on, and reports WMAPE and bias side by side rather than an accuracy figure alone. The tail comes out ranked by volume so the review has a worklist, and promoted weeks are separated from baseline weeks by default.

The boundary: Scout does not generate the forecast. It measures the one you have, against what happened, at the grain your decisions are actually made at.

The short version

• Forecast accuracy is at least three questions. One number answers one of them.
• WMAPE is the better default than MAPE: it survives zeroes and is not distorted by small-volume items.
• Bias is the metric that predicts inventory, and the one most often missing. A small consistent bias costs more than a large random error.
• The grain you measure at determines the score. Measure at the grain the decision is made at, and expect the number to look worse than the one people are used to quoting.
• A good average still hides error concentration, promotional error, and failure to beat a naive baseline. Report all three.
• A review that does not end in a changed parameter has not done anything.