Score an upper cost forecast without rewarding unlimited contingency

A team forecasting the 90th percentile of cost wants actual costs to fall below that estimate about 90% of the time in comparable repeated cases. Raising every estimate far above any plausible cost can improve the observed coverage while making the forecast less useful for allocating money.

Quantile loss evaluates both the direction and size of the error for a specified percentile. Hyndman and Athanasopoulos describe the quantile score, also called pinball loss, as an asymmetric scoring rule. It gives a larger penalty to understating a high cost quantile while still penalizing unnecessary overstatement.

Define the target before scoring. A P90 total-cost estimate, a mean remaining-cost forecast and an approved spending cap are different quantities. A management contingency chosen for other reasons should not be presented as a statistically evaluated quantile without a defensible forecasting process.

For a 90th-percentile forecast, this guide uses twice the usual unscaled pinball loss, matching the cited textbook's quantile-score convention. If actual cost exceeds the forecast, multiply the shortfall by 2 times 0.9. If actual cost is below the forecast, multiply the excess forecast amount by 2 times 0.1.

Thus an estimate $20,000 below actual cost receives a score of $36,000, while an estimate $20,000 above actual receives $4,000. The asymmetric penalty reflects the selected quantile. At the 50th percentile, this doubled convention becomes ordinary absolute error.

Some software reports the score without the factor of two. That changes its numerical scale but not the ranking of models evaluated with the same convention. Record the formula so an apparent improvement is not merely a change in reporting scale.

Consider four completed cost outcomes of $80,000, $90,000, $100,000 and $120,000. Method A forecast $100,000 as P90 for every case; method B forecast $120,000; method C forecast $200,000. These deliberately simple values illustrate scoring, not a credible percentile model.

Method A's scores are $4,000, $2,000, zero and $36,000, averaging $10,500. Method B's are $8,000, $6,000, $4,000 and zero, averaging $4,500. Method C's are $24,000, $22,000, $20,000 and $16,000, averaging $20,500.

Both B and C cover all four outcomes, yet B has the better score because C adds much more unsupported cushion. A covers only three of four and receives a large penalty for its $20,000 underestimate. Four outcomes demonstrate the arithmetic; they cannot establish reliable 90% calibration.

Compare methods at the same quantile level and on the same completed cases. A P50 forecast is expected to behave differently from P90. Changing the percentile or selecting easier projects can change the score without improving the method intended for the original decision.

Match scope, currency, price basis and forecast stage. A total-cost forecast made before scope definition should not be compared indiscriminately with one issued when most supplier invoices are already known. The remaining uncertainty and information available differ materially.

When final cost includes separately approved new work, retain a scope-change bridge. Decide how the original forecast target is evaluated before seeing which treatment improves its score. Otherwise an expansion of the project can be mistaken for an error in predicting the originally defined cost.

Preserve the forecast as issued. Updating the P90 amount after an overrun becomes visible may be appropriate for current management, but it cannot replace the earlier estimate in the accuracy record. Retain both versions with their dates and horizons.

Keep track of cases whose costs are not yet final. If lengthy, expensive work remains open while short routine work closes quickly, scoring only the currently completed projects can present an unrepresentative picture. Define mature evaluation cohorts and report the unresolved population alongside them. Do not substitute an optimistic latest estimate for a final actual merely to make every row scoreable.

Track the fraction of outcomes at or below the forecast alongside the average score. The two measures reveal different problems. A model may understate the target percentile systematically, or it may have reasonable overall coverage while producing occasional very large errors.

Review meaningful groups when there are enough observations. New product work, routine renewals and complex implementation projects may have different uncertainty patterns. An aggregate score can conceal poor performance in the group for which the forecast is most valuable.

Show sample sizes and acknowledge sampling uncertainty. Closely spaced projects may share suppliers or cost conditions, so the number of rows can exceed the amount of independent evidence. One favorable reporting period should not justify a strong claim that a high percentile is calibrated.

A dollar-based score gives more influence to larger dollar errors. That may fit a capital or operating budget where the cash exposure matters. It can also make the results largely reflect a few large projects, leaving smaller projects almost invisible in the aggregate.

If using a scaled score or fixed exposure weights, define the denominator and weights before outcomes are known. Avoid dividing by a realized cost that can be zero or unusually small without considering the resulting instability. Keep the ordinary dollar view available for the funding consequence.

Do not reinterpret the score as an expected payment or a required reserve. The $4,500 average in the example is an evaluation statistic in dollar units. It is neither the next project's expected overrun nor an amount that should automatically be added to its budget.

Investigate large underestimates through their actual causes: missing scope, weak quantity assumptions, price changes or omitted dependencies. Investigate repeated overstatement too, especially when it unnecessarily reserves funds. The scoring rule identifies performance patterns; it does not identify their cause by itself.

Compare proposed improvements on later outcomes and retain an appropriate reference method. Keep the budget approval separate from the statistical estimate so management can explain any additional caution it chooses. That preserves both a useful risk forecast and an honest account of the decision made from it.