Check whether invoice collection probabilities deserve your confidence

An invoice assigned an 80% probability of payment can remain unpaid without proving that the probability was unreasonable. Likewise, one payment arriving on time does not validate the estimate. A probability describes uncertainty across comparable situations, so review a collection of forecasts whose outcomes are now observable.

The Brier score provides one way to evaluate binary probability forecasts. ECMWF's forecast-verification guidance defines it as the mean squared difference between the stated probability and an outcome coded as zero or one. The same arithmetic can be applied to a clearly defined invoice-payment event, although the business must design and validate its own collection model.

Define that event before producing the forecasts. For example, an event could mean receiving the full undisputed balance by a named Friday. Payment eventually, payment before the contractual due date and payment within the next 14 days are different events. A score is interpretable only when forecasts and outcomes refer to the same one.

Suppose four invented invoices receive payment probabilities of 0.8, 0.6, 0.3 and 0.1 for a common forecast horizon. The first and third meet the defined payment event; the second and fourth do not. Their outcome codes are therefore one, zero, one and zero.

Subtract each outcome from its probability and square the difference. The resulting losses are 0.04, 0.36, 0.49 and 0.01. Their average is 0.225. Lower scores represent smaller probability errors on the same evaluated sample, with zero corresponding to probabilities that match every binary outcome exactly.

A forecast of 0.5 for every invoice would score 0.25 on this sample. The example's relative improvement is 1 minus 0.225 divided by 0.25, or 10%. Four observations demonstrate the calculation; they are far too little evidence to establish dependable collection performance.

A fixed 50% reference is convenient for arithmetic but may be a poor business benchmark. If most eligible invoices normally pay within the horizon, an estimate based on a suitable historical payment rate can be more informative. Establish that reference using information available when the forecast would have been made.

Compare the proposed model and the reference on the same invoices, forecast dates and outcome definitions. Changing the evaluation population can change the score because some groups are easier to predict than others. A better score on a different customer mix does not by itself establish model improvement.

Keep the reference sufficiently stable to show what the new method adds. If the reference uses customer-specific history, document that choice. If a new customer has no payment record, specify the broader group or prior assumption used instead of quietly excluding the difficult case.

Preserve unsuccessful collection attempts in the eligible evaluation population. Dropping invoices after they become disputed would change the sample precisely when the forecast encounters a difficult outcome.

Group forecasts into sensible probability ranges and compare their average predicted probabilities with observed payment frequencies. If invoices assigned around 80% payment probability pay only 60% of the time in an adequately supported sample, that part of the model deserves investigation.

Show how many observations support each comparison. A bin containing five invoices is much less informative than one containing hundreds of comparable observations. Report uncertainty and avoid treating a noisy difference as proof of systematic bias. Combining too many unlike customers can also hide a problem within a smaller group.

A well-calibrated forecast need not be highly informative. Predicting the same overall payment rate for every invoice may be broadly calibrated while failing to distinguish stronger and weaker payment prospects. The overall score, calibration pattern and useful separation between cases answer related but different questions.

Save the probability before the payment outcome becomes known. Retain its timestamp, horizon, invoice balance and relevant input version. A model reviewed using later customer correspondence or a corrected dispute status can appear more accurate than the version finance actually used.

Decide how to handle repeated forecasts for the same invoice. Daily updates are legitimate decisions, but they are related observations. A large daily dataset can overstate the amount of independent evidence if most rows come from the same small group of invoices and customers.

Only score outcomes after the full horizon has elapsed and the receipt data are sufficiently complete. A recent unpaid invoice is not yet a failed 30-day collection forecast. Maintain explicit rules for partial payments, credits, disputes and reversals so the outcome label cannot change opportunistically.

An ordinary Brier score gives each invoice equal weight. That is useful for evaluating probabilities across cases, but a $200 invoice and a $200,000 invoice have very different cash consequences. Report exposure and the performance of large invoices alongside the ordinary score.

If an amount-weighted score is used, state the weights and keep them fixed by information known at the forecast date. Explain that the result answers a different evaluation question. Do not silently replace an equal-invoice measure and then describe the movement as a pure improvement in the same metric.

A model can score well overall and still miss the few receipts that matter for payroll. Review the actual decision process, including the funding response to uncertain large invoices. Probability accuracy supports that process; it does not make weighted expected receipts spendable cash.

Identify whether problems concentrate in a horizon, customer group, dispute status or forecast probability range. Check the underlying records before changing the model. A late bank feed requires a data correction, while consistently optimistic estimates for unresolved disputes require a different forecasting response.

Keep a dated comparison of the old method, proposed revision and reference forecast on genuinely subsequent outcomes. That record makes the review useful: finance can explain whether probabilities became more dependable, where uncertainty remains and which collection decisions still require individual judgment.