Evaluate an expense-coding AI by category errors and unanswered items

An expense-coding tool can read an invoice amount correctly and still assign the transaction to the wrong category. Evaluate the coding decision using the company's approved categories and evidence. A demonstration that extracts supplier names and dates answers a different question from whether the resulting expense recommendation is suitable for your records.

State the decision the model is allowed to make. It might recommend an expense category, a department or a project. Those are different labels and may need different supporting information. A supplier can serve several departments, and one invoice can contain several kinds of spending. Decide whether the unit being classified is an invoice, a line or an allocation before the vendor presents a success percentage.

Include an explicit outcome for insufficient evidence. Some transactions require a contract, an operating explanation or a specialist judgment. A tool should be able to send those items for review without manufacturing a confident label. That unanswered population still creates work, so it must remain visible beside the correctly classified transactions.

Ask the accounting owner to approve the categories and the evidence used to assign them. Historical coding is useful input, but it may contain old policies or mistakes. Repeating every historical label is not necessarily the intended result. Resolve a sample of ambiguous items before evaluating the vendor, and record items for which the available evidence does not support a single answer.

Preserve the relevant policy version and transaction context. If the company changes a department structure, an older label can be correct for the original reporting period and inappropriate for a new transaction. Tell the vendor which policy and effective dates the evaluation uses. Otherwise a disagreement caused by a changed reference can be mistaken for a model failure or improvement.

Keep the evaluation separate from examples used to configure the tool. Include unfamiliar supplier descriptions and exceptions that matter to your business, within a representative population. Where additional difficult cases are deliberately added, report them separately or preserve their sampling weights. A stress set can reveal weaknesses without pretending to describe the frequency of those cases in ordinary work.

Consider a fictional evaluation of 1,000 transactions that all have an approved single reference label. Nine hundred belong to subscriptions and one hundred to delivery costs. The model assigns 870 subscription items correctly, mislabels twenty as delivery and leaves ten unanswered. For actual delivery items, it assigns fifty correctly, mislabels forty as subscriptions and leaves ten unanswered.

The table shows why one accuracy percentage is insufficient. A model that labels everything subscriptions would already be correct on 900 transactions because that category dominates the sample. The proposed model finds some delivery items, but still misses half of the actual delivery population when unanswered items are included. Management needs that result before deciding whether the tool can support reliable category reporting.

Use a category table appropriate to the actual number of labels. Preserve closely related categories separately where confusing them would change a decision or review. Do not combine difficult categories into a broad other group solely to improve the headline percentage. An aggregation can be useful for presentation, but the underlying errors should remain available to the evaluator.

In the example, the tool produces 980 classifications and leaves twenty unanswered, so coverage is 98 percent. It makes 920 correct classifications. Correctness among the classified items is 920 divided by 980, about 93.9 percent. Correct classifications as a share of every evaluated transaction are 92 percent. Those measures answer different questions and should not share an unlabeled accuracy heading.

For the delivery category, fifty of the seventy items labeled delivery are correct. Precision is therefore about 71.4 percent. Fifty of the one hundred actual delivery items receive the correct label, giving 50 percent recall when the ten unanswered delivery items count as not automatically identified. Google's classification guidance explains precision and recall using true positives, false positives and false negatives. State the treatment of unanswered cases whenever applying those definitions.

A proposal that quotes precision only can conceal how little work receives a recommendation. A proposal that quotes coverage only can conceal incorrect recommendations. Request both, by meaningful category, with counts. If the tool changes its confidence threshold or rules, rerun the comparison on the same reference set and record the version. A new threshold can change the workload and the pattern of mistakes even when the interface looks unchanged.

Investigate the actual coding errors with the accounting owner. A wrong department, wrong period or wrong nature of expense can have different consequences. Record the amount affected and the decision or report that could be distorted. Avoid inventing one universal financial loss per error: a correction found before posting is different from an error that reaches an external report.

Measure the human work in both populations. Unanswered items require a decision, while recommended items may still need review under the approved process. In the illustration, sending only the twenty unanswered items to a person would leave sixty known incorrect recommendations in the classified population if nothing else checked them. High coverage has not removed that review question.

Test whether the suggested label helps the reviewer or anchors them to a wrong answer. Compare the evidence available, time needed to reach a supported decision and corrections made after initial acceptance. The purpose is to establish a workable division between software and people. A vendor's promise of automatic coding does not itself authorize unattended ledger posting or a change to the company's approval controls.

Give the vendor an acceptance record showing the label definitions, evaluated population, reference-review method, model version, category results and unresolved limitations. Identify which transaction classes may proceed to the next trial and which remain outside scope. NIST's AI Risk Management Framework is voluntary guidance for managing AI risk; it is not a certification that a particular accounting model is fit for this use.

Use company-approved access and data handling for the trial. Transaction descriptions can contain personal or commercially sensitive information even when bank account numbers are removed. The FTC's business guidance supports limiting collection, access and retention to a legitimate need. Agree what the provider may retain or use to improve its service, including evaluation labels supplied by your staff.

After deployment, inspect results when the chart of accounts, supplier mix, business activity or coding policy changes. Keep the old evaluation and the revised reference definitions so the team can distinguish a different task from deteriorating performance on the same task. The purchase should leave a measurable classification process with known review work, rather than an accuracy claim that becomes impossible to interpret once real transactions arrive.