Evaluate invoice extraction with a field-level scorecard

An invoice-capture tool can read most of a page correctly and still produce an unusable accounting record. A wrong supplier, currency or total may matter more than several missed words in a description. Before comparing accuracy claims, define the fields the company needs and how each will be used.

Separate extracting information from deciding its accounting treatment. Reading a tax amount, for example, does not establish whether the company can recover it. Identifying an invoice date does not decide the correct accounting period. The evaluation should make clear which work belongs to the tool and which remains with the preparer or reviewer.

Ask the vendor what its quoted accuracy measures: characters, fields, complete documents or records accepted without correction. Those measures are not interchangeable. A useful comparison uses the same sample, expected answers and scoring rules for every shortlisted tool.

Select documents from the sources your team actually receives, with appropriate permission and protection of sensitive information. Include common supplier layouts, multipage invoices, credit notes, scans and digital PDFs in proportions that resemble the workload. Keep a separate challenge group for unusual but consequential cases.

Create a verified answer sheet before running the tools. Have a knowledgeable reviewer confirm the supplier, invoice number, dates, currency, totals and other required fields against the source document. Resolve ambiguity in the sample rather than marking the software wrong for choosing among unclear answers.

Microsoft's Document Intelligence transparency guidance recommends evaluating extraction on the customer's own use case and using the results to decide when human review is needed. Follow that principle without assuming a vendor's illustrative confidence threshold is appropriate for your invoices.

Keep some documents out of the configuration exercise and use them only for the final comparison. If a provider has tuned the tool against every evaluated page, the results may tell you more about those known layouts than about the next unfamiliar invoice. Record which documents were available during setup.

Mark each required field as correct, missing or incorrect using the agreed answer sheet. Record the type of error, especially when the wrong value looks plausible. A missing value can be easy to spot; a convincing but incorrect account number or amount may pass through a hurried review.

In a hypothetical sample of 100 invoices with six required fields each, there are 600 field checks. If 570 are correct, field accuracy is 95 percent. If those 30 errors occur on 25 invoices, only 75 invoices have every required field correct. The complete-document rate is therefore 75 percent, even though the field-level figure is higher.

Do not combine all fields into one score without showing the important categories. Report total and currency errors separately from minor description issues. The buyer can then see whether the product's strengths match the company's risk and review priorities.

Retain the actual extracted values and the source references. A percentage alone is difficult to investigate later. The useful record allows the provider and company reviewer to understand why an item failed and whether a configuration change would address that class of error.

If the tool supplies confidence values, compare them with the verified outcomes. Inspect incorrect fields that received high confidence and correct fields sent unnecessarily for review. The score should help route work, but it is not a substitute for measuring performance on your sample.

Ask which fields receive a confidence value and what the number represents in the product. Avoid treating a document-level score as proof that every field is equally dependable. A confident total can coexist with an uncertain supplier or date.

Choose review rules based on the observed error pattern and the consequence of a mistake. The company may require manual review for particular fields or document types even when the model is confident. Document those choices as operating requirements so a later optimization does not silently remove them.

Test how the review queue presents the evidence. A reviewer should be able to locate the source value, correct it and see what will be saved. If the interface makes every correction slow, a strong extraction result may still leave a disappointing overall workflow.

Time a representative review session, including opening documents, correcting fields, resolving missing information and confirming the resulting record. Use ordinary staff with appropriate training. A vendor expert who knows the sample by heart may move faster than the team that will operate the tool.

Keep extraction errors separate from other exceptions, such as missing purchase orders, duplicate invoices or unclear business purpose. Those issues may remain even when every printed field is read correctly. Include them in the operating estimate without blaming the extraction model for decisions it was not designed to make.

Compare the full workflow with the current process using the same document population. Report staff time and unresolved items, not just processing speed. If a tool shifts work from data entry to review, identify who has the capacity and authority to perform that review.

Ask whether correcting an item changes future behavior and under what conditions. Do not assume every correction automatically trains the system. Record the supported configuration or learning process and the effort needed to maintain it as suppliers change layouts.

The acceptance record should identify the sample, required fields, observed results, review rules and remaining limitations. State where performance was not evaluated, such as a language or document type absent from the sample. This makes the result useful without pretending it predicts every future invoice.

Plan a targeted review when new suppliers, formats or capture channels become significant. Compare the emerging error pattern with the original scorecard. A change in document quality or mix can justify adjusting the workflow even when the software itself has not changed.

Choose the tool that produces usable records with an acceptable review burden for your company. The most impressive headline percentage may not describe the fields or decisions on which your accounting process depends.