15 questions to evaluate an accounting AI assistant

Evaluate an accounting assistant against a small set of records your team understands. Choose a bank balance, an unpaid customer invoice, a supplier bill, and a completed monthly report. Write the expected answer and its source before asking the assistant. A fluent explanation is useful only when the amount, entity, period, and accounting meaning are right.

NIST's AI Risk Management Framework organizes the work of identifying and managing AI risks. For a buyer, a practical application is to test the intended task and the consequences of getting it wrong. Answering where a document is stored and authorizing a vendor payment deserve different evaluation requirements. Decide which tasks the assistant will be allowed to perform before judging its overall usefulness.

Build the first test around ordinary finance questions, then add one exception. For example, a bank balance and a cash-available figure can both be correct while representing different things. Ask the assistant to identify the source and explain exclusions. It should distinguish a forecast from a posted ledger amount and a processor balance from settled cash in the bank.

Use a correction to test freshness. In a permitted test environment, update an invoice or upload a revised report, then check when the assistant recognizes the change. Record the version or synchronization time it uses. A delayed system can still be usable when the delay is visible; an unexplained mixture of old and new records makes a confident answer difficult to review.

Test access with accounts and records created for the evaluation. A user assigned to one entity should not receive information from another entity simply because the assistant can search across a broader database. Check summary answers as well as document links. Refusing to open a file is insufficient if the answer has already disclosed its contents.

Ask how the service uses questions and attached records, including processing providers, retention, and any training use. The FTC's guidance supports limiting access to sensitive information and reviewing service-provider safeguards. Require product-specific answers rather than assuming that an assistant inherits every permission or data-handling rule from the screen around it.

If the assistant can change records, define what it can prepare, what it can commit, and which actions require approval. Test a draft classification separately from a payment or tax-related action. An approval screen should identify the actual object and change. A broad confirmation such as proceed does not help a reviewer catch the wrong supplier or entity.

Include a failed connection and an unsupported question. The assistant should expose a material limitation and offer an appropriate next step instead of filling a gap with a plausible amount. Keep the test focused on the service being purchased. The purpose is to establish useful boundaries, not to reward the assistant for answering every imaginable question.

Separate a cosmetic wording issue from the wrong currency, omitted liability, unauthorized disclosure, or duplicated action. For each failure, keep the question, expected result, actual result, relevant source version, and reviewer decision. Avoid placing live credentials or sensitive customer details in a test log when a fictional record can reproduce the issue.

Repeat the affected test after a correction. Do not replace your original examples with easier questions that happen to pass. Keep a small recurring set for changes to models, retrieval, integrations, and permissions. The release decision should reflect the tasks you plan to rely on and the material failures that remain, rather than an average score that hides a serious access problem.

An assistant may be useful for finding documents and explaining balances while still needing review for journal proposals or forecasts. Write down the approved uses and train staff to verify the parts that affect financial decisions. A product demonstration cannot establish professional authority for tax, legal, or accounting judgments outside the service's agreed scope.

Approve broader use only when the relevant evidence supports it. Identify who can pause an action feature, correct a misleading answer, and communicate a material issue to affected users. Keep source access and financial controls authoritative as the assistant evolves. That gives your team a useful tool with an understandable boundary and a repeatable way to decide when that boundary should change.