How accurate is AI at processing purchase orders?

There is no single honest figure, because it depends on your documents and on what is being counted. Vendors usually quote field or character accuracy on clean samples. What matters to a wholesaler is how many whole orders come out right with no edits, and whether the errors that remain are flagged. Measure that on a held-out sample of your own orders before you believe any number.

What is the difference between line accuracy and order accuracy?

Line accuracy counts the lines read correctly. Order accuracy counts orders where every line, the customer and the delivery details are right. One wrong line spoils the order, so order accuracy is always lower, and it falls as orders get longer. At 98% per line, a 10-line order comes out fully right about 82% of the time if errors are spread evenly.

Can I trust the confidence score on each line?

Use it as a sorting signal, not a guarantee. Microsoft describes a 0.95 confidence as a prediction likely to be right 19 times out of 20, so one in twenty still isn't. Set your threshold by checking scores against your own keyed orders, and send anything below it, plus every price change or new product, to a person.

Does structured output mean the AI read the order correctly?

No. Structured outputs make the model return data in the shape you defined, such as a list of lines with a product code and quantity. Anthropic's documentation describes this as a guarantee of schema compliance, not of correct values. A well-formed line can still carry the wrong product or the wrong unit, so business rules and approval still have to check the content.

How many orders do I need for a fair accuracy test?

Enough to include every customer format you get often, plus the awkward ones: photos, multi-page PDFs and spreadsheets. Two weeks of real orders is a practical starting point for most wholesalers. Keep them out of the set-up work, so the test measures how the system handles orders it hasn't seen, not how well it remembers ones it has.

When can orders skip the approval screen?

Only when your own re-test shows that the errors that would cause harm, such as a wrong product or unit, are always flagged. Even then, keep price differences, unknown products, new addresses and credit-limit breaches with a person. Start with one or two trusted customers whose orders are regular, and keep sampling their approved orders every week.

AI Purchase Order Processing Accuracy: Test It Yourself

Short answer: AI purchase order processing can read typed emails and PDFs very well, but a single accuracy percentage tells you almost nothing about your own orders. Errors compound across lines, so at 98% per line a 10-line order still needs a fix nearly one time in five. Ask three questions instead: how many whole orders come out right with no edits, what share of the wrong lines get flagged for a person, and what each missed error would cost you. Then measure all three on a held-out sample of your own orders.

This guide shows how to run that test. For how an AI order inbox works end to end, see our AI order processing page.

Work out what a line accuracy means for whole orders

Move the sliders to match your orders. The calculator assumes errors are spread evenly across lines, which is the simplest model. In real tests they tend to bunch in a few customers' orders, so treat the result as a guide, not a forecast.

Lines on a typical order 10Count a normal order from a regular customer

Lines read correctly 98.0%Use your own test result if you have one

Emailed or PDF orders a day 40Orders someone keys in today

Move the sliders to see what a line accuracy means for whole orders.

Send me what this means for my orders

Your numbers go into the message for you. Add your name and email, and we reply by email with how we'd test your orders. Nothing is sent until you press send.

This runs in your browser. Nothing you move is sent anywhere unless you choose to send it.

The maths is simple: if each line is right 98% of the time, a 10-line order is fully right 0.98 × 0.98 … ten times, about 82%. At 30 lines it drops to about 55%. That's why a wholesaler whose customers send long weekly restock orders should care much more about flags than about the headline figure.

Four numbers worth asking for

Most published accuracy figures are field or character accuracy on the vendor's own samples. For order processing, ask any supplier, us included, for these four instead, measured on your orders.

Number	What it counts	Why it matters
Line accuracy	Lines where product, quantity and unit all match the order your team keyed	The basic reading quality, but it flatters long orders
Orders right with no edits	Orders where every line, the customer, PO number and delivery details are right	How much of the inbox could, one day, need only a glance
Error flag rate	Of the lines that were wrong, how many the system flagged for a person	The safety number. An unflagged error goes to the warehouse
Flag precision	Of the lines flagged, how many were really wrong	Too many false alarms and people stop reading the flags

The last two are what statisticians call recall and precision. The UK Information Commissioner's Office explains both in its guidance on statistical accuracy in AI, and makes a point that applies directly here: a plain accuracy rate can look high while the system fails at the one job you need. Its example is a spam filter that is 90% accurate simply because 90% of mail is spam. Most order lines are easy, so a high line accuracy can hide a poor record on the hard ones. The ICO notes that this guidance is under review after the Data (Use and Access) Act 2025, but the definitions themselves are standard.

For order processing, we'd trade some flag precision for a better flag rate every time. A false alarm costs a few seconds of someone's attention. A missed unit error costs a short delivery, a credit note and an annoyed customer.

Accuracy test screen for a trade supplier comparing AI drafts with orders keyed by staff across 120 orders and 1,140 lines: 98.0% of lines right, 84.2% of orders right with no edits, and 21 of 23 wrong lines flagged. Errors are broken down by type, with 9 unit or pack size errors the largest group. A side panel says the system is not yet ready for faster approval because 2 errors were not flagged, and lists three rules to add before a re-test. — A held-out test scored line by line. The headline 98% looks fine; the two unflagged errors are what decide the next step. Illustrative screen, sample data.

What confidence scores can and can't tell you

Many document-reading services return a confidence score with each field. Microsoft's documentation for Azure Document Intelligence describes field confidence as an estimated probability that the prediction is correct: 0.95 means likely right 19 times out of 20. Three points from that page matter for purchase orders:

One in twenty is still wrong. Microsoft suggests using confidence to decide whether to accept a result or send it for human review where accuracy is critical, and recommends close to 100% for sensitive cases such as financial records.
A table can be wrong while its cells look fine. For tables, Microsoft notes that a correctly read cell can sit in a row with other misses, so the cell's confidence is high and the row's is low. PO lines are table rows. Check confidence at row level, not just per field.
Not every field gets a score. Microsoft says not all document fields return one. Missing scores need a rule of their own, not a default pass.

Language models used for order reading often don't return a calibrated score at all. Either way, set your threshold by comparing scores with keyed orders from your own test, not from a default.

A well-formed answer isn't a correct answer

Modern models can be made to return data in a fixed shape: a customer, a PO number and a list of lines, each with a product code, quantity and unit. Anthropic's structured outputs documentation describes this as guaranteeing schema-compliant responses, meaning valid JSON with the required fields and types. It says nothing about the values being right, and the page itself lists cases, such as a response cut off at its token limit, where even the shape may not match.

In practice that means a draft can be perfectly formatted and still order 5 singles instead of 5 cases. Shape checks belong in the build, but your own business rules (price list, unit table, customer history, credit limit) do the checking that matters.

Photos and scans: set a lower bar on purpose

Anthropic's vision documentation says its models may make mistakes when interpreting low-quality, rotated or very small images, and advises against using them for tasks needing perfect precision without human oversight. A phone photo of a paper order pad is exactly that kind of input. Score photos as their own group in your test, and keep every line from them going to a person, whatever the typed-PDF results say.

How to run a fair test on your own orders

Collect a real sample. Two weeks of orders as they arrived, the awkward ones included: long restock orders, spreadsheets, multi-page PDFs and photos.
Hold it back. Don't use these orders to build customer mappings or prompts. Otherwise the test measures memory, not reading.
Use what your team keyed as the answer. Compare every draft line with the order that was actually picked and invoiced, field by field.
Score all four numbers, and split them by format and by customer. Errors usually bunch.
Rank errors by cost. A wrong product or unit that ships costs far more than a blank line that can't post.
Add rules, then re-test on fresh orders. Faster approval for a customer comes only after a re-test shows no harmful errors going unflagged.

If you're in the UK: two official yardsticks

Orders carry personal data. Buyer names, emails, phone numbers and delivery addresses on a purchase order are personal data, and the ICO's accuracy principle asks you to take reasonable steps to keep it correct and to fix it without delay when it isn't. In practice, a draft that changes a delivery address or contact should never pass without a person, and a correction made on the approval screen should update the customer record, not just the order.

A useful checklist, even for a private firm. The government's AI Playbook (February 2025) is written for public bodies, but two of its principles fit order processing well: fully test the product before deployment and keep regular checks on the live tool, and make sure people validate the high-risk decisions. For a wholesaler, "high-risk" means anything that changes what is picked, what it costs or where it goes.

How we build for accuracy

We build AI order processing on top of the system you already use, or on our trade portal, which covers sales and warehousing and is used every day by a UK wholesaler. The AI order inbox itself is a build, not an off-the-shelf feature. We start with every order going to an approval screen and add the rules your test results call for. You get a fixed quote after a 20-minute walkthrough, with the full cost known before work starts, then a fit check of your customers, products, units and price lists, set-up on a fixed plan and close support through the first weeks. Our development team is in India.

For what the build costs to run, see how much AI automation costs. If you're weighing platforms, n8n vs Make vs Zapier shows why order lines change the bill on some of them.

Book a 20-minute walkthrough or send us a description of your order formats. We'll tell you how we'd test them before quoting anything.

Sources

Information Commissioner's Office, What do we need to know about accuracy and statistical accuracy? (marked as under review following the Data (Use and Access) Act)
Information Commissioner's Office, Principle (d): Accuracy
UK government, Artificial Intelligence Playbook for the UK Government (10 February 2025), principles 4 and 5
Microsoft Learn, Interpret and improve model accuracy and confidence scores (Document Intelligence v4.0, accessed 19 October 2026)
Anthropic, Structured outputs (accessed 19 October 2026)
Anthropic, Vision: limitations (accessed 19 October 2026)

Last reviewed 19 October 2026. The calculator uses a simple model in which errors are independent across lines; real results vary with your documents. Vendor documentation changes; check each page before relying on it.

AI purchase order processing accuracy: what the percentage hides, and how to test it