Run it on twenty of my real enquiries from last week. Can I watch, and can I pick which twenty?

Because the enquiry in a demo was chosen by the person selling you the software.

Look at what a demo enquiry usually contains. A clean subject line. An origin and a destination, spelled correctly. A commodity. Gross weight and dimensions. An Incoterm. A ready date. Nothing contradicts anything else, and nothing important is missing.

Now look at Monday morning in your own pricing inbox. A WhatsApp forward with a photograph of a packing list, taken at an angle. A one-line message that says “pls quote urgent” with an attachment called final_final_2.xlsx. A reply on a thread from March, where the new enquiry sits three lines under six screens of quoted history. An enquiry with dimensions but no weight. Two of the twenty are not enquiries at all. One is a rate sheet from an agent, and one is somebody asking where their cargo is.

The gap between the demo enquiry and Monday morning is the entire product. Everything a vendor is charging you for lives in that gap. A demo steps over it.

The NIST AI Risk Management Framework, which is intended for voluntary use, puts the same idea in more careful language. Under its measure function, “AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s)”. Conditions similar to the deployment setting. Your pricing inbox is the deployment setting. Their sample data is not.

Public buyers work the same way. The UK guidance on responsible AI in recruitment, written for a very different kind of purchase, still says that “prior to the deployment of this system, it is recommended that your organisation pilots the technology ...”, and that “understanding how the model performs in your organisation’s real-world environment ... is essential before deploying any model at scale”. A forty-person forwarder cannot run a six-month pilot. Twenty enquiries and an afternoon is the version that fits.

The test has seven steps. It costs you about two hours in total, and most of that is spent watching.

The last step is the one people skip and the one that pays. Twenty enquiries used once is an anecdote. The same twenty run by three vendors is comparable evidence.

Twenty is a practical number, not a statistical one. Four or five of them will decide the result, and they are never the four or five anybody would have chosen.

Take them in date order and you will get a fair sample by accident. If you would rather choose, choose for variety of mess rather than variety of lane.

That last row is worth insisting on. A product that prices an agent’s rate sheet as a customer enquiry will do it again at eight in the morning, when nobody is watching.

The formats matter more than most buyers expect. FreighAI, for example, states that it “Reads PDF, Excel, Word, forwarded email, WhatsApp messages, photos and scans”. Whatever the vendor in front of you claims on that point, your twenty are where you find out, because a real week contains at least three formats nobody chose to demonstrate.

You are not scoring accuracy. You are watching behaviour.

The strongest signal is in the fifth row. A vendor who looks at your twenty, says “four, eleven and seventeen will be poor, and here is why”, and is then proved right, has run at real volume. One who explains each failure afterwards is meeting your kind of work for the first time.

The worst signal is a run with no failures at all. Twenty real enquiries from a real forwarding mailbox contain at least two the product should not get right. If nothing failed, you are not watching your enquiries.

Four, and all four are ordinary rather than dishonest. They happen because the person selling wants a good hour.

• 01The overnight trap. You send the twenty on Monday for a Thursday call, so what you watch includes three days of configuration and rate loading done for your twenty. Send them the same day.
• 02The sandbox trap. The screen is not the product a customer runs. It is an internal build with the parts that break switched off. Ask whether this is the environment their existing customers are on, while the screen is still shared.
• 03The self-selected sample. You send twenty and eleven get run, quietly, because nine were awkward. Number them before you send them, and ask for all twenty back in order.
• 04The quiet person in the loop. Between your file arriving and the draft appearing, somebody on the vendor’s side is finishing the work by hand. The hardest one to see, and the easiest to expose.

The fourth trap has a counter that costs you nothing. Keep two enquiries back and produce them in the meeting. Nothing on this list survives an enquiry the vendor has never seen.

Note what the test does not tell you. Nothing about behaviour at volume in peak season, nothing about how long it takes to run inside your business, nothing about who owns the data it just read. Those are questions B1, B3 and C1.

Write four things down the same day, for each vendor. Not later, and not from memory.

• How many of the twenty produced a draft a pricer could have approved.
• How many failed, and whether the vendor named them in advance.
• What the product did when something was missing.
• What the vendor said they would change, and by when.

Then read which of the three endings you actually got.

The strongest position a buyer can be in is the same twenty enquiries, run by two vendors, in front of the same three people from your business. It takes a week and costs nothing. The UK Guidelines for AI procurement describe themselves as best practice for public-sector buyers, written to help them “evaluate suppliers”. The twenty-enquiry test is what that looks like at the scale a forwarder buys.

One more thing to carry into the next call. The vendor who has just run your twenty will now quote you a percentage. Ninety of a hundred went straight through, or eighty, or seventy-five. That number is question A2, and it means nothing until somebody defines it.