How to Vet an AI Vendor Before You Trust Their Verdicts

The FIFA offside controversy is a procurement story as much as a sporting one. Someone signed off on a system that ships verdicts in millimeters from data known only in centimeters, with no published error rate. Most enterprise AI buyers are one contract away from the same mistake. This guide walks through a practical vetting process before you deploy any AI that will decide something that matters.

Step 1: Force the vendor to name the decision

Before anything else, write down the exact decision the AI will make: approve or deny, offside or onside, hire or reject, flag or pass. If the vendor can't state the decision in one sentence, they're selling you a demo, not an instrument.

Step 2: Ask for the published error rate

Request a specific number: false-positive rate, false-negative rate, and the test set they were measured on. "High accuracy" is not an answer. "96.4% precision on a held-out set of 12,000 labeled examples from Q3 2025" is an answer. If they refuse, treat it as a refusal to be measured.

Step 3: Get the decision rule in writing

What threshold triggers a positive verdict? What margin triggers a human review? The Premier League uses a 5 cm tolerance; FIFA's auto-alert only fires above 10 cm. Those are decision rules. Your vendor should have one, and it should be documented in the contract, not the sales deck.

Step 4: Demand confidence per verdict

A trustworthy system tells you how sure it is. If every output arrives with the same confident tone regardless of input quality, you have no way to route the hard cases to a human. Ask to see the confidence score in the output payload.

Step 5: Test the operating limits

Every model has conditions where it degrades — lighting, accent, document format, edge case. Ask the vendor for the defined operating envelope, then feed the system inputs from just outside it. Watch what happens.

Step 6: Require an auditable record

Can you reconstruct any single decision six months later? Inputs, model version, confidence, the human who confirmed or overrode it? If the log doesn't exist, neither does accountability.

Step 7: Ask who has independently verified it

The vendor's own benchmarks don't count. Look for third-party evaluation, published methodology, or at minimum a customer running the system in adversarial conditions who will talk to you.

Step 8: Design the override path before you buy

Who can reverse a bad verdict, on what timeline, with what