How to make AI admit uncertainty: a buyer's checklist

Most AI demos look great until you ask a question outside the vendor's happy path. Before signing a contract, run this short evaluation so you know how the system behaves when it hits the edge of its knowledge.

Step 1: Prepare an uncertainty test set

Write 20 prompts in three buckets: in-scope with clear answers, in-scope but unanswerable from the knowledge base, and out-of-scope entirely. The middle bucket is where overconfidence hides.

Step 2: Score the responses

For each answer, mark one of: correct with citation, honest refusal, hand-off to a human, or confident guess. Any confident guess on the unanswerable set is a red flag.

Step 3: Check for source attribution

Every substantive claim should link back to a document or data source. If citations are missing or fabricated, the system is not grounded. See Inside citations and source attribution in enterprise AI answers.

Step 4: Verify the hand-off path

Ask a question that should escalate (dosage, legal advice, pricing commitments). A responsible system routes to a human instead of improvising.

Step 5: Re-test after tuning

After the vendor adjusts prompts or retrieval, run the same set again. Scores should improve on honest refusals, not just on confident answers.

For the framework behind this, see our Responsible AI-by-Design Framework.