Inside eval datasets: how to curate the cases that catch real failures

The rubric gets the attention, but the dataset decides whether your evals are useful. A polished rubric run against weak cases produces confident, meaningless scores.

What an eval dataset actually is

A versioned collection of tasks, each with the inputs your system will see in production and — where possible — a reference of what a good response looks like. It's a living asset, not a one-time deliverable.

The four sources that matter

• Production logs. The single richest source. Sample real user inputs, especially the ones that produced complaints or thumbs-down.
• Subject-matter experts. They know the edge cases users haven't hit yet. Ask them for the ten inputs they'd use to trip up a junior employee.
• Adversarial synthesis. Deliberately generated prompt injections, jailbreaks, and off-topic asks. See Prompt Injection Defense for AI Agents.
• Regression cases. Every production failure becomes a permanent test case. This is how the suite compounds.

Structuring the dataset

Each case should carry:

• A stable ID so scores are comparable across runs.
• Tags for category, difficulty, and risk tier.
• The expected output or the rubric criteria that apply.
• Provenance — where the case came from and who approved it.

Balance matters more than size

A thousand near-duplicate happy-path cases will hide the failure that ships. Aim for coverage across:

• Input length and complexity.
• User intent categories.
• Sensitive or regulated topics.
• Known adversarial patterns.

Keeping the dataset honest

• Freeze a holdout slice. Never let it inform prompt or model tuning, so you always have a clean measurement.
• Rotate cases in. Retire stale examples the system has clearly mastered.
• Audit annotation. Two experts should agree on the reference answer for high-risk cases. Disagreement is a signal the rubric needs work.

The compounding asset

A six-month-old eval dataset with 500 curated, tagged, and reviewed cases is worth more than any single model choice. It's the artifact that lets you swap models, change prompts, and still know whether the product got better.

Silverberry helps organizations build and maintain these datasets as part of our Responsible AI-by-Design Framework.