AI evals: frequently asked questions
responsibleai
Clear answers to the questions leaders and engineers ask most about AI evals, from cost to cadence to who owns them.
Teams starting with AI evals hit the same questions again and again. Here are direct answers.
Q: How is an AI eval different from a traditional software test?
A traditional test asserts a deterministic output for a given input. AI evals score probabilistic outputs against a standard — often a rubric or a reference answer — because the same input can produce different outputs across runs and model versions.
Q: When should we start writing evals?
Before the first prototype ships to a real user. Even 20 hand-written cases beat shipping blind. Evals should exist the moment there's a prompt worth defending.
Q: Who owns the evals — engineering, product, or the domain expert?
All three. Domain experts define what good looks like, product owns the priority of which failures matter, and engineering runs the pipeline. Evals without a subject-matter expert tend to grade the wrong thing.
Q: How many eval cases do we need?
Start with 30–100 covering happy path, edge cases, and adversarial inputs. Grow the set every time production surprises you.
Q: Can an LLM grade its own outputs?
LLM-as-judge is useful at scale but drifts. Calibrate it against human review on a sampled slice, and version the judge prompt like any other model asset.
Q: How often should evals run?
On every prompt change, model swap, or retrieval update — and continuously in production against sampled live traffic.
Q: What does an eval program cost?
Mostly people time to curate cases and rubrics. Compute is a small line item compared to the cost of a bad answer reaching a customer.
Q: Do evals catch prompt injection and safety failures?
Only if you write cases for them. Add adversarial examples and safety rubrics deliberately — see our guide on Prompt Injection Defense for AI Agents.
Silverberry helps organizations stand up eval programs through our Responsible AI-by-Design Framework.