Evals for people who don’t write code: a spreadsheet method
“Is this AI feature good enough?” is a question that engineers and product people often answer by feel. A small, honest test is better, and it needs a spreadsheet, not code.
The method
Figure 1 · A first eval
Pick 20 real inputsWrite the expected resultRun the featureMark pass or failGroup the failures
- Pick 20 real inputs. Take them from actual emails, tickets, documents, or questions. Include a few ugly ones.
- Write what a good result looks like. For each, note what must be true, such as the correct total or the key clause.
- Run the feature on each input and paste the output next to the input.
- Mark each pass or fail. Add a one-line reason for every fail.
- Group the failures. Three failures with the same cause are one problem to fix, not three.
What the sheet looks like
Figure 2 · A starter eval sheet
| Input | Expected | Output | Pass? | Reason |
|---|---|---|---|---|
| Refund request, order missing | Asks for the order number | Asks for the order number | Yes | |
| Angry message in Hindi and English | Polite reply, same language mix | English only | No | Ignored the language |
| Request outside policy | Declines and offers an alternative | Agrees to refund | No | Made up a policy |
Tips that make it useful
- Include hard and unusual cases. Twenty easy examples will tell you nothing.
- Keep the set fixed. That lets you compare versions fairly.
- Add new failures from real use. Each real mistake becomes a new row.
- Decide the bar beforehand. For example, “at least 18 of 20, and no serious failures”.
- Have someone else score some of them. You will learn where “good” is unclear.
Twenty rows will not prove anything statistically, but they will show you what breaks, and that is the point. Later you can grow the set and automate it. Judging the quality of an output is part of our corporate training agenda.