Evals for people who don’t write code: a spreadsheet method

    “Is this AI feature good enough?” is a question that engineers and product people often answer by feel. A small, honest test is better, and it needs a spreadsheet, not code.

    The method

    Figure 1 · A first eval

    Pick 20 real inputsWrite the expected resultRun the featureMark pass or failGroup the failures
    Do this before shipping and again after every change.
    1. Pick 20 real inputs. Take them from actual emails, tickets, documents, or questions. Include a few ugly ones.
    2. Write what a good result looks like. For each, note what must be true, such as the correct total or the key clause.
    3. Run the feature on each input and paste the output next to the input.
    4. Mark each pass or fail. Add a one-line reason for every fail.
    5. Group the failures. Three failures with the same cause are one problem to fix, not three.

    What the sheet looks like

    Figure 2 · A starter eval sheet

    InputExpectedOutputPass?Reason
    Refund request, order missingAsks for the order numberAsks for the order numberYes
    Angry message in Hindi and EnglishPolite reply, same language mixEnglish onlyNoIgnored the language
    Request outside policyDeclines and offers an alternativeAgrees to refundNoMade up a policy
    Illustrative rows from an imagined support assistant, not real results.

    Tips that make it useful

    • Include hard and unusual cases. Twenty easy examples will tell you nothing.
    • Keep the set fixed. That lets you compare versions fairly.
    • Add new failures from real use. Each real mistake becomes a new row.
    • Decide the bar beforehand. For example, “at least 18 of 20, and no serious failures”.
    • Have someone else score some of them. You will learn where “good” is unclear.

    Twenty rows will not prove anything statistically, but they will show you what breaks, and that is the point. Later you can grow the set and automate it. Judging the quality of an output is part of our corporate training agenda.

    Want help putting this into practice?

    Book a free consultation and tell us about your team or school.

    Book a free consultation