Blog

July 21, 2026 · All posts

How to measure LLM output quality

How are you testing AI-generated text before it ships? If your answer is "we ran a few examples and it looked fine," that's demoing – not testing. Real testing means running the same test cases before every prompt or model change – and including representative adjacent cases and curveballs.

Happy-path demos targeting a few easy cases feel productive. They get features out the door. But they don't catch the scenarios that generate support tickets or the one edge case that blows up on social media. LLM outputs can vary run to run, so you can't rely on them to produce good outputs all the time. And just because one output looks fine doesn't mean the next one will too.

A good test set covers the situations your users are most likely to encounter. That includes common cases, realistic variations, adjacent scenarios that are still plausible, and deliberate curveballs that test whether your prompt handles unexpected inputs gracefully.

Collect them from support logs, QA notes, near-misses in staging or create them yourself, based on your product's requirements and your rubric. Run the set before every change — that's how you know whether quality improved or regressed.

Let's explore an example: A test set for a customer support chatbot

One of the most common LLM use cases is a customer support chatbot. Suppose you're building a chatbot for an online furniture marketplace. The chatbot should answer product questions accurately, refuse questions it can't answer, and avoid inventing information.

A simple question-handling test set might look like this:

• Common cases: Questions that are relevant to the product and the customer support process and are common, e.g. "What size is the product?". (These likely split into questions that you can repeat for different products in the same category and more unique questions that are specific to each product, but we'll keep it simple for now.)
• Variations of common cases: Questions that are relevant to the product but not as common, e.g. "What is the spatial footprint of this piece?" or "How tall is the smaller version?"
• Adjacent cases: Questions that are not directly relevant to the product and the core utility of the chatbot but are still possible because they're related to the business, e.g. "What is your return policy?"
• Curveballs: Questions that are not directly relevant to the product or business or other unusual, wrong or unexpected questions, e.g. "How many calories is this?", "What is your name?", "What is the weather forecast for tomorrow?", "How much big (sic) is this?", "What is they (sic) sizes?"

Each category catches a different failure mode. Common cases measure core functionality, variations test robustness, adjacent cases check scope boundaries, and curveballs reveal whether the model behaves sensibly when users do something unexpected.

How big should your test set be?

As you saw in the example, your test set should be a representative set of the scenarios you care about, spread across all 4 categories. But how many test cases should you have?

There's no universal size for a good test set. Start with enough examples to cover each category well. As your application grows, keep adding new cases whenever you discover failures in production or support tickets. Once your test cases work reliably for one product category (e.g. sofas), repeat them across additional categories (e.g. tables, dressers, etc.) to see whether performance generalizes. By applying the same test cases across many products in that category, you can make sure performance is consistent.

This seems complicated. Can I get a shortcut?

You can always test less, but every scenario you skip becomes something you're choosing not to measure. That's evaluation debt.

The setup may seem complicated, but notice how this process carefully scales from category to category, allowing you to easily see whether your chatbot application scales across product categories. This process also shows you immediately how you can solve issues as you scale: You'll quickly see which issues are worth solving first, giving you a roadmap for improving quality instead of reacting after something breaks.

If this still seems complicated, don't worry. We're here to help.