Trust, but Calibrate: Building Reliable LLM Eval Suites

A while back I posted on LinkedIn that I’m not worried about vendor lock-in with the big AI labs because any product using LLMs needs evals anyway, and a good eval suite is what lets you swap models when you need to. My friend Abhi replied with the obvious follow-up question:

Any chance there’s an example of what you consider a good eval suite + codebase that does all this? […] I’m especially interested in seeing what step 2 looks like and the model fallback setup.

It’s easy to say “just have evals” and much harder to say what that actually means in a codebase and what “good” looks like.

Writing it up clarified something I’d left implicit in the original post. Actually choosing a mode becomes almost mechanical once you can measure quality. Much of the learnings are about the measuring instrument itself being broken: a judge that invented most of the hallucinations, an answer key that contradicted itself, labels applied to hastily. A broken measurement tool will leave you wondering why all models perform badly on simple task and you’ll waste a ton of time digging up why this happened.

TL;DR

• Calibrate your judges. - An LLM-as-judge is untested code. Hand-label 10–15 pairs across the whole score range and confirm the judge agrees with you before you believe a single number it produces. Two of the judge models I tried scored 0.12 against my labels, and both looked entirely plausible until measured.
• 10–20 examples - This is more like unit testing, not ML training. We need coverage of case types, the second example of a type buys very little.
• Derive your labels yourself, before you write the scoring key - The friction of labeling by hand is how you discover your definition of “correct” is ambiguous. Rubber-stamping a model’s proposed labels costs far more later than deriving twenty yourself costs now.
• Make sure you get judge output and reasoning - When tuning your judges, you don’t just need aggregates but need to see what questions they scored how and what their reaoning was. This is what enables prompt
• Report a score vector, not a verdict - No pass/fail in exit codes. Rates with numerators and denominators, plus the offending items. This is usually about tradeoffs.
• Have a fallback model from a different lab, and run it through the same eval suite. - If you want a fallback, you need to know which works and ideally the fallback is from a different lab or provider.
• If all models get a example wrong the issue is probably your label
• If the same model disagrees with itself across runs your prompt or scoring rubrik is probably ambiguous

Suite Components

A solid eval suites needs

• labeled examples
• scorers that run agains the LLM input/output pairs
• calibration tests for any LLM-as-judge scorers you have (sorry, I’ll probably use “scorers” and “judges” interchangeably throughout this article)
• a runner that sends examples to the LLM with the prompt we are workign on and calls scorers

In my case of Baseline, The product is a maintenance-diagnosis assistant. At the very basic level, a tenant reports an issue like “toilet won’t flush, water rising” and an LLM talks them through it. Behind the LLM sits an engine that decides which question to ask next, so the LLM’s job is to greet the tenant, present that question naturally, extract the answer as structured data, and hand it back.

That’s one system prompt, and it’s eval suite has these files associated with it:

eval/diagnostic-conversation/
  types.ts                                  # Scenario shape, response types
  calibration.ts                            # Hand-written scenarios
  scorers.ts                                # Deterministic scorers + LLM judges
  scorers.test.ts                           # Unit tests for the deterministic ones
  conciseness-calibration.ts                # Hand-labeled pairs for one judge
  conciseness-calibration.integration.test.ts
  question-presentation-calibration.ts
  gratitude-calibration.ts
  diagnostic-conversation.integration.test.ts   # The multi-model runner

The Runner?

One note before we get into the nitty gritty: I had Claude hand-roll my own one-off, mini-framework for this. There are libraries for it, Promptfoo and DeepEval, and Braintrust, which I already use for tracing, will sell you the whole platform. Ironically, I currently believe that due to the power level coding agents have achieved, much of that AI tooling and AI platform stuff doesn’t really carry its weight but that’s a separate blog post and I might have used things wrong. In the meantime I am happy with my vibe-coded solution and that it allows me to shape it exactly like I need it.

The Examples

Each example in the suite is in essence a single turn in a conversation + properties of the expected output for some of your scorers to score against. Of course even a single turn will carry all preceeding messages in that conversation, since LLMs are stateless.

The system prompt is what we are iterating on, so for your examples that must be a variable we pass in. Here’s is a trimmed example from one of my suites:

{
  label: "first turn: clogged toilet greeting + present first question",
  type: "first_turn",
  messages: [
    systemMessage(),   // the actual production system prompt, imported
    { role: "user", content:
      "Tenant name: Maria\n\nOrganization name: Acme Property Management\n\n" +
      "Issue description: Toilet won't flush. Water rising." },
    // ... assistant calls the engine, engine answers with the question to ask:
    { role: "tool", content: JSON.stringify({
        nextQuestion: {
          questionText: "Are any other drains in the home slow or backing up?",
          answerOptions: [ /* yes / no */ ],
        },
        done: false,
      }) },
  ],
  expectedBehavior: {
    shouldGreet: true,
    expectedQuestionText: "other drains",
  },
}

systemMessage() is the slot we kept for the prompt. It pulls the live prompt out of the application code rather than a copy, and the tool definitions in the runner come from the same place. So this functions a little bit like a integration test as well.

One more detail that matters: Negative expectations get written out explicitly. If a scenario expects the model to not call a tool, that’s shouldCallEngineTool: false. Not just absence of an expected call. An undefined expectation produces a scorer that silently returns “not applicable” which reads as a pass.

The suite covers the first turn, the middle of a conversation, the end of one, tool errors, and the complex edge cases. For example we have one with a tenant who says “I can’t check the breaker. I’m in a wheel chair” where the correct behavior is to skip the question instead of inventing an answer.

Mechanical Scorers

Ideally, what you want to check can be checked mechanically. That’s most reliable and cheaper. The scorers can be pretty simple in many cases. Here’s one comparing a field against a gold-labeled record:

export function categoryMatch(capture, testCase): ScorerResult {
  if (!capture.summary) {
    return { score: null, notes: "No summary produced — not scoreable" };
  }
  const actual = capture.summary.category;
  const expected = testCase.category.trim();
  const match = actual.toLowerCase() === expected.toLowerCase();
  return {
    score: match ? 1 : 0,
    notes: match ? undefined : `expected: ${expected}, actual: ${actual}`,
  };
}

A string comparison. Some are smaller than that:

export function turnCount(capture): number {
  return capture.metadata.turnCount;
}

That one reports a number instead of scoring anything, and it’s still one of the most useful things in the suite. A model that needs 19 turns to reach another model’s 6-turn answer is a worse product even when every quality score ties.

One more, for a number that should land in a range, with graceful falloff instead of a cliff:

export function costInRange(matchedOption, testCase): ScorerResult {
  const actualCost = computeOptionCosts([matchedOption])[0]!;
  const { min, max } = testCase.expectedCost;
  if (actualCost >= min && actualCost <= max) return { score: 1, notes };

  const distance = actualCost < min ? min - actualCost : actualCost - max;
  const relativeError = distance / (actualCost < min ? min : max);
  return { score: Math.exp(-COST_DECAY_K * relativeError ** 2), notes };
}

Rule of thumb: if you can write the assertion in a unit test, it belongs here and not in a LLM judge. Scores that fit this bill:

• A field matches the gold label, exactly or after normalizing case and whitespace
• A value belongs to an expected set, or a required item appears in a returned list
• A number lands in a range: cost, count, score, confidence
• Counts and lengths: turns to resolution, tool calls, retries, response length
• A required string is present — the org name, the disclaimer, the unit number
• A forbidden string is absent — the leaked answer, PII, a competitor’s name, a banned phrase
• The output parses: valid JSON, passes the schema, required keys survived
• The right tool was called, the wrong one wasn’t, and its arguments have the right shape
• Identifiers were copied through verbatim rather than invented
• Every ID it cited actually exists in the input you gave it
• It terminated, instead of looping until your turn cap saved it

Two conventions I’d push:

1. Every scorer carries a notes field explaining the failure: expected: Plumbing, actual: Plumbing Labor is what saved me from concluding “the model is bad at categories” when the real answer was a taxonomy mismatch.
2. scorers return null, not zero, when they don’t apply to a case, so they drop out of the aggregate instead of poisoning it.

Of course, mechanical scorers get unit tests, since they’re pure functions this is a no-brainer. Again, a miscalibrated scorer is worse than no scorer, because it wastes your time or gets you to ship broken prompts.

None of this needs an LLM which is good because LLM judges are non-deterministic, slow, expensive and need their own calibration. When you can, avoid them!

LLM Judges

Of course there are limits to mechanical scorers. Things like “did the LLM thank the tenatn?” is one. This is where we need LLMs as judges which starts with a scoring rubrik like this:

const prompt = `## Conciseness Evaluation

**Full Conversation Transcript:**
${transcript}

The assistant's prompt instructs it to "Keep responses concise. One or two
sentences of context, then the question."

Score 0-1 for how well the assistant's LAST response follows this guideline:
- **0.9-1.0**: One or two sentences of context followed by a clear question or
  action. No unnecessary elaboration.
- **0.6-0.8**: Slightly verbose but still focused. Maybe three sentences where
  two would do.
- **0.4-0.6**: Noticeably long. Multiple paragraphs, unnecessary explanations,
  or repeating information the tenant already provided.
- **0.0-0.3**: Monologue. The assistant lectures, provides unsolicited education
  about plumbing/HVAC/etc., or restates the entire diagnosis history.

Exceptions:
- The first turn greeting is allowed to be slightly longer.
- When the engine returns done=true, a brief summary before calling the summary
  tool is appropriate.`;

The rubric quotes the production prompt’s own instruction back at the judge, so what gets scored is adherence to one specific line of one specific prompt. Conciseness in the abstract is nobody’s question. The carve-outs at the bottom exist because the first thing an uncalibrated judge does is penalize behavior you deliberately asked for.

Note: We also have a multi-turn eval suite that uses a fake tenant with configurable personality to play through a whole diagnostic conversation to catch issues that might derail a conversation only over the course of multiple messages. Some of the examples here might make more sense in that context.

Calibrating the Judge

This is the most important takeaway in this article!

A judge is itself an LLM call with a prompt, which makes it exactly as untested as the thing it’s judging. Handing it a rubric and trusting the numbers can be an expensive mistake. Each judge needs its own suite of hand-labeled (or at least verified) pairs:

{
  label: "9: full paragraph of education before asking a question",
  transcript: `[user]: My toilet is clogged.

[assistant]: I understand you're dealing with a clogged toilet, and I want to
help you get this resolved as quickly as possible. Toilet clogs are actually
one of the most common plumbing issues we see [...] To help me understand the
scope of the problem, I'd like to ask you a question. Are any other drains in
your home running slowly or backing up?`,
  expectedScore: 0.15,
}

Fo this above example, we have twelve cases, spread deliberately across the range. A calibration set holding only “perfect” and “terrible” examples can’t tell you which judges compress the middle, and the middle is where all the interesting product decisions live.

Scoring a judge can look like this:

const DECAY_K = 4;
const calibrationScore = (actual, expected) => Math.exp(-DECAY_K * (actual - expected) ** 2);
function geometricMean(values: number[]): number {
    if (values.length === 0) return 0;
    const logSum = values.reduce(
        (sum, v) => sum + Math.log(Math.max(v, 1e-10)),
        0,
    );
    return Math.exp(logSum / values.length);
}

I like using Gaussian decay because it means being off by 0.1 barely punishes, being off by 0.3 scores 0.70, off by 0.5 scores 0.37. The geometric mean lets one catastrophic miss tank the whole model, which is the behavior I want. A judge that scores a 0.15 case as 1.0 is unusable however good its average looks!

Then you run that across candidate judge models, exactly like you’d run the product prompt across candidate models:

It’s a little surprising to what degree different judges work better with different models. Haiku is our best conciseness judge at 0.963. Gemini Flash wins question presentation scoring at 0.926 and gratitude scoring at 0.973. Assuming one model is simply “the good judge model” can cost you either in quality or in $$$ and again gets you lock in.

There’s a corollary. When every candidate judge disagrees with your expected score, your label is probably wrong. Candidate consensus turned out to be the best label linter I have. If judges disagree a lot on an example, the example might be unclear (potentially very good) or your scoring rubrik is unclear (bad!).

The Runner

The interface I settled on is a list of candidate models, a trial count, and every scenario run against each one, scored by the deterministic functions and the judges together, with results dumped to JSON for offline poking:

const MODELS_TO_TEST = [
  { id: "anthropic/claude-sonnet-4-6", temperature: 0.1 },
  { id: "openai/gpt-5.5",              temperature: 0.1 },
  { id: "google/gemma-4-26b-a4b-it",   temperature: 0.1 },
  { id: "anthropic/claude-haiku-4-5",  temperature: 0.1 },
];
const TRIALS = Number(process.env.EVAL_TRIALS ?? 3);

Two things I learned here:

1. Since LLMs aren’t deterministic you want to run each example multiple times.
2. Play around with temperature settings. Especially for judges, lower is probably better.

Where the Examples and Labels Come From

The hard part is sourcing good examples and labels.

Small Numbers are Fine

Evalsuites are much closer to writing unit tests. You don’t write ten thousand unit tests for a function. You write one per branch, one per equivalence class, one per boundary, and then you stop, because the eleventh test of the same branch tells you nothing the first one didn’t. Eval examples work the same way. They cover types of case, and once two examples exercise the same behavior they stop providing value. So my suites are usually in the 12-25 example range. If your problem space is much larger, you of course might need much more examples than I have, just like unit test suites for complex code can grow a lot. However, the key point stands: Example diversity and coverage over simple quantity.

The conciseness set is 12 pairs spread on purpose: five clearly-good, three mixed, three bad. A thirteenth clearly-good example is free and worthless. The first mixed one carries all the information, because the middle of the range is where models actually differ.

Writing tests is a design activity, and the friction of writing one is often what tells you the interface is wrong. Labeling behaves identically. The friction of deriving a label by hand is how you find out your definition of correct is ambiguous and you need to do some more thinking.

Of course this is all only true for eval suites. If you want to do something like fine-tuning, you again need as many examples as you can get your hands on.

Where to get Examples

Examples and labels can come from a variety of sources. The ideal source is examples that were done in a manual fashion by human experts. This might be from legacey processes you are trying to automate.

You can create them yourself from scratch or have an LLM generate them. If you go the LLM route, I recommend having a panel of models from different model families review the choice of examples while giving clear guidance that additional examples must provide real, marginal value.

For labels, it’s prudent to first ask what kind of label it is:

If the label is a domain fact, go get the expert. Our main multi-turn test-case bank is a CSV written by a domain expert. It has expected root causes, expected cost, etc. I am a software developer, I cannot label that. Models can make a decent attempt but a real expert is better in this case.

If the label is a product policy, label it yourself. “Does answering this question require extra work for which the agent should thank the tenant?” is a decision about product intent.

You can safe some work by having a panel of models pre-label for you. You still have to review all the work! However, it safes you time typing at scale and might help keep you from slipping up 50 labels into the process.

I’ve found it also to be efficient to get something that works at all ready to play around with locally on in staging and play through scenarios there. Cases that could have gone better are quickly converted into new, labeled examples that I know provide additional value since it tripped up the current setup.

Step 2: Optimization and Model Selection

Once the suite is there I generally do three steps:

1. Single run of the eval suite with naive prompt against a 5-10 candidate models.
2. Prompt tuning in an optimzation loop
3. Model bake-off and pinning

Candidate selection

I will hand-pick 5-10 models from a variety of families that I expect to form well on the task. I usually take some models that did well on similar tasks but also take a look at what’s good these days on OpenRouter and LMArena and try to pick a mix of cheap and more expensive models.

Once the initial candidates are chose, I run the eval suite once for each model in the candidate set with a naive prompt. I pick 2-3 models that look promising and are from different families and take them to he optimization loop

The optimization loop

With the eval suite in place and likely top models chosen, we can bring the prompt we are trying to improve and the loop turns out to be simple. It’s less magical than “prompt optimization” sounds. There’s no clever search:

1. Run N trials against the current prompt and model.
2. Read the failure examples, not the rates.
3. Change one thing in the prompt to address the class of a specific failure you read.
4. Re-run and compare.

Step 2 is the crux. As a concrete example in my project we have a tool that that pulls factual claims out of a finished conversation. Its hard invariant: extract only what the tenant said, never the assistant’s own hypotheses.

Every prompt improvement in that suite came from reading a specific miss:

• Extracted claims like “yes” with no referent, because the tenant answered a question the extractor couldn’t see → added a rule requiring each claim to carry its referent.
• Extracted “the door frame needs replacing” from an image description → added an explicit evidence-vs-conclusion boundary.
• Extracted a hypothesis the assistant had floated in a leading question → added a confirmed-observable rule.

All these came from the runner printing the offending claim next to its fixture. If my eval only prints aggregates, I cannot do this. You just see it’s bad but don’t know why.

I’ve had great success with having a LLM do the actual prompt optimization and having a reviewer LLM keep it honest about not having test data leak into the prompt directly (we can still overfit our prompt just like we could overfit our old-school ML models!).

Step 3: The Bake-Off, and What Pinning Looks Like

In this step all models form step 1 come back but we run against the optimized prompt and multiple runs of the suite for each model (remember LLMs aren’t deterministic!).

Out of this you should get geometric mean along the relevant scores, speed and error rate. Based on this we can pick the moel we think is best for the task and ideally a fallback model.

If the chosen models turn out to not be the ones that went through the prompt optimization loop there might be room to do a few more loops with the chosen model. However, I’ve found that usually the scores were close to perfect at this point for the problems I work on.

In addition to pinning the top model and its fallback, it’s also useful to make sure you note the “why” of the model choice and keep the actual raw data somewhere. This can be handy in hindsight.

Simple Fallbacks, Which Your Gateway Probably Already Supports

Now the part I claimed in the original post: you should already have a fallback from a different lab or provider, wired up and tested.

OpenRouter makes is simple to just pass a models array in priority order and OpenRouter fails over automatically:

{
  "models": ["openai/gpt-5.5", "anthropic/claude-opus-4.6"],
  "messages": [ ... ]
}

That triggers when a model is down, rate-limited, fails a context-length check, or trips a moderation filter. Provider-level failover within a single model is on by default. It’s a per-request parameter, so each call site can carry its own chain, which matters more than it sounds: the right second choice for your summarizer is rarely the right second choice for your judge, and the bake-off already told you the ranking rather than just the winner.

Ensuring Timely Responses, Not Just Successful Ones

There’s one failure the OpenRouter config doesn’t cover, and it’s the common one: a provider that hasn’t failed, but is slow now. Failover triggers on errors, providers are tried in sequence rather than raced, and a model returning a perfectly valid empty completion doesn’t trip anything at all.

The closest native controls are preferred_max_latency and preferred_min_throughput, percentile cutoffs over a rolling five-minute window, plus sort: "latency". They help, and they don’t solve this. They’re statistical and predictive. They change which provider you land on based on its recent record. So once you’re eight seconds into a request that’s going badly, nothing in that config does anything for you. There’s no server-side request timeout to reach for. They also work on the wrong axis, selecting among providers serving one model, where the thing I want is a race between two models from different labs.

So this is the part I built myself. What works for me is a hedge, in the sense of Dean and Barroso’s “The Tail at Scale”:

The primary gets T milliseconds. If it hasn’t produced anything by then, the fallback launches alongside it, and whichever returns real output first wins. The loser is aborted. Worst case the user waits a bit over T, and you pay for one extra call on the tail of your latency distribution.

Around it sits ordinary retry logic, with one rule that took an incident to learn: a request timeout is not retryable on the same provider. You already waited the full budget, so re-running it just doubles the user’s wait. Rate limits and 502/503/529 get one retry with jittered backoff. Timeouts go straight to the hedge.

Tradeoffs are baked into that design, so you can make them deliberately:

• The fallback costs money on the tail. Set T too aggressively and you’re paying twice for a meaningful share of traffic.
• The hedge is on the non-streaming path. Streaming has retry but no hedging, because racing two streams and picking a winner mid-token is a harder problem than I’ve needed to solve.
• Both models still go through one gateway. OpenRouter remains a single point of failure. That’s a deliberate trade off for now. It’s something I’d like to make more flexible in the future, but for now it’s good enough.

Once Real Users Show Up

If you catch turns that didn’t go well you can naturally start enhancing you eval suite and use it to make your prompt more robust.

Tools like Braintrust allow you to run your evals against a set percentage of live requests. In fact you can build your suite up on Braintrust and pull examples straight in from live logs. I found that to not always work as I want and get expensive fairly quick. So I’ve had better luck with just pulling down sanitized turn records and run scorers against them locally.

Back to Lock-In

I built non of this with the intention to avoid lock-in. I built this because I believe in test-driven development and eval suites are the test suite for your model configs and prompts. For me it was a happy side effect that this allowed me to avoid lock-in to a particular model from a particular vendor I didn’t ralize till I started reading about avodoiding vendor lock-in as one of the reasons to use open-weight models.