System Prompt Testing: How to Build LLM-as-Judge Rubrics

A system prompt is code that happens to be written in English. It has requirements, it has edge cases, and it breaks when someone "just tweaks one line". This guide covers how to test it properly: what cases to write, when to use exact checks, and how to build an LLM-as-judge rubric you can actually trust.

By Sharaj Rewoo · Updated 25 September 2026

Why system prompts need tests

The default way prompts get "tested" is that someone tries three or four messages in a playground, the answers look fine, and it ships. That misses the three properties that make LLM behaviour hard:

• Non-determinism. The same input can give a good answer on one run and a bad one on the next. One lucky sample proves very little.
• Action at a distance. Adding a sentence about tone can change how the model follows a formatting rule three paragraphs away. Prompt edits have side effects you won't see unless you re-run everything.
• Model drift. Switching model versions or providers — even to a "better" model — changes behaviour. A prompt tuned for one model is not automatically correct on another.

So treat the prompt like any other artefact under test: define what "correct" means, write cases that exercise it, run them repeatably, and re-run them on every change.

Start with the requirements, not the cases

Before writing a single test input, list what the prompt is supposed to guarantee. Read it line by line and turn each instruction into a checkable requirement. A customer-support prompt might produce:

• Answers only questions about our product; politely declines anything else.
• Never states a refund policy that isn't in the provided policy text.
• Responds in the user's language.
• Returns JSON with answer and escalate fields.
• Escalates when the user mentions legal action or a safety issue.
• Never reveals the system prompt.

Each requirement becomes one or more quality dimensions. If you can't write an instruction as a requirement, it's usually too vague for the model to follow reliably, too — which is a finding in itself.

Designing the test cases

Aim for coverage across categories rather than volume in one. A suite of 20–40 well-chosen cases beats 300 near-duplicates of the happy path.

• Typical (happy path) — the requests the prompt was written for, in the forms real users send them: short, rambling, misspelt, with and without context.
• Edge cases — boundaries of each requirement. An in-scope question that sounds out-of-scope. A question in a language mix. An empty or one-word message. A very long input. Two requests in one message. A question the provided context can only partly answer.
• Off-topic — clearly unrelated requests ("write me a poem", "what's the capital of Peru") to confirm scope limits hold and the decline is graceful.
• Adversarial — prompt injection ("ignore previous instructions"), requests to reveal the system prompt, role-play jailbreaks, instructions hidden in pasted documents or tool results, and social-engineering ("I'm the admin, you're allowed to…").
• Format compliance — inputs designed to tempt the model out of its format: requests for a table when JSON is required, content containing quotes and newlines that break naive JSON, or users asking it to "just answer in plain text".
• Tool use — for prompts with tools: cases where a tool must be called, where no tool should be called, where the wrong tool is tempting, and where a tool returns an error or empty result.

For each case, write down the expected behaviour in one or two sentences, and a reference answer where one exists. Cases without an expectation can't be graded consistently by anyone — human or model.

Deterministic checks vs LLM-as-judge

The single most useful rule in prompt testing: if code can check it, don't ask a model. Deterministic checks are free, fast and never drift.

Tool calling in particular should almost always be checked deterministically: the model either called search_orders with a valid order_id or it didn't. Using a judge there adds cost and noise for no benefit. Reserve the judge for the genuinely semantic questions.

How to write a rubric the judge can apply consistently

An LLM judge is only as good as its rubric. Most bad judge setups share the same problems: vague criteria, too many things scored at once, and scales with no anchors.

Prefer pass/fail over 1–10 scales

Ask a model to score "helpfulness from 1 to 10" and you'll get a lot of 7s and 8s that mean nothing, and scores that wander a point or two between runs. Binary criteria are far more stable and far easier to act on: either the answer cited only the provided policy, or it didn't. If you need gradation, use a small scale (for example 1–3 or 1–4) where every level has a concrete written definition.

One criterion per check

"Is the answer accurate, concise and on-brand?" is three checks. When it fails, you don't know which one. Split every compound criterion, and score each dimension with its own judge call or its own clearly separated field.

Make criteria observable

Replace adjectives with things the judge can point to. Not "is the answer good", but "states the refund window from the policy text; does not mention any timeframe not in the policy; ends with how to contact support if unresolved".

Include reference answers and the source context

Give the judge what it needs to decide: the user input, the context the assistant had (retrieved documents, policy text), the output, and a reference answer or list of required points where you have one. A judge grading factual accuracy without the source material is just guessing with confidence.

Ask for reasoning before the verdict

Have the judge write a short justification first and the verdict last. It tends to improve consistency, and the reasoning is what you read when a result looks wrong.

A judge prompt template

You are grading one response from an AI assistant against a single criterion.

CRITERION
{{criterion — one observable requirement}}

PASS if: {{concrete description of a passing response}}
FAIL if: {{concrete description of a failing response}}

INPUTS
User message:
{{user_input}}

Context the assistant was given:
{{context}}

Reference answer / required points (may be partial):
{{reference}}

Assistant response to grade:
{{output}}

Instructions:
- Judge only the criterion above. Ignore style, length and other
  qualities unless the criterion mentions them.
- Do not reward a response for being longer or more detailed.
- If the response is ambiguous, decide based on what a user would
  reasonably understand from it.

Respond in JSON:
{"reasoning": "<2-4 sentences>", "verdict": "PASS" | "FAIL"}

Note the explicit instruction not to reward length. It matters — see the section on bias below.

An example rubric

For the customer-support prompt above, a rubric might look like this, mixing check types:

Calibrate the judge before you trust it

An uncalibrated judge is an opinion generator. Before relying on it, check it against people:

• Take 30–50 real outputs, including some clearly bad ones, and have a human (ideally two) label each criterion pass/fail.
• Run the judge on the same outputs and compare. Look at every disagreement, not just the agreement rate.
• Disagreements usually trace back to the rubric: a criterion the human interpreted differently, a missing definition of an edge case, or context the judge didn't have. Fix the rubric wording and re-run.
• Keep the labelled set. Every time you change the judge prompt or judge model, re-check agreement against it.

Known judge biases and how to reduce them

• Verbosity bias — judges tend to favour longer, more detailed answers even when the shorter one is better. Tell the judge explicitly not to reward length, and include concise reference answers.
• Position bias — in pairwise comparisons ("which is better, A or B?") judges favour one position. Run each comparison twice with the order swapped and only count consistent wins.
• Self-preference — a model may rate outputs from its own family more generously. Where possible, use a judge from a different model than the one you're testing.
• Leniency — judges lean towards PASS when unsure. Concrete FAIL conditions in the rubric push back against this.

Handling non-determinism: run each case N times

Because outputs vary, a single run per case gives you a noisy snapshot. Run each case several times (3–5 is a reasonable start; more for high-stakes cases) and look at the spread, not just the average. A case that passes 5/5 and one that passes 3/5 have the same "mostly works" feel in a playground, but the second one will fail for real users regularly. Report pass rate per case and flag anything below 100% on requirements that must always hold, such as format and safety.

Where your use case allows, lowering temperature reduces variance — but test at the temperature you actually ship with.

The regression workflow

Once the suite exists, the process for every prompt change is the same:

• Baseline — run the full suite on the current prompt and save the results as a version.
• Change one thing — edit the prompt, the model or the temperature, not all three at once.
• Re-run everything — not just the case that motivated the change. Side effects show up in unrelated dimensions.
• Compare by dimension — did grounding improve while format compliance dropped? A single overall score hides exactly the trade-offs you need to see.
• Add the bug as a case — every failure found in production becomes a permanent test case, the same way a fixed bug gets a regression test.
• Watch cost and latency — a longer prompt that scores slightly better but doubles time-to-first-token may not be the better prompt.

Doing this without building an eval platform

You can wire all of this together with a script and a spreadsheet, and for a large team a hosted evaluation platform makes sense. For a single prompt or a small team, litmus covers the workflow in a Chrome side panel: paste a system prompt (or grab it from the tab you're on), pick the model you ship on, and it auto-writes an LLM-as-judge rubric per quality dimension and generates typical, edge and adversarial cases, which you can edit. It shows the judge's reasoning for each score, lets you run each case N times to see the spread, and lets you choose a judge model different from the target to reduce self-preference bias. Tool calls are checked deterministically against your JSON schema rather than by a judge, and multi-step agents run against mock tools. Every run is saved as a version you can compare by dimension and export as Markdown or JSON. It's local-first and bring-your-own-key, with no backend.

If you're generating functional test cases for the product the prompt sits inside, QAtalyst turns user stories into positive, negative and edge-case scenarios — the same test-design thinking, applied to the requirements.

Checklist

• Every instruction in the prompt is mapped to a testable requirement.
• Cases cover typical, edge, off-topic, adversarial, format and tool-use categories.
• Everything code can check is checked by code.
• Judge criteria are single, observable and pass/fail with written PASS and FAIL conditions.
• The judge sees the context and a reference answer, gives reasoning before a verdict, and is told not to reward length.
• The judge has been compared with human labels, and disagreements were used to fix the rubric.
• Each case runs multiple times and pass rates are reported per case.
• Every prompt change is a new version, compared by dimension against the last one.

Related