Skip to main content

Read the results

Once a run finishes, the results show how your system scored against each criterion, the issues to address, and how far you can trust the judges' grades.

Scores​

Each criterion has a pass rate: the share of graded answers that passed. Answers a judge marked as not applicable, and answers that errored, don't count. The run's overall score is the share of all graded answers that passed.

A pass rate based on only a few answers can change a lot from one run to the next. Treat it with care until a criterion has more graded answers.

To see whether a change helped, choose Compare against run and pick an earlier run.

Issues​

Key issues to address lists the most important problems; All issues lists every one. Each issue is graded:

  • Critical: a criterion whose pass rate is below its passing score.
  • Major: a weakness Litmus found in the graded answers.

Litmus writes the issues from the graded answers. If they look out of date, choose Regenerate analysis. Check each criterion's pass rate against its passing score too, not just the issues list.

For each issue, decide what to do:

  • Change your system, then mark the issue as resolved. It shows as Resolved — unverified until a later run checks it again, then as Re-verified with the run and date.
  • Accept it as residual risk, with a note explaining why. You can't accept a Critical issue.
  • Re-open an issue if you change your mind.

Only the system's owner can record these decisions.

How far to trust the grades​

Judges are AI models, so they can be wrong. How much can you trust these grades? helps you check them.

  1. Choose Start spot-checking, and grade a sample of answers yourself. Litmus compares your grades with the judges'. A meter shows how far you've got, from Provisional to Fair to Solid, against a target of about 5 answers for each criterion in the run.
  2. Each judge is shown as Not checked yet, Not enough to tell yet, or Below the bar on the criteria where it disagreed with you too often. Litmus never calls a judge reliable: passing the bar only means it wasn't shown to be unreliable.
  3. If a judge is below the bar on a criterion, choose Stop counting it there, and give a reason. Litmus recalculates the scores without it, and keeps both readings.

Test again​

After you change your system, start a new run and compare it with the last one. Litmus doesn't record which version of your system a run tested, so keep a note.

What good looks like​

  • No open Critical issues.
  • Every Major issue resolved and re-verified, or accepted with a clear reason.
  • You've spot-checked the judges, and none is below the bar on the criteria that matter most.

If you're stuck​

  • You can't save an issue decision: only the system's owner can. Ask them to record it.
  • You can't accept an issue as residual risk: it's Critical. Change your system and re-run instead.

Still stuck? Ask us using the onboarding form⁠ (opens in a new tab), and say which step you're on.