Run the tests
A run sends your dataset's test cases to your system, then the judges grade each answer against your criteria.
Before you start
- Your system's connection check has succeeded. If you've changed the connection settings since, run the check again.
- Every criterion has a judge.
- You have a finalised dataset version.
Start a run
Choose New run, then set up the run:
- Who runs the tests. Choose Litmus runs them to have Litmus call your system. Or, if you've already run the test cases yourself, choose I already ran them and upload the answers. See upload your own results.
- Dataset. Choose the finalised version to run.
- Scope.
- Smoke runs a small sample, about one test case for each criterion and persona. It's a quick way to check everything works before a full run.
- Full runs every test case.
- Judge models. By default, each criterion uses its own judge. To grade every criterion with the same judges, tick up to 3 models. Each answer then passes if most of them pass it; a tie counts as not passed.
Check the summary: the number of test cases and safety probes, the calls Litmus will make to your system, the AI judgements, and an estimate of the time it will take. Then choose Run … test cases.
The run shows as Starting…, then Scoring…, and finishes as Completed, Error or Cancelled. If a run ends in an error, see troubleshooting.
Upload your own results
If you tested your system yourself, Litmus can grade the answers you collected, without calling your system. Your system still needs a successful connection check first.
- Choose I already ran them, then download the template. It's filled in with your dataset's test cases.
- Add your system's answer to each test case, then upload the file.
The file can be CSV, JSON or Excel, up to 20 MB:
| Column | Required? | What to put in it |
|---|---|---|
input (or question) | Yes | The test case. |
output (or answer) | Yes | Your system's answer. Rows without one are left out. |
case_id (or id) | No | The test case's ID, from the template. |
persona, type | No | As in the dataset. |
verdict: <criterion name> | No | Your own pass or fail for that criterion, if you have one. |
JSON files can be a list of objects with these fields. A run from uploaded results is labelled Self-reported results wherever its scores appear. You can't include the platform's standard safety criteria in an uploaded run, because Litmus can't send its safety probes to your system.
What good looks like
- You run a smoke test first, and a full run once it looks right.
- You keep a note of which version of your system each run tested. Litmus doesn't record it.
If you're stuck
- "Run the connection check on the Connect step before launching": Litmus hasn't confirmed it can reach your system. Changing the address, method, authentication or request template clears an earlier check, so run it again.
- The run ends in an error, or many answers are errors: see troubleshooting runs.
Still stuck? Ask us using the onboarding form (opens in a new tab), and say which step you're on.