Build a dataset
A dataset is the set of test cases Litmus sends to your system. In this step you build one, review it, and finalise a version to run.
Add test cases
Choose New dataset, then add test cases in any of three ways:
- Generate questions: Litmus writes test cases in your personas' voices, about your contexts and your knowledge base.
- Import existing: upload test cases you already have, as a CSV file. See the import format.
- Add your own: write test cases one at a time.
Generated and imported test cases wait in a review queue. Approve the ones that fit and reject the rest, one at a time or with Approve all and Reject all.
Each test case has:
| Part | What it is |
|---|---|
| Text | The input Litmus sends to your system. |
| Persona | Who is asking. |
| Type | Typical (an everyday request), edge (unusual but legitimate) or adversarial (trying to make the system misbehave). |
| In scope | Whether it's something your system should be able to answer. |
| Criteria | The criteria this test case checks. Only answers to tagged test cases are graded against a criterion. |
Finalise a version
Before you can finalise, the dataset must cover your criteria:
- each criterion needs at least 5 test cases tagged to it;
- each Safety and Security criterion needs at least 1 adversarial test case.
The review shows how the test cases break down by persona and type, with a coverage grid so you can spot gaps. When it's ready, choose Finalise. A finalised version is frozen, and it's the one runs use. To change it later, edit the draft of a new version and finalise that.
To keep a copy, choose Export CSV.
Safety probes
If your criteria include the platform's standard safety criteria, Litmus also sends your system its own set of adversarial prompts on hatefulness, misconduct, toxicity, sexual content, self-harm and violence. They appear as a Platform-managed card, and run alongside your dataset. You don't need to add them yourself.
The import format
Choose Download template for a CSV with the right columns, with a row for each of your criteria. You need at least one criterion before you can download it.
| Column | Required? | What to put in it |
|---|---|---|
text | Yes | The input to send to your system. Litmus also accepts input, question or prompt as the column name. |
type | No | typical, edge or adversarial. Blank means typical. |
persona | No | The name of one of your personas, exactly as it appears in Litmus. |
in scope | No | yes or no. Blank means unknown. The template leaves this column out, so add it if you need it. |
criteria | No | The names of the criteria the test case checks, separated by ; or |. A row that names a criterion your system doesn't have is rejected. A row with no criteria is imported untagged, and waits until you tag it. |
Column names aren't case-sensitive, and Litmus ignores any other columns. The file must be UTF-8 encoded, and no larger than 20 MB.
What good looks like
- Every criterion has well over the minimum of 5 test cases, so one odd answer doesn't swing its pass rate.
- The test cases include adversarial ones for every Safety and Security criterion, and edge cases for the situations your system finds hard.
- The test cases sound like your real users, not like a test.
If you're stuck
- "Finalise a dataset version to continue": you need a finalised version before you can run tests. Check the coverage grid for criteria that need more test cases.
- "Failed to decode CSV file": save the file as UTF-8 CSV and import it again.
- "The CSV needs a column named 'text'": name the column with the input
text.
Still stuck? Ask us using the onboarding form (opens in a new tab), and say which step you're on.