Troubleshooting connections and runs

For whoever connects your system. The settings themselves are explained in connecting your system.

The connection check

Run connection check sends one request to your system, with the text Connection test from Litmus. as the test case, and waits up to 30 seconds for a reply. Any 2xx reply passes. Changing the address, method, authentication or request template clears an earlier check, so run it again after any change.

What you see	What to check
"Connection timed out"	Your system didn't reply within 30 seconds. Check that it's running and not overloaded.
"Could not connect to the endpoint"	The address is wrong, or a firewall is blocking Litmus.
"…resolves to a non-public address…"	Your system must be reachable at a public internet address. Litmus can't reach systems that are only on an intranet or private network.
"unsupported URL scheme" or "malformed URL"	The address must be a full URL starting with https://.
"HTTP 301", "HTTP 302" or another 3xx	Litmus doesn't follow redirects. Use the final address.
"HTTP 401" or "HTTP 403"	Your system rejected the authentication. Check the token, key or header name, or the signing secret.
"HTTP 404" or "HTTP 405"	Check the address, and that your system accepts POST.
"The authentication configuration is invalid or incomplete"	Something is missing in the authentication settings. If you chose HMAC signing, generate the signing secret first.
"The request could not be built"	The request body template isn't valid JSON once the test case is filled in.
"— nothing matched the response template —"	The check passed, but the answer path doesn't match your system's reply. Compare the path with the reply shown.
"Connection test could not be run" or "…due to an internal error"	The problem is on Litmus's side. Try again in a few minutes, then ask us.

Request template mistakes

• The placeholder must be a whole JSON string value: "question": "{{testcase}}" works, but in "question": "Q: {{testcase}}" the placeholder is sent as it is.
• Runs only send POST. If your system only accepts GET, Litmus can't run tests against it.
• Use a simple path for the answer, such as $.choices[0].message.content or $.response.

During a run

Litmus sends up to 5 requests a second to your system, or fewer if you set a lower limit. For each test case:

What happens	What Litmus does
Your system doesn't reply within 10 seconds	Tries again, up to 3 attempts in all. If your system needs longer, use the callback option: reply 202 Accepted and post the answer within 5 minutes.
Your system replies 5xx, the connection fails, or it redirects	Tries again, up to 3 attempts in all, then marks the test case as an error.
Your system replies 429, or 503 with a Retry-After header	Slows down and tries again later, without using up attempts. It waits as long as Retry-After asks, up to 5 minutes.
Your system replies with another 4xx	Marks the test case as an error straight away.
A content filter blocks the request, or a firewall returns a 403 web page	Counts it as your system refusing, and grades it.

That last row matters for safety criteria: a refusal often passes them. If a firewall in front of your system blocks Litmus's test cases, your results can look better than your system really is. Make sure Litmus's requests reach your system.

If you use the callback option

• Reply with exactly 202 Accepted. Any other reply, including 200, counts as a failure and is tried again.
• Then POST {"job_id": "…", "answer": …} to the callback URL that came with the request, with the job_id from the request. Litmus reads answer as it is; the answer path isn't used.
• Sign the callback the same way Litmus signs its requests (see below).

Litmus replies to your callback	Meaning
202	Received.
200	Already received.
400	The body isn't valid JSON, or the job_id is missing or invalid.
401	The signature is missing or doesn't match.
404	Litmus doesn't recognise the job_id.
409	That test case has already finished.

Verifying Litmus's signatures

With HMAC signing, each request from Litmus carries two headers:

• X-Evaluate-Timestamp: the time it was sent, in seconds since 1 January 1970 (UTC).
• X-Evaluate-Signature: sha256= followed by the hex HMAC-SHA256 of the timestamp, a full stop, and the raw request body, using your signing secret as the key.

To check a request, compute the same HMAC over <timestamp>.<raw body> and compare it with the signature. Use the raw bytes of the body, before your framework parses it.

Litmus shows the signing secret once, when you generate it. Generating a new one stops the old one working straight away.

Still stuck? Ask us using the onboarding form⁠ (opens in a new tab), and say which step you're on.