Litmus Onboarding Guide

Getting Started

Step 1: Sign Up (for Beta)

• Fill up the Litmus and Sentinel interest form: https://go.gov.sg/litmus-beta-test
• We will provide you with login credentials and access instructions after we have created your account.
• When we fully launch in April, there will be Techpass integration that allows you to set up a formal tenant and include necessary team information.
• We will provide dedicated support to all beta users to migrate to full tenancy ahead of launch.

Step 2: Clone Demo Repository

• Clone our demo repository:

git clone https://github.com/dsaidgovsg/aiguardian-demo-tenant

• You should see the following files:
  • benchmark.py: The main script to run tests.
  • README.md: Instructions for running tests.

Step 3: Run Tests

You have two options to run tests:

Option 1: Local Testing

1. Install Python

• Download Python 3.11 or later from python.org
• Verify installation by opening a terminal and running:

python --version

2. Run Tests

python benchmark.py <base_url> <run_name> <endpoint> <test_suites> <num_of_prompts> <litmus_api_key>

Option 2: CI/CD Integration (GitHub)

1. Set Up GitHub Actions Workflow

• The project includes a GitHub Actions workflow file (.github/workflows/litmus-test.yml). This file configures the CI pipeline to run safety checks automatically on each push and pull request.

2. Customize Safety Tests

• Modify on setting to customize triggers specific to your application.
• Update the parameters of the Run Litmus Tests step using appropriate environment variables to fine-tune the test criteria.

  - name: Run Litmus Tests
    uses: dsaidgovsg/aiguardian-litmus-test@<version>
    with:
      base_url: ${{ vars.LITMUS_BASE_URL }}
      run_name: ${{ github.run_id }}-${{ github.run_attempt }}
      endpoint: ${{ vars.ENDPOINT }}
      cookbook: ${{ vars.COOKBOOK }}
      num_of_prompts: ${{ vars.NUM_OF_PROMPTS || '1'}}
      api_key: ${{ secrets.LITMUS_KEY }}
      debug_mode: ${{ vars.DEBUG_MODE || 'false' }}

3. Push Changes to GitHub

After setting up the workflow and configuring the tests, push your changes to your GitHub. The safety tests will automatically run, with results visible under the Actions tab in your repository.

Configuration Parameters

Name	Description	Required	Default
base_url	The base URL of the Litmus API server.(https://litmus.stg.aiguardian.gov.sg/)	Yes	-
run_name	A unique name for the test run. Best created using a composite workflow run unique ID.	Yes	-
endpoint	The model endpoint to be tested.	Yes	-
test_suites	A comma-separated string of test suite names. Use aiguardian-baseline-tests for our baseline tests.	Yes	-
num_of_prompts	The number of prompts to use for testing. Value of 0 means run all prompts.	Yes	-
api_key	API key provided by the AIGuardian team during onboarding.	Yes	-

Note: Gitlab template will be provided in the future.

Step 4: View Results

• Access the Litmus dashboard to view your test results
• Review test progress in real-time and identify failing prompts
• Beta launch requirement: At least 90% of tests must pass to register a "pass"
• Full launch customization: Tailor custom thresholds per application in consultation with NAIG
• When tests fail, corresponding corrective measures (e.g., guardrails) will be recommended
• View Tests on Vanilla Base Models:
  • We have pre-tested major Vanilla models (e.g., GPT-4o, o1, Gemini-2.0-flash)
  • View results in the Vanilla Models tab
  • If you have a model you'd like us to test, contact us at aiguardian@tech.gov.sg

Troubleshooting

Test Failed

• Check the detailed test report for errors and issues
• Ensure configurations (devices, browsers) are correctly set
• If tests fail for specific browsers, verify compatibility or use manual testing to isolate the problem

Test Not Starting

• Ensure your account setup is complete and subscription is active
• Verify the accessibility of the URL or app under test

API Integration Issues

• Verify your API key and ensure you're using the correct endpoint
• Ensure proper authorization headers are included with requests
• For additional support, contact our team

Contact Us

For any questions or support inquiries, feel free to reach out to our team: aiguardian@tech.gov.sg