TRCLI in CI/CD pipelines

Why TRCLI fits naturally in CI/CD pipelines

TRCLI is a command‑line client for TestRail that automates uploading test results into your TestRail instance via its API. It fits neatly into CI/CD workflows because it: 

• Streamlines automated reports from JUnit-style XML (e.g. from Playwright, Selenium, etc.) into TestRail.
• Supports config-based automation and overrides, enabling predictable, repeatable upload behaviour across environments.
• Is open source and lightweight - a perfect match for scripting in CI stages.
• Allow failures so result upload always runs - even if tests fail.
• Artifacts shared between jobs.
• Use pipeline environment variables for credentials and metadata

Prerequisites

• A TestRail instance with API access enabled
• Your automation framework configured to output JUnit-style XML
• TRCLI installed (via pip install trcli) in your CI environment
• Credentials stored securely (env vars or CI secrets)

Key parameters

stages:
  - test
  - upload-results

test-job:
  image: node:latest
  script:
    - npm install
    - npx playwright test --reporter junit
  artifacts:
    when: always
    paths: [test-results/]
  allow_failure: true

upload-results:
  image: python:latest
  dependencies: [test-job]
  script:
    - pip install trcli
    - trcli -y \
        -h https://your.testrail.io/ \
        --project "My Project" \
        --username $TR_USERNAME \
        --password $TR_API_TOKEN \
        parse_junit \
        -f test-results/junit-report.xml \
        --title "CI Test Run - $CI_PIPELINE_ID" \
        --run-description "GitLab pipeline $CI_PIPELINE_URL" \
        --close-run

Best Practices

• Use config files (config.yml) to centralize settings (project, suite, timeout, batch size).
• Favor environment variables for credentials - keeps secrets out of version control. TRCLI prioritizes CLI flags, then config file, then env vars.
• Enable retry/throttling behavior to handle TestRail API rate limits - it’s built into TRCLI.
• Batch size tuning: Default is 50 records per request, but can be adjusted for large scale using batch_size.
• Version pinning: Use consistent TRCLI version across pipelines to avoid breaking changes - especially when combined with gurock/trcli-action on GitHub.

Troubleshooting tips

• Check exit code: 0 means success; non-zero indicates errors in upload or auth.
• Enable --verbose or --verify for detailed logging.
• If failing due to missing cases, switch on --auto_create_cases_yes or ensure case matcher is set properly (e.g. auto, name, property).
• Monitor for API throttling - logs will reflect retries automatically.

Starter checklist for CI/CD Integration

✅ Confirm automation framework exports JUnit XML output.

✅ Add TRCLI install step (e.g., pip install trcli) in CI job.

✅ Secure credentials via CI environment variables/secrets.

✅ Use parse_junit command with appropriate flags and envs.

✅ Add --run-description linking back to CI run.

✅ Use --close-run and optionally --run-id for run reuse.

✅ Adjust batch_size, timeout, and threading as needed.

✅ Optional: map custom statuses via --case_result_statuses.

Setup a config file for CI/CD pipelines

Using a config.yml file with TRCLI simplifies your CI/CD pipeline by moving static configuration out of your command calls. This keeps your pipelines cleaner, more secure, and easier to maintain.

• Cleaner commands: Remove repeated arguments like --host, --project, etc.
• Better secrets management: Refer to environment variables in your config instead of hardcoding them.
• Portability: Use the same config file in local scripts, GitHub Actions, GitLab, Jenkins, or any CI tool.
• Environment-specific control: Load different config files per environment (e.g. config.staging.yml, config.prod.yml).

Example:

- name: Upload test results to TestRail
  uses: gurock/trcli-action@v1
  with:
    args: >
      -y
      --config .testrail/config.yml
      parse_junit
      -f test-results/junit.xml

Never do this:

• Hardcode real credentials directly in config.yml
• Commit config.yml with secrets into public repositories

Example of GitHub Action to upload Test Results to TestRail via TRCLI

This workflow automates:

1. Installing dependencies and running tests
2. Uploading JUnit-style results to TestRail with TRCLI

name: Playwright Tests + Upload to TestRail

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test-and-upload:
    runs-on: ubuntu-latest
    env:
      TESTRAIL_USERNAME: ${{ secrets.TESTRAIL_USERNAME }}
      TESTRAIL_API_KEY: ${{ secrets.TESTRAIL_API_KEY }}
      TESTRAIL_HOST: https://yourcompany.testrail.io
      TESTRAIL_PROJECT: "My Project Name"

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 18

      - name: Install dependencies
        run: npm ci

      - name: Run Playwright tests and generate JUnit report
        run: |
          npx playwright install
          npx playwright test --reporter=junit
        continue-on-error: true  # Always upload results, even if tests fail

      - name: Upload results to TestRail
        uses: gurock/trcli-action@v1
        with:
          args: >
            -y
            --host ${{ env.TESTRAIL_HOST }}
            --project "${{ env.TESTRAIL_PROJECT }}"
            --username "${{ env.TESTRAIL_USERNAME }}"
            --password "${{ env.TESTRAIL_API_KEY }}"
            parse_junit
            -f test-results/junit.xml
            --title "GitHub Run #${{ github.run_number }}"
            --run-description "GitHub Actions run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            --close-run

Environment Variable Setup

To avoid exposing secrets in your repo:

1. Go to GitHub → Your Repository → Settings → Secrets and variables → Actions
2. Add:
   • TESTRAIL_USERNAME: your TestRail login email
   • TESTRAIL_API_KEY: your TestRail API token (found in My Settings → API Keys)

Notes

• --close-run finalizes the run immediately after upload.
• --run-description includes a clickable GitHub Actions link inside TestRail.
• Adjust test-results/junit.xml to match your framework output.
• continue-on-error: true ensures results are uploaded even when tests fail.

Example of GitHub Action to parallel jobs and a single run

This setup allows multiple test jobs to push results into a shared TestRail run, enabling full-suite execution across platforms.

name: Parallel Playwright Tests + TestRail Upload

on:
  workflow_dispatch:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  create-run:
    runs-on: ubuntu-latest
    outputs:
      run-id: ${{ steps.create-run.outputs.run-id }}
    env:
      TESTRAIL_USERNAME: ${{ secrets.TESTRAIL_USERNAME }}
      TESTRAIL_API_KEY: ${{ secrets.TESTRAIL_API_KEY }}
      TESTRAIL_HOST: https://yourcompany.testrail.io
      TESTRAIL_PROJECT: "My Project Name"
    steps:
      - name: Create TestRail run
        id: create-run
        run: |
          pip install trcli
          RUN_ID=$(trcli -y \
            --host "$TESTRAIL_HOST" \
            --project "$TESTRAIL_PROJECT" \
            --username "$TESTRAIL_USERNAME" \
            --password "$TESTRAIL_API_KEY" \
            create_run \
            --title "CI Run #${{ github.run_number }}" \
            --run-description "Triggered by GitHub Actions" \
            --suite-id 1 \
            --json | jq -r '.id')
          echo "run-id=$RUN_ID" >> "$GITHUB_OUTPUT"

  test-linux:
    needs: create-run
    runs-on: ubuntu-latest
    env:
      TESTRAIL_RUN_ID: ${{ needs.create-run.outputs.run-id }}
      TESTRAIL_USERNAME: ${{ secrets.TESTRAIL_USERNAME }}
      TESTRAIL_API_KEY: ${{ secrets.TESTRAIL_API_KEY }}
      TESTRAIL_HOST: https://yourcompany.testrail.io
      TESTRAIL_PROJECT: "My Project Name"
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 18

      - name: Install dependencies
        run: npm ci

      - name: Run Playwright tests (Linux)
        run: |
          npx playwright install
          npx playwright test --reporter=junit
        continue-on-error: true

      - name: Upload results (Linux)
        uses: gurock/trcli-action@v1
        with:
          args: >
            -y
            --host ${{ env.TESTRAIL_HOST }}
            --project "${{ env.TESTRAIL_PROJECT }}"
            --username "${{ env.TESTRAIL_USERNAME }}"
            --password "${{ env.TESTRAIL_API_KEY }}"
            parse_junit
            -f test-results/junit.xml
            --run-id ${{ env.TESTRAIL_RUN_ID }}

  test-windows:
    needs: create-run
    runs-on: windows-latest
    env:
      TESTRAIL_RUN_ID: ${{ needs.create-run.outputs.run-id }}
      TESTRAIL_USERNAME: ${{ secrets.TESTRAIL_USERNAME }}
      TESTRAIL_API_KEY: ${{ secrets.TESTRAIL_API_KEY }}
      TESTRAIL_HOST: https://yourcompany.testrail.io
      TESTRAIL_PROJECT: "My Project Name"
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 18

      - name: Install dependencies
        run: npm ci

      - name: Run Playwright tests (Windows)
        run: |
          npx playwright install
          npx playwright test --reporter=junit
        continue-on-error: true

      - name: Upload results (Windows)
        uses: gurock/trcli-action@v1
        with:
          args: >
            -y
            --host ${{ env.TESTRAIL_HOST }}
            --project "${{ env.TESTRAIL_PROJECT }}"
            --username "${{ env.TESTRAIL_USERNAME }}"
            --password "${{ env.TESTRAIL_API_KEY }}"
            parse_junit
            -f test-results/junit.xml
            --run-id ${{ env.TESTRAIL_RUN_ID }}

  close-run:
    needs: [test-linux, test-windows]
    runs-on: ubuntu-latest
    env:
      TESTRAIL_RUN_ID: ${{ needs.create-run.outputs.run-id }}
      TESTRAIL_USERNAME: ${{ secrets.TESTRAIL_USERNAME }}
      TESTRAIL_API_KEY: ${{ secrets.TESTRAIL_API_KEY }}
      TESTRAIL_HOST: https://yourcompany.testrail.io
      TESTRAIL_PROJECT: "My Project Name"
    steps:
      - name: Close the TestRail run
        run: |
          pip install trcli
          trcli -y \
            --host "$TESTRAIL_HOST" \
            --project "$TESTRAIL_PROJECT" \
            --username "$TESTRAIL_USERNAME" \
            --password "$TESTRAIL_API_KEY" \
            close_run \
            --run-id $TESTRAIL_RUN_ID

Why this structure?

• create-run sets up the TestRail run ID used by all test jobs
• Each test job uploads independently using --run-id
• close-run finalizes the run only after all results are in
• Easily extendable for more platforms or matrix builds

Where to get help

• Official TRCLI documentation
• GitHub: testrail-cli
• TestRail Full System Documentation