Specification-first workflow

The specification-first workflow is a test automation approach where you define your test cases in TestRail before writing the corresponding automated test scripts. It separates the planning of what should be tested from the implementation of how it's tested.

This approach is especially useful when:

• Your team maintains detailed documentation for QA processes
• You already have a repository of test cases in TestRail
• You need to collaborate across technical and non-technical roles
• You want traceability between requirements, manual tests, and automated scripts

For example, a QA lead may create and review test cases in TestRail with input from business analysts or testers. Then, automation engineers link those approved cases to automated scripts using the case IDs.

By doing this, the entire team shares a common understanding of what is being tested and why. TestRail becomes the single source of truth for both manual and automated testing efforts.

If you are writing test cases directly in your codebase and don't have those test cases documented in TestRail, you might want to read more about the code-first automation approach, which is better suited for rapid iteration and smaller teams.

In many automation frameworks, a single automated test may validate multiple requirements within the same workflow.

Because of this, TestRail supports mapping one automated test to multiple TestRail case IDs. This allows automation to remain clean and maintainable while still preserving traceability between automated tests and TestRail cases.

Using this method, you:

1. Design and review test cases in TestRail
2. Link those cases to your automation code via IDs
3. Use the TestRail CLI to map, match, and upload results

Benefits and Trade-offs

The specification-first approach offers several compelling advantages, but it also comes with a few limitations depending on your workflow, team size, and test maturity.

Mapping one Automated Test to multiple TestRail Cases

In a specification-first workflow, test cases are created in TestRail before automation is written. Each automated test is then linked to the corresponding TestRail cases.

While some teams create one automated test per TestRail case, this is not always practical. Many modern automation frameworks use:

• shared validation logic
• reusable steps
• parameterised tests

In these situations, a single automated test may verify several behaviors that correspond to multiple TestRail test cases.

For example:

When results are imported into TestRail, each case receives an individual result, ensuring that traceability and reporting remain accurate.

This allows teams to:

• keep automation code simple and reusable
• maintain clear requirement traceability
• avoid creating duplicate automation tests solely to match case structure

Step-by-step workflow

Prerequisites

Before you begin, make sure:

• TestRail CLI is installed
• You have a TestRail project with test cases already documented
• You know the TestRail test case IDs (e.g., C123, C2645)
• You are using a supported test runner (e.g., JUnit)

Step 1: Map your Test Cases in your automation code

Automation tests are linked to TestRail test cases and results are reported back to TestRail. Depending on the automation structure, a single automated test may update one or multiple TestRail cases, while each case still receives its own result. There are two ways to do this:

Option 1: Match by Name

Include the case ID in your test name. Examples:

• C123 login_valid_credentials
• test_login [C123]
• C123_test_login

JUnit example:

<!-- XML (JUnit) format -->
<testcase classname="tests.LoginTests" name="C123_test_login" time="650"/>

Use this with --case-matcher "name" in the CLI.

Option 2: Match by Property

Embed the test ID as a JUnit property:

<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests">
    <properties>
      <property name="setting1" value="True"/>
    </properties>
    <testcase classname="tests.LoginTests" name="C2647_test_case_1" time="159">
      <skipped type="pytest.skip" message="Please skip">
        skipped by user
      </skipped>
    </testcase>
    <testcase classname="tests.LoginTests" name="C2645_test_case_2" time="650">
    </testcase>
    <testcase classname="tests.LoginTests" name="C2648_test_case_3" time="159">
      <failure type="pytest.failure" message="Fail due to...">
        failed due to…
      </failure>
    </testcase>
  </testsuite>
</testsuites>

Use this with --case-matcher "property" in the CLI.

• The test case ID you should use is the one displayed in the Test Cases page with the prefix C.

Step 2: Upload Test Results

Use the CLI to upload results. Here’s a sample command:

trcli -n \
  -h https://<INSTANCE>.testrail.io \
  --project "<PROJECT_NAME>" \
  --username <EMAIL> \
  --password <API_KEY> \
  parse_junit \
  --case-matcher "name" \
  --title "Automated Test Run" \
  -f results.xml

Flags explained:

• -n: Do not auto-create new cases
• --case-matcher: Choose "name" or "property" as mapping method
• --title: Name for the test run
• -f: Path to your JUnit result file

Expected output:

Parsing JUnit report.
Processed 3 test cases in 1 sections.
Checking project. Done.
Creating test run. Run created: https://INSTANCE-NAME.testrail.io/index.php?/runs/view/123
Adding results: 3/3, Done.
Submitted 3 test results in 5.5 secs.

This test run is now visible to your whole QA team.

Advanced Use Case: Update an Existing Test Run

In many QA workflows, test runs are created in advance by QA managers or leads to track progress against a milestone, sprint, or release. These runs often include a mix of manual and automated tests. When automated tests are executed later, their results can be uploaded into the existing test run using the TestRail CLI - rather than creating a new run each time.

This is helpful when:

• You want a single test run that includes both manual and automated execution results
• You want to re-run automated tests and update only their outcomes
• You're working on a long-lived test run shared across multiple executions or team members

Step-by-Step Example

Assume a test run already exists in TestRail, created under your project. The run includes these test cases:

• C101  - Manual (tested manually)
• C102 - Manual (tested manually)
• C103 - Automated (to be updated by CLI)
• C104 - Automated (to be updated by CLI)

The automation engineer runs the test suite and generates a JUnit XML report with results for cases C103 and C104. The report file is named ./results.xml.

CLI Command to Update That Run

trcli -n \
  -h https://<INSTANCE>.testrail.io \
  --project "<PROJECT_NAME>" \
  --username <EMAIL> \
  --password <API_KEY> \
  parse_junit \
  --case-matcher "property" \
  --title "Regression - Sprint 18" \
  --run-id 52 \
  -f ./results.xml

Key Parameters

• --run-id 52: This tells TestRail CLI to update the test run with ID 52 instead of creating a new run.
• --case-matcher "property": Matches test cases by the JUnit test_id property.
• -n: Ensures new test cases are not auto-created for unmatched entries.

Getting the Run ID

To find the run ID:

1. Open the test run in TestRail.
2. Look at the browser's URL: https://yourcompany.testrail.io/index.php?/runs/view/52 - the number 52 is the run ID.

What Happens Next?

• TestRail CLI reads the results XML
• It matches C103 and C104 with test cases in Run ID 52
• It uploads the test results (pass/fail/skip) for those two test cases
• Manual results (C101, C102) remain untouched

This allows the team to maintain a single, centralized view of the test execution status - great for reporting, test signoff, and audits.

You can repeat this step whenever you re-run automated tests. Just point to the same run ID to update results without duplication.

Where to get help

• Official TRCLI documentation
• GitHub: testrail-cli
• TestRail Full System Documentation