1. TestRail Support Center
  2. Test Automation
  3. TestRail CLI

Automation workflows - Specification-first

Updated

The specification-first automation workflow

Specification-first is an approach where test case design is done prior to implementing the automated tests code. In this workflow, teams first outline and design their test cases in TestRail, giving them an opportunity to write test cases in multiple formats, review, categorize, prioritize and select for automation. This approach typically suites teams who are automating test cases that they already have documented in TestRail, work in a context that requires being more thorough documenting their test cases, or want to use TestRail to help prioritize which test cases to automate and when.

 

If you are writing test cases directly in your code base and don't have any of those test cases in TestRail, you might want to read more about the code-first automation approach.

Using the TestRail CLI, you can make use of the --case-matcher option by passing the value name or property, depending on how you add the ID of your TestRail test cases in your test automation framework.

mceclip0_3.png

Pros:

  •   Test case mappings are kept when structural code changes occur
  •   Reduces chances of duplicating test cases through automatic creation
  •   Enables thorough test case design and planning
  •   Suites teams who require good test documentation

Cons:

  •   Test cases without case ID in the code will not be created and result will not be submitted
  •   Mapping requires manual work which can be laborious and error-prone

Mapping test cases

There are two ways you can add your TestRail test case ID in your test automation code:

  • If you use the --case-matcher "name" option, then add the case ID from your test case in the name of the test. This will ensure that the case ID appears the JUnit report as part of the test case name. You can add the ID following multiple patterns, such as: [C123] test_case_1, test_case_1 [C123], C123 test_case_1, C123_test_case_1. test_case_1 C123, test_case_1_C123
    <testcase classname="tests.LoginTests" name="C123_test_case_1" time="650">
</testcase>
  • If you use the --case-matcher "property" option, then you have to set the JUnit property named test_id to be attached to your test cases in the JUnit report
    <testcase classname="tests.LoginTests" name="test_case_1" time="650">
        <properties>
            <property name="test_id" value="C123"/>
        </properties>
</testcase>

The test case ID you should use is the one displayed in the Test Cases page with the prefix C.

mceclip0.png

Using the TestRail CLI to upload test automation results

The TestRail CLI is designed to be simple and efficient. Using the specification-first approach, once the CLI has been installed and your TestRail instance is properly configured, a JUnit results file can be passed through the command line to quickly create a run and add test results for the matching test cases in your test automation project.

Below, you can see a sample JUnit XML with the test case IDs in the test name. In this case, we will use the --case-matcher "name" option to match the test cases in TestRail, but if you want to use the --case-matcher "property" option, the remaining steps are the same. You can execute a simple command such as the one just below it to send the test results to TestRail.

 

We recommend using the -n option to skip automatic test creation if you are using a specification-first approach. Nevertheless you can use the -y option if you want tests that do not have IDs in your code to be created, but beware that if you do not map the IDs in your code afterwards, these test cases will be duplicated the next time the CLI runs.

<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests">
    <properties>
      <property name="setting1" value="True"/>
    </properties>
    <testcase classname="tests.LoginTests" name="C2647_test_case_1" time="159">
      <skipped type="pytest.skip" message="Please skip">
        skipped by user
      </skipped>
    </testcase>
    <testcase classname="tests.LoginTests" name="C2645_test_case_2" time="650">
    </testcase>
    <testcase classname="tests.LoginTests" name="C2648_test_case_3" time="159">
      <failure type="pytest.failure" message="Fail due to...">
        failed due to…
      </failure>
    </testcase>
  </testsuite>
</testsuites>
$ trcli -n \
>    -h https://INSTANCE-NAME.testrail.io \
>    --project "TRCLI Test" \
>    --username user@domain.com \
>    --password passwordORapikey \
>    parse_junit \
>    --case-matcher "name" \
>    --title "Automated Tests Run" \
>    -f results.xml

Parsing JUnit report.
Processed 3 test cases in 1 sections.
Checking project. Done.
Creating test run. Run created: https://INSTANCE-NAME.testrail.io/index.php?/runs/view/123
Adding results: 3/3, Done.
Submitted 3 test results in 5.5 secs.

Once the process is complete, you can go to the Test Runs and Results page where you will see a new run titled Automated Tests Run within the TRCLI Test project. Also notice that the TestRail CLI outputs a direct link to the Test Run in TestRail.

Test Run

By opening the test run, you can see the results for each test case. You can then drill further into a failed test case and check the error message that was imported directly from the JUnit report. This can be helpful to have a quick overview of what went wrong during the test.

Test results

Additional use cases for specification-first approach

New test run for the automated test cases from the existing documented test cases

Specification-first is an approach that involves designing test cases before writing the automated test code. In this section, you will explore various practical use cases for this methodology.

With the specification-first approach, you have the possibility to create a new test run that automatically uploads the results for the previously automated test cases. Consider the following example where in a project you have some documented test cases.

Gav1.JPG

Now, consider a scenario where you've automated some of the test cases from these documented test cases ( for ex. C109 and C110) and you intend to upload the results of these automated test cases utilizing the TestRail CLI while following the specification-first approach.

If you are using JUnit framework , you can annotate the automated test cases using the TestRail annotation feature provided by the testrail-junit-extensions. Here is the code snippet for this purpose.


@Test
@TestRail(id="C109")
void verifyTitleOfHomePage() {
   // Assertion: Check its title is correct
   assertTrue(driver.getTitle().contains("TestRail"));
}

@Test
@TestRail(id="C110")
void verifyPresenceOfDemoLinkOnHomePage() {
   // Assertion: Check the presence of demo link
   By demoButtonSelector = By.linkText("Get a Demo");
   WebElement demoButton = driver.findElement(demoButtonSelector);
   assertTrue(demoButton.isDisplayed());
}

As you can observe, C110 and C109 represent IDs of the test cases that have been previously documented in TestRail. To create a new test run and upload the results of the automated tests to the test run using the TestRail CLI, you can run the following command:

$ trcli -n \
>    -h https://INSTANCE-NAME.testrail.io \
>    --project "PROJECT-NAME" \
>    --username user@domain.com \
>    --password passwordTestRail \
>    parse_junit \
>    --case-matcher "property" \
>    --title "TEST RUN TITLE" \
>    -f "./target/TEST-junit-jupiter.xml" \

Upon executing the above command, you will observe that a new test run has been created, and the results of the automated tests have been successfully uploaded.

Gav2.JPG

Upload attachment as a part of test result

In many cases, it becomes necessary to include attachments as part of the test results. A common scenario is when a test fails, and you have captured a screenshot of the screen at the point of failure. You may want to upload this attachment along with the test results using the TestRail CLI.

Attachments can also be uploaded with the help of TestRail CLI. If you are using JUnit framework then you can extend your test classes with the annotation on the code snippet below and add the TestRailTestReporter argument to your test methods. Setting properties is as simple as using the setProperty() method with the desired key and value. In the snippet below you can see an example of setting an attachment to be sent to TestRail along with the test result.


@ExtendWith(TestRailTestReporterParameterResolver.class)
class HomePageTest {
   @Test
   @TestRail(id = "C110")
   void verifyPresenceOfDemoLinkOnHomePage(TestRailTestReporter customReporter) {
       // Assertion: Check the presence of demo link
       By demoButtonSelector = By.linkText("Get a Demo");
       WebElement demoButton = driver.findElement(demoButtonSelector);
       assertTrue(demoButton.isDisplayed());
       customReporter.setProperty("testrail_attachment_1", "sample_reports/testrail.JPG");
   }
}

To create a new test run and upload the results of the automated tests to the test run using the TestRail CLI, you can run the following command:

$ trcli -n \
>    -h https://INSTANCE-NAME.testrail.io \
>    --project "PROJECT-NAME" \
>    --username user@domain.com \
>    --password passwordTestRail \
>    parse_junit \
>    --case-matcher "property" \
>    --title "TEST RUN TITLE" \
>    -f "./target/TEST-junit-jupiter.xml" \

After the results are successfully uploaded, you can view the comments within the corresponding test result.

Gav3.JPG

Add comment as a part of result

In certain situations, you may have specific comments for the automated tests, and you'd like to include these comments as part of the test result. With the assistance of TestRail CLI, this is indeed possible.

If you are using JUnit framework you can extend your test classes with the annotation on the code snippet below and add the TestRailTestReporter argument to your test methods. Setting properties is as simple as using the setProperty() method with the desired key and value. In the snippet below you can see an example of setting an attachment to be sent to TestRail along with the test result.


@ExtendWith(TestRailTestReporterParameterResolver.class)
class HomePageTest {
   @Test
   @TestRail(id = "C109")
   void verifyTitleOfHomePage(TestRailTestReporter customReporter) {
       // Assertion: Check its title is correct
       assertTrue(driver.getTitle().contains("TestRail"));
       customReporter.setProperty("testrail_result_comment", "This is a the test comment for the test case");
   }
 }

To create a new test run and upload the results of the automated tests to the test run using the TestRail CLI, you can run the following command:

$ trcli -n \
>    -h https://INSTANCE-NAME.testrail.io \
>    --project "PROJECT-NAME" \
>    --username user@domain.com \
>    --password passwordTestRail \
>    parse_junit \
>    --case-matcher "property" \
>    --title "TEST RUN TITLE" \
>    -f "./target/TEST-junit-jupiter.xml" \

After the results are successfully uploaded, you can view the comments within the corresponding test result.

Gav6.JPG

Update automated test results within existing test run

Occasionally, test runs are generated for a specific milestone or release by utilizing documented test cases. In such instances, updating the results of automated test cases for an existing test run using TestRail CLI is entirely feasible.

Consider a project scenario where you've established a test run for a release, incorporating a mix of both automated and manual test cases. For instance, in the test run shown in the screenshot below, test cases C109 and C110 are automated, while C100 and C101 are manual test cases. To upload the results of the automated test cases to the pre-generated test run, you can follow the steps outlined below, adhering to the specification-first approach using TestRail CLI.

Gav7.JPG

If you are using JUnit framework, you can annotate the automated test cases using the TestRail annotation feature facilitated by the testrail-junit-extensions. The following code snippet serves this purpose.


@Test
@TestRail(id="C109")
void verifyTitleOfHomePage() {
   // Assertion: Check its title is correct
   assertTrue(driver.getTitle().contains("TestRail"));
}

@Test
@TestRail(id="C110")
void verifyPresenceOfDemoLinkOnHomePage() {
   // Assertion: Check the presence of demo link
   By demoButtonSelector = By.linkText("Get a Demo");
   WebElement demoButton = driver.findElement(demoButtonSelector);
   assertTrue(demoButton.isDisplayed());
}

As evident, C110 and C109 correspond to the IDs of test cases documented earlier in TestRail. To upload the results of the automated tests to the current test run using TestRail CLI, execute the following command:

$ trcli -n \
>    -h https://INSTANCE-NAME.testrail.io \
>    --project "PROJECT-NAME" \
>    --username user@domain.com \
>    --password passwordTestRail \
>    parse_junit \
>    --case-matcher "property" \
>    --title "TEST RUN TITLE" \
>    --run-id "32" \
>    -f "./target/TEST-junit-jupiter.xml" \

You can obtain the run id of the test run from the URL of the test run.

After updating the results for the automated tests, the test run appears as follows:

Gav10.JPG

Was this article helpful?
4 out of 11 found this helpful