Code-first workflow

The code-first workflow

Code-first is an approach typically used by teams which work on test design straight in their code base. This means they plan the test cases directly in the test code, implement the test scripts and manage the execution outside a test management tool. These teams don't want the overhead of having to create and manage test cases outside their code base, but it is still important to bring visibility to the test results and enable generating reports based on these results.

This is the default approach supported by the TestRail CLI. It uses the --case-matcher auto option, which automatically creates test cases and adds their results in TestRail.

Pros:

•   Automatic test case creation
•   Automatic test case mapping
•   Manage test cases in code base only
•   Suites more technical teams who approach test design directly in the code

Cons:

•   Mapping is more brittle, may break if automation project structure or test names change
•   Teams with low experience may end up duplicating test cases
•   Hard and laborious to create mappings if test cases already exist in TestRail

Configuring your TestRail instance

To use the code-first approach, first add a custom case field to your TestRail instance called automation_id. This field will store the mapping between your automated tests code and the test cases in TestRail, and allow you upload test results to TestRail without duplicating test cases or having to write TestRail case IDs in your test automation code.

You can create a new custom field by accessing Administration > Customizations and clicking Add Field. After you’ve reached the field creation screen, we recommend you configure the field as per the image below. If you need more information please refer to the Configuring custom fields page.

Using the TestRail CLI to upload test automation results

The TestRail CLI is designed to be simple and efficient. Using the code-first approach, once it has been installed and your TestRail instance is properly configured, a JUnit results file can be passed through the command line to quickly create a run, add test results, and even automatically create cases for automated test cases that do not yet exist on TestRail.

Using the sample JUnit XML report below you can execute a simple command such as the one just below it to send the test results to TestRail. Notice the -y option on the command itself, which skips the prompts to automatically create new tests. This is useful when running the TestRail CLI through your CI tools.

<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests">
    <properties>
      <property name="setting1" value="True"/>
    </properties>
    <testcase classname="tests.LoginTests" name="test_case_1" time="159">
      <skipped type="pytest.skip" message="Please skip">
        skipped by user
      </skipped>
    </testcase>
    <testcase classname="tests.LoginTests" name="test_case_2" time="650">
    </testcase>
    <testcase classname="tests.LoginTests" name="test_case_3" time="159">
      <failure type="pytest.failure" message="Fail due to...">
        failed due to…
      </failure>
    </testcase>
  </testsuite>
</testsuites>

$ trcli -y \
>    -h https://INSTANCE-NAME.testrail.io \
>    --project "TRCLI Test" \
>    --username user@domain.com \
>    --password passwordORapikey \
>    parse_junit \
>    --title "Automated Tests Run" \
>    -f results.xml

Parsing JUnit report.
Processed 3 test cases in 1 sections.
Checking project. Done.
Found 3 test cases not matching any TestRail case.
Adding missing sections to the suite.
Adding missing test cases to the suite.
Adding test cases: 3/3, Done.
Creating test run. Run created: https://INSTANCE-NAME.testrail.io/index.php?/runs/view/123
Adding results: 3/3, Done.
Submitted 3 test results in 8.9 secs.

Once the import process is complete, if you go to the Test Cases page in your TestRail project, you’ll notice that the TestRail CLI automatically created the test cases that were on your test results report. Notice that it added a unique Automation ID by combining the classname and name attributes on the JUnit report using the pattern classname.name for each test on the report. This Automation ID is used to map the tests on your automation code base to the test cases on TestRail. This means that each time you run the TestRail CLI, it first attempts to match an existing test case on TestRail and only creates a new one in case there is no test case with that Automation ID.

Example:

<testcase
  classname=”tests.LoginTests”  name=”test_case_1″  time=”159″>
</testcase>

tests.LoginTests.test_case_1

If you go to the Test Runs and Results page you will see a new run titled Automated Tests Run within the TRCLI Test project.

By opening the test run, you can see the results for each test case. You can then drill further into a failed test case and check the error message that was imported directly from the JUnit report. This can be helpful to have a quick overview of what went wrong during the test.