Usage examples

Updated September 07, 2023 16:58

The TestRail CLI has multiple features to support the most varied needs when it comes to managing your automated test results upload and enriching these results with more data.

Enriching test results

When submitting test results, TestRail enables adding various types of data in various fields. For instance, you may want to upload a screenshot as an attachment, add a text comment to your result or use custom result fields to store data in a more structured way. Using the TestRail CLI, you can do all of this, in multiple ways.

To add the same value for a field of all results, you can pass the value directly in the command line by using the --result-fields argument one or more times.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --result-fields "version:1.2" \
>    --result-fields "custom_environment:qa02" \
>    -f "results.xml"

If you need to add different values for each result, you can also achieve that by adding properties to the JUnit report under each test case. Using this technique, you can not only enrich your results with fields data, but also add comments and attachments.

The properties for these kinds of data should be added as follows:

Attachments
- Name: Equal to or prefixed with testrail_attachment
- Value: Path to the file to be uploaded
Result fields
- Name: Equal to or prefixed with testrail_result_field
- Value: Using the pattern field_name:field_value
Comments
- Name: Equal to or prefixed with testrail_result_comment
- Value: Text to append to the comment field
Result steps
- Name: Equal to or prefixed with testrail_result_step
- Value: Using the pattern status: step description
- Available statuses:
  - passed
  - untested
  - skipped - maps to retest
  - failed

<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests">
    <properties><property name="setting1" value="True"/></properties>
    <testcase classname="tests.LoginTests" name="test_case_2" time="650">
     <properties>
         <property name="testrail_attachment" value="path_to/logs.log"/>
         <property name="testrail_attachment" value="path_to/screenshot.jpg"/>
         <property name="testrail_result_field" value="version:1.2"/>
         <property name="testrail_result_field" value="custom_environment:qa02"/>
         <property name="testrail_result_comment" value="Finding 1"/>
         <property name="testrail_result_comment" value="Finding 2"/>
         <property name="testrail_result_step" value="passed: Insert login credentials"/>
         <property name="testrail_result_step" value="failed: Click submit"/>
         <property name="testrail_result_step" value="untested: User should be logged in"/>
     </properties>
    </testcase>
  </testsuite>
</testsuites>

Enriching test case creation

If you are using a code-first approach with automatic test case creation, you can also enrich your test cases with data on any field. When submitting test results, TestRail enables adding various types of data in various fields. For instance, you may want to upload a screenshot as an attachment, add a text comment to your result or use custom result fields to store data in a more structured way. Using the TestRail CLI, you can do all of this, in multiple ways.

To add the same value for a field of all test cases, you can pass the value directly in the command line by using the --case-fields argument one or more times.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --result-fields "version:1.2" \
>    --result-fields "custom_environment:qa02" \
>    -f "results.xml"

If you need to add different values for each test case, you can also achieve that by adding properties to the JUnit report under each test case, using the following convention:

Case fields
- Name: Equal to or prefixed with testrail_case_field
- Value: Using the pattern field_name:field_value

<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests">
    <properties><property name="setting1" value="True"/></properties>
    <testcase classname="tests.LoginTests" name="test_case_2" time="650">
     <properties>
         <property name="testrail_case_field" value="refs:STORY-1"/>
         <property name="testrail_case_field" value="custom_preconds:My preconditions"/>
     </properties>
    </testcase>
  </testsuite>
</testsuites>

Keeping new test cases in one section

If you are using a code-first approach with automatic test case creation, you may want these to be segregated from other test cases in the test cases repository. To keep all your new test cases together under one section, you can make use of the --section-id argument. By doing so, all your automated suites and test cases will be created under the specified section.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --section-id "213" \
>    -f "results.xml"

Adding a test run under a milestone

Milestones are a great way to organize your testing efforts. You can use them to aggregate test runs or plans with a common purpose. If you want your automated test runs created under a milestone, you can pass the --milestone-id argument and the TestRail CLI will create the test run under the specified milestone.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --milestone-id "57" \
>    -f "results.xml"

You should now see the new test run under the specified milestone. This is one way to keep track of your automated test results in one place.

Adding a test run under a test plan

Test plans also aggregate test plans with a common purpose, with the plus of allowing you to specify certain execution configurations. In example, you can have the same set of tests being executed using different browsers and have that information showing on your test plan. If you want your automated test runs created under a test plan, you can pass the --plan-id argument and the TestRail CLI will create the test run under the specified test plan. Although optional, on top of that, if you want to associate the test run with a certain configuration, you can also pass the --config-ids argument with a comma separated list of configuration IDs.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --plan-id "57" \
>    --config-ids "25,34" \
>    -f "results.xml"

You should now see the new test run under the specified test plan. This is another way to keep track of your automated test results in one place.

Updating test results on existing test run

Imagine you already ran the TestRail CLI and you have your automated test results on TestRail, but some of the tests failed and you want to rerun those and update the existing test run with the new results. In order to do so, you just need to pass the --run-id argument and the TestRail CLI will update the test run with that id.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --run-id "32" \
>    -f "results.xml"

You should now see the new test result on the test details panel. This is one way to keep track of your automated test results under the same test run.

Test result updated

Closing the test run

If you want to immediately close your newly created test run, you can simply pass the --close-run argument and the TestRail CLI will perform that action after all the results have been added. This is useful if you don’t want to allow changes to be made to the results after the run has finished.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --close-run \
>    -f "results.xml"

You can find your test run under the Completed test runs section.

Test run closed

Results elapsed time in milliseconds

For TestRail versions 5.5 and above, it is possible to submit the test results elapsed time in milliseconds (i.e.: 0.001s). The default TestRail CLI behavior will round the time to seconds in order to work for every TestRail version. If you are running one of the versions that supports milliseconds and want to see the elapsed time in your result using milliseconds, you can pass the --allow-ms argument.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --allow-ms \
>    -f "results.xml"

In your test results, you should now see the elapsed field using milliseconds.

Using config files to store alternate configurations

Quickly and easily submit results to different instances or projects, use different credentials, or other preset parameters in your commands by using an alternate config file. The configuration file will be written in YAML format, named config.yml, and stored in the same directory as the TRCLI executable file, unless otherwise specified. Environment variables can also be used. If a configuration file is referenced in the command, all parameters within the configuration file will override the environment variables. Any parameters specified in the command will override the config file.

The following example displays the use of an alternate configuration file that stores users credentials:

host: https://INSERT-INSTANCE-NAME.testrail.io 
project: TRCLI Test
username: username@domain.com 
password: passwordORapikey
title: Automated Tests Run

$ trcli -y \
>    --config alternate_config.yaml \
>    parse_junit \
>    -f "results.xml"