1. TestRail Support Center
  2. Test Automation
  3. TestRail CLI

Usage examples

Updated

The TestRail CLI has multiple features to support the most varied needs when it comes to managing your automated test results upload and enriching these results with more data.

Enriching test results

When submitting test results, TestRail enables adding various types of data in various fields. For instance, you may want to upload a screenshot as an attachment, add a text comment to your result or use custom result fields to store data in a more structured way. Using the TestRail CLI, you can do all of this, in multiple ways.

To add the same value for a field of all results, you can pass the value directly in the command line by using the --result-fields argument one or more times.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --result-fields "version:1.2" \
>    --result-fields "custom_environment:qa02" \
>    -f "results.xml"

Checking project. Done.
Creating test run. Done.
Adding results: 3/3, Done.
Submitted 3 test results in 3.2 secs.

If you need to add different values for each result, you can also achieve that by adding properties to the JUnit report under each test case. Using this technique, you can not only enrich your results with fields data, but also add comments and attachments.

The properties for these kinds of data should be added as follows:

  • Attachments
    • Name: Equal to or prefixed with testrail_attachment
    • Value: Path to the file to be uploaded
  • Result fields
    • Name: Equal to or prefixed with testrail_result_field
    • Value: Using the pattern field_name:field_value 
  • Comments
    • Name: Equal to or prefixed with testrail_result_comment
    • Value: Text to append to the comment field
<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests">
    <properties><property name="setting1" value="True"/></properties>
    <testcase classname="tests.LoginTests" name="test_case_2" time="650">
     <properties>
         <property name="testrail_attachment" value="path_to/logs.log"/>
         <property name="testrail_attachment" value="path_to/screenshot.jpg"/>
         <property name="testrail_result_field" value="version:1.2"/>
         <property name="testrail_result_field" value="custom_environment:qa02"/>
         <property name="testrail_result_comment" value="Finding 1"/>
         <property name="testrail_result_comment" value="Finding 2"/>
     </properties>
    </testcase>
  </testsuite>
</testsuites>

Enriching test case creation

If you are using a code-first approach with automatic test case creation, you can also enrich your test cases with data on any field. When submitting test results, TestRail enables adding various types of data in various fields. For instance, you may want to upload a screenshot as an attachment, add a text comment to your result or use custom result fields to store data in a more structured way. Using the TestRail CLI, you can do all of this, in multiple ways.

To add the same value for a field of all test cases, you can pass the value directly in the command line by using the --case-fields argument one or more times.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --result-fields "version:1.2" \
>    --result-fields "custom_environment:qa02" \
>    -f "results.xml"

Checking project. Done.
Creating test run. Done.
Adding results: 3/3, Done.
Submitted 3 test results in 3.2 secs.

If you need to add different values for each test case, you can also achieve that by adding properties to the JUnit report under each test case, using the following convention:

  • Case fields
    • Name: Equal to or prefixed with testrail_case_field
    • Value: Using the pattern field_name:field_value
<testsuites name="test suites root">
  <testsuite failures="0" errors="0" skipped="1" tests="1" time="0.05" name="tests.LoginTests">
    <properties><property name="setting1" value="True"/></properties>
    <testcase classname="tests.LoginTests" name="test_case_2" time="650">
     <properties>
         <property name="testrail_case_field" value="refs:STORY-1"/>
         <property name="testrail_case_field" value="custom_preconds:My preconditions"/>
     </properties>
    </testcase>
  </testsuite>
</testsuites>

Adding a test run under a milestone

Milestones are a great way to organize your testing efforts. You can use them to aggregate test with a common purpose. If you want your automated test runs created under a milestone, you can pass the --milestone-id argument and the TestRail CLI will create the test run under the specified milestone.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --milestone-id "57" \
>    -f "results.xml"

Checking project. Done.
Creating test run. Done.
Adding results: 3/3, Done.
Submitted 3 test results in 2.5 secs.

You should now see the new test result on the test details panel. This is one way to keep track of your automated test results under the same test run.

Updating test results on existing test run

Imagine you already ran the TestRail CLI and you have your automated test results on TestRail, but some of the tests failed and you want to rerun those and update the existing test run with the new results. In order to do so, you just need to pass the --run-id argument and the TestRail CLI will update the test run with that id.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --run-id "32" \
>    -f "results.xml"

Checking project. Done.
Adding results: 3/3, Done.
Submitted 3 test results in 2.5 secs.

You should now see the new test result on the test details panel. This is one way to keep track of your automated test results under the same test run.

Test result updated

Closing the test run

If you want to immediately close your newly created test run, you can simply pass the --close-runargument and the TestRail CLI will perform that action after all the results have been added. This is useful if you don’t want to allow changes to be made to the results after the run has finished.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --close-run \
>    -f "results.xml"

Checking project. Done.
Creating test run. Done.
Adding results: 3/3, Done.
Closing test run. Done.
Submitted 3 test results in 3.2 secs.

You can find your test run under the Completed test runs section.

Test run closed

Results elapsed time in milliseconds

For TestRail versions 5.5 and above, it is possible to submit the test results elapsed time in milliseconds (i.e.: 0.001s). The default TestRail CLI behavior will round the time to seconds in order to work for every TestRail version. If you are running one of the versions that supports milliseconds and want to see the elapsed time in your result using milliseconds, you can pass the --allow-ms argument.

$ trcli -y \
>    -h "https://INSERT-INSTANCE-NAME.testrail.io" \
>    --project "TRCLI Test" \
>    --username "user@domain.com" \
>    --password "passwordORapikey" \
>    parse_junit \
>    --title "Automated Tests Run" \
>    --allow-ms \
>    -f "results.xml"

Checking project. Done.
Creating test run. Done.
Adding results: 3/3, Done.
Submitted 3 test results in 3.2 secs.

In your test results, you should now see the elapsed field using milliseconds.

mceclip1.png

Using config files to store alternate configurations

Quickly and easily submit results to different instances or projects, use different credentials, or other preset parameters in your commands by using an alternate config file. The configuration file will be written in YAML format, named config.yml, and stored in the same directory as the TRCLI executable file, unless otherwise specified. Environment variables can also be used. If a configuration file is referenced in the command, all parameters within the configuration file will override the environment variables. Any parameters specified in the command will override the config file.

The following example displays the use of an alternate configuration file that stores users credentials:

host: https://INSERT-INSTANCE-NAME.testrail.io 
project: TRCLI Test
username: username@domain.com 
password: passwordORapikey
title: Automated Tests Run
$ trcli -y \
>    --config alternate_config.yaml \
>    parse_junit \
>    -f "results.xml"

Checking project. Done.
Creating test run. Done.
Adding results: 3/3, Done.
Submitted 3 test results in 3.2 secs.

 

Was this article helpful?
1 out of 1 found this helpful