Overview and installation

The TestRail CLI is a command line interface tool that allows you to effortlessly upload test automation results from any JUnit-style XML file to TestRail.

Overview

You can use the TestRail CLI to parse test automation results from nearly any test automation tool or framework (as long as that tool can export test results in a JUnit-style XML file). Some of the most used test automation frameworks that can easily generate reports which can be used by the TestRail CLI include:

• Cypress.io
• Playwright
• JUnit
• TestNG
• NUnit
• Pytest
• Robot Framework

By parsing and uploading JUnit style test results into TestRail directly from the command line or by running the CLI as part of an automated build pipeline in Jenkins, GitLab CI, GitHub Actions, or other CI tools, you can focus on writing test code instead of having to worry about API calls and other technical details related to uploading test results.

The TestRail CLI is an open source project hosted on GitHub, so anyone can contribute by creating issues or even developing code to implement new features or improvements.

For sample test automation projects with different frameworks, you can also see our Automation Frameworks Integration project on GitHub.

Installing the TestRail CLI

The TestRail CLI is programming-language agnostic when it comes to uploading your test results, but the tool itself is developed in Python and can be installed from the public Python Package Index (PyPI). To install it on your machine or build agent, we recommend downloading the Python 3.10.4 installer matching your operating system and following the install wizard, making sure pip is also installed. To make sure the install was successful, try executing the commands python --version and pip --version from your command line and they  should output their versions.

After you’re done installing Python, installing the TestRail CLI is as easy as running one line on your system’s command line or adding to script in your build pipeline.

$ pip install trcli

If the install command ran successfully, you should be able to directly call the TestRail CLI using the command trcli and see the following output.

$ trcli
TestRail Connect v1.4.0 
Copyright 2021 Gurock Software GmbH - www.gurock.com
Supported and loaded modules: 
- junit: JUnit XML Files (& Similar) 

Configuring your TestRail instance

Before using the TestRail CLI, you first need to enable the TestRail API for your instance. You can do this by accessing Administration > Site Settings, clicking on the API tab, and checking the Enable API option. You can refer to the API Introduction page for more information.

Test automation workflows

When you are doing test automation, you can approach test management in different ways. Some teams prefer to work on test design straight in their code base ("code-first" approach). Others prefer to document their test cases outside the automation code base using a tool like TestRail first, and then implement the test code and map to the specified tests ("specification-first" approach).

The TestRail CLI has mechanisms to support using either of these approaches. If you choose to write tests in your code first, without adding test cases to TestRail, the TestRail CLI will simply update your test case repository with your new automation tests the first time you upload a new test result for them.

Or, if you are automating tests based on test cases you have already documented in TestRail, you can link your automation code with your test cases in TestRail to report automation results for those test cases.

For a practical example, you can check our Cypress test automation project on GitHub, which contains test suites prepared for both workflows, along with instructions on how to use them.

Code-first		Specification-first

Code-first 

An approach typically used by teams which work on test design straight in their code base. This means they plan the test cases directly in the test code, implement the test scripts and manage the execution outside a test management tool. 

Pros:

•   Automatic test case creation
•   Automatic test case mapping
•   Manage test cases in code base only
•   Suites more technical teams who approach test design directly in the code

Cons:

•   Mapping is more brittle, may break if automation project structure or test names change
•   Teams with low experience may end up duplicating test cases
•   Requires extra time create mappings if test cases already exist in TestRail

You can read more about how to implement this approach on the TestRail CLI - Code-first page.

Specification-first

An approach where test case design is done prior to implementing the automated tests code. In this workflow, teams first outline and design their test cases in TestRail, giving them an opportunity to write test cases in multiple formats, review, categorize, prioritize and select for automation. 

Pros:

•   Helps teams prioritize which test cases to automate
•   Test case mappings are kept when structural code changes occur
•   Reduces chances of duplicating test cases through automatic creation
•   Enables thorough test case design and planning
•   Suits teams who require good test documentation

Cons:

•   Test cases without case ID in the code will not be created and result will not be submitted
•   Mapping requires manual work which can be laborious and error-prone

You can read more about how to implement this approach on the TestRail CLI - Specification-first page.

CLI reference and usage examples

You can get to know all about the TestRail CLI parameters by using the commands trcli --help and trcli parse_junit --help, or by checking the TestRail CLI - Reference page. You can also see the TestRail CLI - Usage examples page for samples on how to use many of the options available in the TestRail CLI.