GitHub Actions makes it easy to automate all your software development workflows. It allows you to build, test, and deploy your code right from GitHub. Workflows can be kicked off by GitHub events like push, issue creation, new release, or even a manual workflow dispatch.
A common use case for GitHub Actions is automatically building and testing your application whenever code changes are submitted. To improve visibility into your automated tests and manage all your test cases and test results—both manual and automated—more easily, you can send your automated test results directly from GitHub to TestRail. In this article, we will explore how you can take advantage of GitHub Actions and the TestRail CLI to easily accomplish this task.
Please note that the TestRail CLI requires that a compatible JUnit report is generated by your test automation framework.
In this example, we will be using GitHub Actions along with Cypress.
If you want to see more examples, please take a look at the Automation Frameworks Integration run-all.yml workflow file and the respective workflow execution.
Using the TestRail CLI in GitHub workflow files
Automated processes can be configured on GitHub by using workflow configuration files. These are YAML files that allow you to specify your processes in a structured manner and make use of GitHub Actions for each step of the way to make it easier to configure your execution environment, as well as execute any action you require.
The easiest way to integrate TestRail with GitHub Actions is by configuring your workflow files to use the TestRail CLI to send automated test results to TestRail. Below is a sample of how a workflow file using the TestRail CLI to send test results to TestRail would look like.
name: send-results-to-testrail
on: [workflow_dispatch]
jobs:
send-test-results-to-testrail:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Execute automated tests
uses: ...
- name: Python setup
if: always()
uses: actions/setup-python@v3
with:
python-version: '3.x'
- name: TestRail CLI upload results
if: always()
run: |
pip install trcli
trcli -y \
-h https://TESTRAIL_INSTANCE.testrail.io/ \
--project "PROJECT NAME" \
-u USER_EMAIL \
-p PASSWORD \
parse_junit \
--title "Automated Tests from GitHub workflow" \
--run-description ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} \
-f "junit-report.xml"Breaking down the file, we can see a couple of interesting details:
- The trigger event
on: [workflow_dispatch]means this workflow will only run if manually executed. There are other events that can trigger a workflow, such as pushing code, creating a pull request, etc. - On
jobswe have one item calledexecute-tests, which means we only have one job on this workflow. - The
execute-testsjob defines its agent withruns-on: ubuntu-latestand contains the steps belowsteps, which we will detail on the following sections.
-
Checkout Execute automated testsPython setupTestRail CLI upload results
-
You can learn more about workflows features on the official GitHub workflows documentation.
Step 1: Checkout
The Checkout step uses the actions/checkout@v3 GitHub Action which executes the code checkout on the agent for your repository.
- name: Checkout
uses: actions/checkout@v3Step 2: Execute automated tests
The Execute automated tests step is a placeholder. This is the point in time where the automated tests you have in your codebase will be executed and generate the JUnit report required to send the results to TestRail. This can be accomplished either by using a GitHub Action or by inserting your custom execution script.
name: Execute Automated tests
uses: ...Step 3: Install Python on the agent
The Python setup uses the actions/setup-python@v3 GitHub Action to configure Python on the agent. Because the TestRail CLI is a tool developed in Python, it is required that it is installed on the environment before you are able to install and run it.
Also, notice the if: always() condition, which forces this step to run, no matter if the automated test run was successful or not, because, by default, most test automation frameworks will exit with an error code if any test failed.
- name: Python setup
if: always()
uses: actions/setup-python@v3
with:
python-version: '3.x'Step 4: Install the TestRail CLI and upload test results
The TestRail CLI upload results step runs a custom shell script to install and call the TestRail CLI in order to parse the JUnit report generated by your test automation framework and send the results to TestRail.
The TestRail CLI is a Python packages hosted on the Python Package Index (PyPI) so, in order to install it, we simply execute the command pip install trcli.
After the TestRail CLI is installed, there are a few mandatory arguments such as your TestRail instance address and credentials, the project you want to report to, the title for your test run and the path to the JUnit report. The run description argument is not mandatory, but we recommend using it to add a link from TestRail to your GitHub workflow execution. This link can be obtained by composing the URL using GitHub embedded variables, such as on the snippet below.
For more information about arguments, please check the TestRail CLI documentation.
- name: TestRail CLI upload results
if: always()
run: |
pip install trcli
trcli -y \
-h https://TESTRAIL_INSTANCE.testrail.io/ \
--project "PROJECT NAME" \
-u USER_EMAIL \
-p PASSWORD \
parse_junit \
--title "Automated Tests from GitHub workflow" \
--run-description ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} \
-f "junit-report.xml"Working example using Cypress
Below is an example of a GitHub workflow file which, in simple terms, runs Cypress tests and sends the test results straight to TestRail.
name: cypress-tests
on: [workflow_dispatch]
jobs:
execute-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Cypress run
id: cypress
uses: cypress-io/github-action@v4
with:
command: npx cypress run --reporter junit --reporter-options mochaFile=reports/TEST-[hash].xml
- name: Python setup
if: always()
uses: actions/setup-python@v3
with:
python-version: '3.x'
- name: TestRail CLI upload results
if: always()
run: |
pip install trcli
trcli -y \
-h https://TESTRAIL_INSTANCE.testrail.io/ \
--project "PROJECT NAME" \
-u USER_EMAIL \
-p PASSWORD \
parse_junit \
--title "Automated Tests from GitHub workflow" \
--run-description ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} \
-f "reports/junit*.xml"Breaking down the file, there are only two different worthy of note between the generic GitHub workflow we went through before and this one:
- The second step was replaced by a
Cypress runthat uses thecypress-io/github-actions@v4GitHub Action, which install the Cypress project dependencies and execute the commands on thewithparameter, triggering an automated test run. - On the
TestRail CLI upload results, with TestRail CLI version 1.6.0, the report path now supports wildcards, allowing you to merge multiple reports seamlessly. This feature enables you to consolidate results from various sources into a single run submission. You can learn more about this on the documentation about Integrating with Cypress.
Creating your own repository
We're going to setup the GitHub workflow above using our Cypress sample project, which you can know more about by checking our documentation about Integrating with Cypress.
By simply following the steps below you'll have your GitHub workflow with test execution and results reporting to TestRail in no time.
- Create a new repository on GitHub
- Download the files on the Cypress sample project
- Add the file cypress-tests.yml under the path .github/workflows and copy the contents from the example above
- Replace TESTRAIL_INSTANCE, PROJECT NAME, USER_EMAIL and PASSWORD on the file content using your details (we recommend that you do not replace the password directly and use GitHub secrets instead)
- Upload the project to your GitHub repository
Executing the workflow
- Go to the Actions page on your repository and you should be able to see your workflow on the list
- Since this is a manual dispatch workflow, to run it, first click the workflow name on the left and then the Run workflow button
- You can then go into your workflow details to see what went wrong. The execution logs are divided by steps. On the image below we can see that the Cypress run failed
- By drilling into the Cypress run logs, we can see that the execution went well, but one test failed. Although the test failed, the Python setup and TestRail CLI upload results steps were still executed because we used the
if: always()condition on our test step to make sure we send the results to TestRail even when tests fail - Go to your TestRail instance and see your Test Run results. Notice that you have a link back to your GitHub workflow so you can easily further debug your test results!
What next?
Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!
You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.