With GitLab CI/CD you can automated all the steps required to build, test and deploy your code. Typically teams use GitLab to store their source code, and with the CI/CD capabilities they also build and test their code inside GitLab by creating pipelines to streamline building and testing the code before being deployed.
As part of the pipeline, teams will commonly want to run their automated tests. To improve visibility into your automated tests and manage all your test cases and test results—both manual and automated—more easily, you can send your automated test results directly from GitLab to TestRail. In this article, we will explore how you can take advantage of GitLab CI/CD and the TestRail CLI to easily accomplish this task.
Please note that the TestRail CLI requires that a compatible JUnit report is generated by your test automation framework.
In this example, we will be using GitLab CI/CD along with Playwright.
Using the TestRail CLI in GitLab pipelines
GitLab pipelines can be created by adding a
.gitlab-ci.yml file to the root of a GitLab repository. This file allows you to specify your processes in a structured manner using YML syntax. You can define the multiple stages of your pipeline using jobs that may run either sequentially or in parallel.
The easiest way to integrate TestRail with GitLab CI/CD is by configuring the TestRail CLI to send automated test results to TestRail in one of the stages in your
.gitlab-ci.yml file. Below is a sample of how a GitLab pipeline using Playwright as test automation framework and the TestRail CLI to send test results to TestRail would look like.
stages: - playwright_tests - trcli_results_upload playwright_tests: stage: playwright_tests image: node:latest artifacts: when: always paths: - test-results/ allow_failure: true script: - npm install - npx playwright install-deps - npx playwright install - npx playwright test trcli_results_upload: stage: trcli_results_upload image: python:latest dependencies: - playwright_tests script: | pip install trcli trcli -y \ -h https://TESTRAIL_INSTANCE.testrail.io/ \ --project "PROJECT NAME" \ -u USER_EMAIL \ -p PASSWORD \ parse_junit \ -f "test-results/junit-report.xml" \ --run-description "URL: $CI_PIPELINE_URL" \ --title "Automated Tests from GitLab CI/CD"
Breaking down the file, there are a couple of details you should pay attention to:
- There are two jobs defined in this pipeline file:
playwright_testswhere we execute our sample automated tests project
trcli_results_uploadwhere we upload the generated results to TestRail
- By default, jobs are executed in parallel. In this case, we want to send the results after the automated tests have finished executing, so we are using the
stagessection to specify the order in which the jobs should be executed.
- In order to share the generated artifacts, in this case our JUnit results file, we declare the
artifactspath in the
playwright_testsjob set it in the
In this example, test execution and results upload happen in two different jobs because they use different technologies, so they need a suitable container
image to run. If you want to have both in the same job, you will need to create a container with both Node.js and Python.
You can learn more about pipeline features on the official GitLab CI/CD documentation.
Step 1: Execute Playwright automated tests
playwright_tests job runs on a Node.js container, which has the main technologies under which Playwright can be executed.
As previously mentioned, the
artifacts parameter is necessary to make the JUnit report available in the following job, to be parsed by the TestRail CLI. Notice the
when: always parameter, which is necessary so that artifacts are kept even if the job fails due to automated tests also failing. The
paths parameter consists of the list of paths we want to make available, in this case the
test-results/ folder, which is where we have the Playwright project configured to output the JUnit report to.
allow_failure parameter is used so that the pipeline executes the next step even if tests fail, so the test results can be uploaded to TestRail.
script is made of a list of shell commands, starting by installing the Playwright project and it's dependencies and finally executing the automated tests.
playwright_tests: stage: playwright_tests image: node:latest artifacts: when: always paths: - test-results/ allow_failure: true script: - npm install - npx playwright install-deps - npx playwright install - npx playwright test
Step 2: Upload results to TestRail
trcli_upload_results job runs a shell script on a Python container to install and call the TestRail CLI, in order to parse the JUnit report generated by Playwright and send the results to TestRail.
dependencies parameter is necessary so that this job has access to the
playwright_tests JUnit results file, essential to the results upload.
The TestRail CLI is a Python packages hosted on the Python Package Index (PyPI) so, in order to install it, we simply execute the command
pip install trcli.
After the TestRail CLI is installed, there are a few mandatory arguments such as your TestRail instance address and credentials, the project you want to report to, the title for your test run and the path to the JUnit report. The run description argument is not mandatory, but we recommend using it to add a link from TestRail to your GitLab pipeline execution. This link can be obtained using the GitLab predefined variable
$CI_PIPELINE_URL, such as on the snippet below.
For more information about arguments, please check the TestRail CLI documentation.
trcli_results_upload: stage: trcli_results_upload image: python:latest dependencies: - playwright_tests script: | pip install trcli trcli -y \ -h https://TESTRAIL_INSTANCE.testrail.io/ \ --project "PROJECT NAME" \ -u USER_EMAIL \ -p PASSWORD \ parse_junit \ -f "test-results/junit-report.xml" \ --run-description "URL: $CI_PIPELINE_URL" \ --title "Automated Tests from GitLab CI/CD"
Setting up your working example
To use GitLab CI/CD pipelines, you need to have an active account in GitLab and also have your code hosted there.
Creating your own repository
We're going to setup the GitLab pipeline using our Playwright sample project, which you can know more about by checking our documentation about Integrating with Playwright.
By following the steps below, you can quickly set up a working example of a GitLab pipeline that executes an automated tests project and send the results to TestRail.
- Create a new project in GitLab
- Clone your newly created project on your local machine
- Download the files on the Playwright sample project and add them to your local project
- Create a
.gitlab-ci.ymlfile in the root of your project and add the example pipeline snippet provided in this article
- Replace TESTRAIL_INSTANCE, PROJECT NAME, USER_EMAIL and PASSWORD on the file content using your details (we recommend that you do not replace the password directly and use GitLab variables instead)
- Push your files to your project's repository in GitLab
Executing the pipeline
- This pipeline is designed for CI/CD, meaning it will run every time there is a new commit. The first time you push your code to the GitLab, the pipeline will immediately run, executing the automated tests project and sending the results to TestRail. To see your pipeline, you should go to CI/CD > Pipelines on the left hand panel. You should see the execution details of the first pipeline, and if everything ran as expected, the pipeline should have passed with a warning.
- By clicking the status of the pipeline, a new page with more execution details will be displayed, giving your an overview of your pipeline, composed by the
playwright_testsjob, which failed, and the
trcli_results_uploadjob, which was successful, as you can see in the image below.
- By clicking the
playwright_testsjob, you can see execution details for that job, such as duration, artifacts and the console logs. With a quick skim through the console logs, we see that 1 test failed, which renders the job as failed too.
- By clicking the
trcli_results_uploadjob, you can see the same information. By looking at the logs, from the TestRail CLI output, we find that 7 test results were sent to TestRail and a URL is available to navigate to the test run under which these results were submitted.
- By clicking on the URL provided in the logs, we can immediately navigate to TestRail and see the newly created test run with all the test results from our Playwright automated tests project. Also notice the URL back to the GitLab pipeline, which is useful to maintain traceability in case you need to debug any issues that occurred during that test run.
- As previously mentioned, after you first push your test automation code, you will be able to see one pipeline, which was automatically triggered by your commit. In case you want to run your automated test cases again without making any changes to the code of your automation project, you, you can manually trigger a new pipeline by clicking the Run pipeline button on the main screen, selecting the branch you want to execute and submitting that run.
Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!
You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.