Integrating with GitLab CI/CD

With GitLab CI/CD you can automated all the steps required to build, test and deploy your code. Typically teams use GitLab to store their source code, and with the CI/CD capabilities they also build and test their code inside GitLab by creating pipelines to streamline building and testing the code before being deployed.

As part of the pipeline, teams will commonly want to run their automated tests. To improve visibility into your automated tests and manage all your test cases and test results—both manual and automated—more easily, you can send your automated test results directly from GitLab to TestRail. In this article, we will explore how you can take advantage of GitLab CI/CD and the TestRail CLI to easily accomplish this task. 

If you want to see more examples, please take a look at the Automation Frameworks Integration .gitlab-ci.yml file.

Using the TestRail CLI in GitLab pipelines

GitLab pipelines can be created by adding a .gitlab-ci.yml file to the root of a GitLab repository. This file allows you to specify your processes in a structured manner using YML syntax. You can define the multiple stages of your pipeline using jobs that may run either sequentially or in parallel. 

The easiest way to integrate TestRail with GitLab CI/CD is by configuring the TestRail CLI to send automated test results to TestRail in one of the stages in your .gitlab-ci.yml file. Below is a sample of how a GitLab pipeline using Playwright as test automation framework and the TestRail CLI to send test results to TestRail would look like.

stages:
  - playwright_tests
  - trcli_results_upload

playwright_tests:
  stage: playwright_tests
  image: node:latest
  artifacts:
    when: always
    paths:
      - test-results/
  allow_failure: true
  script:
    - npm install
    - npx playwright install-deps
    - npx playwright install
    - npx playwright test

trcli_results_upload:
  stage: trcli_results_upload
  image: python:latest
  dependencies:
    - playwright_tests
  script: |
    pip install trcli
    trcli -y \
      -h https://TESTRAIL_INSTANCE.testrail.io/ \
      --project "PROJECT NAME" \
      -u USER_EMAIL \
      -p PASSWORD \
      parse_junit \
      -f "test-results/junit-report.xml" \
      --run-description "URL: $CI_PIPELINE_URL" \
      --title "Automated Tests from GitLab CI/CD"

Breaking down the file, there are a couple of details you should pay attention to:

• There are two jobs defined in this pipeline file: 
  1. playwright_tests where we execute our sample automated tests project
  2. trcli_results_upload where we upload the generated results to TestRail
• By default, jobs are executed in parallel. In this case, we want to send the results after the automated tests have finished executing, so we are using the stages section to specify the order in which the jobs should be executed.
• In order to share the generated artifacts, in this case our JUnit results file, we declare the artifacts path in the playwright_tests job set it in the dependencies of the trcli_results_upload job.

You can learn more about pipeline features on the official GitLab CI/CD documentation.

Step 1: Execute Playwright automated tests

The playwright_tests job runs on a Node.js container, which has the main technologies under which Playwright can be executed.

As previously mentioned, the artifacts parameter is necessary to make the JUnit report available in the following job, to be parsed by the TestRail CLI. Notice the when: always parameter, which is necessary so that artifacts are kept even if the job fails due to automated tests also failing. The paths parameter consists of the list of paths we want to make available, in this case the test-results/ folder, which is where we have the Playwright project configured to output the JUnit report to.

The allow_failure parameter is used so that the pipeline executes the next step even if tests fail, so the test results can be uploaded to TestRail.

The script is made of a list of shell commands, starting by installing the Playwright project and it's dependencies and finally executing the automated tests.

playwright_tests:
  stage: playwright_tests
  image: node:latest
  artifacts:
    when: always
    paths:
      - test-results/
  allow_failure: true
  script:
    - npm install
    - npx playwright install-deps
    - npx playwright install
    - npx playwright test

Step 2: Upload results to TestRail

The trcli_upload_results job runs a shell script on a Python container to install and call the TestRail CLI, in order to parse the JUnit report generated by Playwright and send the results to TestRail. 

The dependencies parameter is necessary so that this job has access to the playwright_tests JUnit results file, essential to the results upload.

The TestRail CLI is a Python packages hosted on the Python Package Index (PyPI) so, in order to install it, we simply execute the command pip install trcli.

After the TestRail CLI is installed, there are a few mandatory arguments such as your TestRail instance address and credentials, the project you want to report to, the title for your test run and the path to the JUnit report. The run description argument is not mandatory, but we recommend using it to add a link from TestRail to your GitLab pipeline execution. This link can be obtained using the GitLab predefined variable $CI_PIPELINE_URL, such as on the snippet below.

For more information about arguments, please check the TestRail CLI documentation.

trcli_results_upload:
  stage: trcli_results_upload
  image: python:latest
  dependencies:
    - playwright_tests
  script: |
    pip install trcli
    trcli -y \
      -h https://TESTRAIL_INSTANCE.testrail.io/ \
      --project "PROJECT NAME" \
      -u USER_EMAIL \
      -p PASSWORD \
      parse_junit \
      -f "test-results/junit-report.xml" \
      --run-description "URL: $CI_PIPELINE_URL" \
      --title "Automated Tests from GitLab CI/CD"

Setting up your working example

To use GitLab CI/CD pipelines, you need to have an active account in GitLab and also have your code hosted there.

Creating your own repository

We're going to setup the GitLab pipeline using our Playwright sample project, which you can know more about by checking our documentation about Integrating with Playwright.

By following the steps below, you can quickly set up a working example of a GitLab pipeline that executes an automated tests project and send the results to TestRail.

1. Create a new project in GitLab
2. Clone your newly created project on your local machine
3. Download the files on the Playwright sample project and add them to your local project
4. Create a .gitlab-ci.yml file in the root of your project and add the example pipeline snippet provided in this article
5. Replace TESTRAIL_INSTANCE, PROJECT NAME, USER_EMAIL and PASSWORD on the file content using your details (we recommend that you do not replace the password directly and use GitLab variables instead)
6. Push your files to your project's repository in GitLab

Executing the pipeline

1. This pipeline is designed for CI/CD, meaning it will run every time there is a new commit. The first time you push your code to the GitLab, the pipeline will immediately run, executing the automated tests project and sending the results to TestRail. To see your pipeline, you should go to CI/CD > Pipelines on the left hand panel. You should see the execution details of the first pipeline, and if everything ran as expected, the pipeline should have passed with a warning.
   
   
2. By clicking the status of the pipeline, a new page with more execution details will be displayed, giving your an overview of your pipeline, composed by the playwright_tests job, which failed, and the trcli_results_upload job, which was successful, as you can see in the image below.
   
   
3. By clicking the playwright_tests job, you can see execution details for that job, such as duration, artifacts and the console logs. With a quick skim through the console logs, we see that 1 test failed, which renders the job as failed too.
   
   
4. By clicking the trcli_results_upload job, you can see the same information. By looking at the logs, from the TestRail CLI output, we find that 7 test results were sent to TestRail and a URL is available to navigate to the test run under which these results were submitted.
   
   
5. By clicking on the URL provided in the logs, we can immediately navigate to TestRail and see the newly created test run with all the test results from our Playwright automated tests project. Also notice the URL back to the GitLab pipeline, which is useful to maintain traceability in case you need to debug any issues that occurred during that test run.
   
   
6. As previously mentioned, after you first push your test automation code, you will be able to see one pipeline, which was automatically triggered by your commit. In case you want to run your automated test cases again without making any changes to the code of your automation project, you, you can manually trigger a new pipeline by clicking the Run pipeline button on the main screen, selecting the branch you want to execute and submitting that run.
   
   

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.