Integrating with Azure Pipelines

Azure Pipelines is the CI/CD orchestration platform within the Microsoft Azure DevOps toolset that allows you to continuously build, test, and deploy to any platform and cloud. You can manage hardware and VMs by using Microsoft cloud-hosted agents, or by setting up your own hosted agents. It also provides full CI/CD pipeline support for every major platform and tool, allowing you to integrate with Azure Repos Git, GitHub, Bitbucket, among others.

As part of the pipeline, teams will commonly want to run their automated tests. To improve visibility into your automated tests and manage all your test cases and test results—both manual and automated—more easily, you can send your automated test results directly from Azure Pipelines to TestRail. In this article, we will explore how you can take advantage of the TestRail CLI in your Azure Pipelines setup to easily accomplish this task. 

Prerequisites

In case you want to use a self-hosted agent (either because you want to manage your own executors or simply didn't subscribe to cloud-hosted executors), you can get the scripts to build and execute a docker agent with Python 3.10.12 preinstalled in order to execute the TestRail CLI from TestRail's automation integration samples repository. 

To build an execute, all you need to do is replace the <AZURE_URL> placeholder with the URL to your instance (i.e.: https://dev.azure.com/yourorganization) and the <TOKEN> placeholder with your Azure Personal Access Token, and execute the following commands in a terminal starting in the same folder.

docker build -t azureagent:latest .
docker run -e AZP_URL="<AZURE_URL>" -e AZP_TOKEN="<TOKEN>" -e AZP_AGENT_NAME="SelfHostedLinux" azureagent:latest

Using the TestRail CLI in Azure Pipelines

Pipelines can be created by adding a azure-pipelines.yml file to the root of any compatible repository. This file allows you to specify your processes in a structured manner using YML syntax. To compose your pipeline, you can define multiple types of steps, such as tasks to accomplish goals such as setting up the environment, as well as scripts where you can write shell commands for any purpose.

The easiest way to integrate TestRail with Azure Pipelines is by configuring the TestRail CLI to send automated test results to TestRail in one of the steps in your azure-pipelines.yml file. Below is a sample of how an Azure Pipeline using Playwright as test automation framework and the TestRail CLI to send test results to TestRail would look like.

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '16.x'
  displayName: 'Install Node.js'
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.10.12'
  displayName: 'Setup Python'

- script: |
    npm install
    npx playwright install-deps
    npx playwright install
    npx playwright test
  displayName: 'Run Playwright tests'
  continueOnError: true
- script: |
    pip install trcli
    trcli -y \
      -h "https://INSTANCE-NAME.testrail.io" \
      --project "PROJECT-NAME" \
      -u "USERNAME" \
      -p "PASSWORD" \
      parse_junit \
      -f "test-results/junit-report.xml" \
      --title "Playwright automated tests" \
      --run-description "$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_build/results?buildId=$(Build.BuildId)"
  displayName: 'Upload results with TestRail CLI'

trigger:
- main

pool: Default

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '16.x'
  displayName: 'Install Node.js'
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.10.12'
    disableDownloadFromRegistry: true
  displayName: 'Setup Python'

- script: |
    npm install
    npx playwright install-deps
    npx playwright install
    npx playwright test
  displayName: 'Run Playwright tests'
  continueOnError: true
- script: |
    pip install trcli
    trcli -y \
      -h "https://INSTANCE-NAME.testrail.io" \
      --project "PROJECT-NAME" \
      -u "USERNAME" \
      -p "PASSWORD" \
      parse_junit \
      -f "test-results/junit-report.xml" \
      --title "Playwright automated tests" \
      --run-description "$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_build/results?buildId=$(Build.BuildId)"
  displayName: 'Upload results with TestRail CLI'

Breaking down the file, we can see the following:

• The trigger which is set to start the job on any push to main for a CI approach
• The pool which defines the agent where the steps will run and varies depending on whether you want to use a cloud-hosted agent or a self-hosted one
• The steps contain two tasks, to set up the environment, and two scripts, to run our tests and upload the results
  
  1. task: NodeTool@0 
  2. task: UsePythonVersion@0
  3. script (Run Playwright tests)
  4. script (Upload results with TestRail CLI)

You can learn more about how to configure your pipeline in the official Azure Pipelines documentation.

Step 1: Install Node.js

Using the NodeTool@0 task allows us to easily install node on the agent, which will be required to execute our Playwright tests.

- task: NodeTool@0
  inputs:
    versionSpec: '16.x'
  displayName: 'Install Node.js'

Step 2: Setup Python

The UsePythonVersion@0 task sets up Python, which is required to install and execute the TestRail CLI. This task needs to be configured differently depending on whether you are running your job on a cloud-hosted agent or a self-hosted one.

For cloud-hosted agents, you can simply add the versionSpec and the task will take care of downloading the requested version and install it on the agent.

For self-hosted agents, the task is unable to download and install Python, so it needs to already be configured in the agent and you need to add the disableDownloadFromRegistry: true input configuration. Note that by using the code provided in this tutorial for setting up a self-hosted agent, your agent will already have Python 3.10.12 pre-installed, so you do not need to worry about installing it yourself.

- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.10.12'
  displayName: 'Setup Python'

Step 3: Run Playwright tests

The Run Playwright tests step contains a shell script that installs the Playwright project, along with all the required system dependencies and executes the tests project. Note that the project is configured to generate a JUnit report in the test-results/ folder.

The continueOnError parameter is used so that the pipeline executes the next step even if tests fail, so the test results can be uploaded to TestRail.

- script: |
    npm install
    npx playwright install-deps
    npx playwright install
    npx playwright test
  displayName: 'Run Playwright tests'
  continueOnError: true

Step 4: Upload results with TestRail CLI

The Upload results with TestRail CLI step runs a shell script to install and call the TestRail CLI, in order to parse the JUnit report generated by Playwright and send the results to TestRail.

The TestRail CLI is a Python packages hosted on the Python Package Index (PyPI) so, in order to install it, we simply execute the command pip install trcli.

After the TestRail CLI is installed, there are a few mandatory arguments such as your TestRail instance address and credentials, the project you want to report to, the title for your test run and the path to the JUnit report. The --run-description argument is not mandatory, but we recommend using it to add a link from TestRail to your Azure Pipelines job execution. The example code contains a sequence of predefined pipeline variables which output the link to your job execution.

- script: |
    pip install trcli
    trcli -y \
      -h "https://INSTANCE-NAME.testrail.io" \
      --project "PROJECT_NAME" \
      -u "USERNAME" \
      -p "PASSWORD" \
      parse_junit \
      -f "test-results/junit-report.xml" \
      --title "Playwright automated tests" \
      --run-description "$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_build/results?buildId=$(Build.BuildId)"
  displayName: 'Upload results with TestRail CLI'

For more information about arguments, please check the TestRail CLI documentation.

Working example using Playwright

To use Azure Pipelines, you need to have an active account in Azure DevOps. Your code can be hosted in Azure Repos within your Azure DevOps instance, or you can also use an external source control tool such as GitHub or Bitbucket. In this example we will use Azure Repos.

Creating your own repository

We're going to setup the Azure pipeline using our Playwright sample project, which you can know more about by checking our documentation about Integrating with Playwright.

By following the steps below, you can quickly set up a working example of an Azure pipeline that executes an automated tests project and send the results to TestRail.

1. Create a new project in Azure DevOps
   
   
   
   
2. Go to the Repos section and create a new repository with the files in our Playwright sample project
   
   
   
   
3. Go to the Pipelines section and click Create Pipeline
   
   
   
   
4. Select your preferred repository type, in this case Azure Repos Git
   
   
   
   
5. Select the repository you have just created
   
   
   
   
6. Select any pipeline template to start with
   
   
   
   
7. Replace the template with the YML file provided above (do not forget to replace the placeholders INSTANCE-NAME, PROJECT-NAME, USERNAME and PASSWORD - we recommend that you do not replace the password directly and use Azure secret variables instead)
   
   
   
   
8. Click Save and run and proceed with the commit which will create the azure-pipelines.yml file in your repository
   
   
   
   
9. Since the pipeline trigger is a push to the master branch, it is immediately queued to be executed
   
   
   
   
10. By clicking in the job, you will be able to see its execution logs and by the very end you'll see the TestRail CLI logs which generate a link to the test run where the Playwright test results were reported
    
    
    
    
11. By clicking the link, you'll be redirected to TestRail, where you can see your test results and debug information, along with a link back to your job execution for traceability and ease of navigation
    
    

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.