Integrating with Jenkins (pipeline)

Jenkins is the leading open source automation server, providing hundreds of plugins to support building, deploying and automating any project. Jenkins can be used as a simple CI server or turned into the continuous delivery hub for any project and it can be easily set up and configured via its web interface.

A common use case for Jenkins is building and testing your application whenever code changes are submitted. In order to take full advantage of TestRail and manage all your test cases and test results, both manual and automated, you can send your automated test results directly from Jenkins to TestRail. In this article, we will explore how you can take advantage of Jenkins scripted pipeline jobs and the TestRail CLI to easily accomplish this task. 

Prerequisites

If you want to quickly setup a Jenkins instance to follow the steps, you can build the Dockerfile (based on the official Jenkins Docker image) below and run it by following these instructions:

1. Save the content below to a file named Dockerfile

   FROM jenkins/jenkins:lts
   
   USER root
   RUN apt-get update
   RUN apt-get install -y \
           build-essential \
           zlib1g-dev \
           libncurses5-dev \
           libgdbm-dev \
           libnss3-dev \
           libssl-dev \
           libreadline-dev \
           libffi-dev \
           libsqlite3-dev \
           wget \
           libbz2-dev
   RUN wget https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tgz && \
           tar -xvf Python-3.10.0.tgz && \
           cd Python-3.10.0 && \
           ./configure --enable-optimizations && \
           make -j 2 && \
           make altinstall && \
           python3.10 --version
   USER jenkins

2. Open your command line on the location you saved the file to

3. Execute the commands below

   docker build -t jenkins:local .
   docker run -p 8080:8080 -p 50000:50000 jenkins:local

4. Copy the key on the console and open http://localhost:8080 on your browser
5. Follow the install wizard

Creating a new Jenkins pipeline job

1. Open your Jenkins instance and click New Item

   

2. Enter your New Item name, select Pipeline and click OK

   

Configuring the pipeline

Scroll down to the Pipeline section and add the script below.

Jenkins scripted pipelines are composed of stages, inside which you can find steps with the commands to be executed. We are now going to breakdown the pipeline script by stages, but in case you are not familiar with the basics of Jenkins pipelines, you can learn more about it in the Jenkins pipeline syntax documentation.

pipeline {
    agent any

    stages {

        stage('Checkout') {
            steps {
                git url: 'https://github.com/gurock/automation-frameworks-integration.git', branch: 'main'
            }
        }

        stage('Setup virtual environment') {
            steps {
                sh """
# Go to test project folder
cd samples/python/pytest
# Setup virtual environment
python3.10 -m venv ./venv
"""
            }
        }

        stage('Execute automated tests') {
            steps {
                sh """
# Go to test project folder and activate virtual environment
cd samples/python/pytest
. ./venv/bin/activate

# Install and run test project
pip install -r requirements.txt
pytest --junitxml "reports/junit-report.xml" "./tests"
"""
            }
            post {
                always {
                    junit '**/reports/junit-report.xml'
                    archiveArtifacts 'reports/*'
                    sh """
# Go to test project folder and activate virtual environment
cd samples/python/pytest
. ./venv/bin/activate

# Install TestRail CLI and upload test results
pip install trcli
trcli -y \
  -h "https://YOUR_INSTANCE.testrail.io/" \
  -u "USER_EMAIL" \
  -p "USER_PASSWORD" \
  --project "PROJECT" \
  parse_junit \
  --title "Automated Test Run" \
  --run-description "CI Build: ${BUILD_URL}" \
  -f "reports/junit-report.xml"
"""
                }
            }
        }

    }
}

stage('Checkout')

The pipeline starts by checking out the code from our repository using the git command with the repository url and the branch to be checked out.

stage('Setup virtual environment')

The next step is creating a Python virtual environment dedicated to this project, which is a good practice to make sure you start from a consistent baseline. Note that in the next stages of the script you have to activate the virtual environment so that your shell uses it instead of the global Python installation.

stage('Execute automated tests')

The first step in this stage is activating the virtual environment, you can directly install the Pytest tests project and execute it. Next, you will run the tests project using the pytest command. Notice the --junitxml "reports/junit-report.xml" option, which commands Pytest to generate a JUnit style report to be used by the TestRail CLI. You can find more details about this on the documentation relating to Integrating with Pytest. 

In this same stage, there is an inner block called post { } and inside it there is an always { } block. This block contains commands that are always executed after the steps { } block is completed, not matter the outcome, which is the perfect situation to send the test results to TestRail. To do so, you need to once again activate the virtual environment, install the TestRail CLI using pip and then execute it with your TestRail instance details. You'll need to replace the placeholders for the -h, -u, -p and --project. Another small but useful detail on the trcli call in this script is the option --run-description "CI Build: ${BUILD_URL}". This will add the current Jenkins build url to your TestRail test run, so you can easily navigate back to it. As a final recommendation, do not use your password or API token directly on the script, make use of the Jenkins credentials manager.

Running the job

1. After you have saved the job with the configurations above, it's time to run it. To do so, you simply hit the Build Now action on the left side panel.

   

2. After the job is complete you should be able to see the script was executed as expected by peeking at the logs on the Pipeline Steps page of the build. If the test execution completed without any bumps and the TestRail CLI was properly configured and found the test results JUnit report in the place it was meant to be, you should see on the logs that it submitted the test results to your TestRail instance.

   

3. You can see more details about the TestRail CLI execution by clicking on the Console Output button on the step that executes it. It will display the output of the TestRail CLI, which contains a very useful link to the test run that was just created in TestRail.
   
   

4. Finally, if you click the link, it will open the test run in your TestRail instance. It should be named Automated Test Run and it should contain your test results, along with the link to your Jenkins job build that generated this test run.

   

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.