Integrating with Jenkins (freestyle)

Jenkins is the leading open source automation server, providing hundreds of plugins to support building, deploying and automating any project. Jenkins can be used as a simple CI server or turned into the continuous delivery hub for any project and it can be easily set up and configured via its web interface.

A common use case for Jenkins is building and testing your application whenever code changes are submitted. In order to take full advantage of TestRail and manage all your test cases and test results, both manual and automated, you can send your automated test results directly from Jenkins to TestRail. In this article, we will explore how you can take advantage of Jenkins freestyle projects  and the TestRail CLI to easily accomplish this task. 

Prerequisites

If you want to quickly setup a Jenkins instance to follow the steps, you can build the Dockerfile (based on the official Jenkins Docker image) below and run it by following these instructions:

1. Save the content below to a file named Dockerfile

   FROM jenkins/jenkins:lts
   
   USER root
   RUN apt-get update
   RUN apt-get install -y \
           build-essential \
           zlib1g-dev \
           libncurses5-dev \
           libgdbm-dev \
           libnss3-dev \
           libssl-dev \
           libreadline-dev \
           libffi-dev \
           libsqlite3-dev \
           wget \
           libbz2-dev
   RUN wget https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tgz && \
           tar -xvf Python-3.10.0.tgz && \
           cd Python-3.10.0 && \
           ./configure --enable-optimizations && \
           make -j 2 && \
           make altinstall && \
           python3.10 --version
   USER jenkins

2. Open your command line on the location you saved the file to

3. Execute the commands below

   docker build -t jenkins:local .
   docker run -p 8080:8080 -p 50000:50000 jenkins:local

4. Copy the key on the console and open http://localhost:8080 on your browser
5. Follow the install wizard

Creating a new Jenkins job

1. Open your Jenkins instance and click New Item

   

2. Enter your New Item name, select Freestyle project and click OK

   

Setting up the Git repository

In this example, we will be using a Git repository to fetch the automated tests project from. In this case, we will use a sample project from our GitHub project automation-frameworks-integration.

1. On the Source Code Management section, start by selecting Git

2. Set the Repository URL (in our example, https://github.com/gurock/automation-frameworks-integration.git)

3. Set the Branch Specifier (in our example, */main)

Configuring the build step

1. On the Build section, use the Add build step dropdown to add an Execute shell step

   

2. On the Execute shell step Command, insert the script with the commands required to:
   • Setup the environment
   • Execute the tests project
   • Upload the test results using the TestRail CLI 

   

Shell script breakdown

Below is a copy of the script used above and a brief explanation of each section.

# Go to test project folder
cd samples/python/pytest

# Setup virtual environment
python3.10 -m venv ./venv
. ./venv/bin/activate

# Install and run test project
pip install -r requirements.txt
pytest --junitxml "reports/junit-report.xml" "./tests" || pytest_exit_code=$?

# Install TestRail CLI and upload test results
pip install trcli
trcli -y \
  -h "https://INSTANCE.testrail.io" \
  -u "USER_EMAIL" \
  -p "API_TOKEN" \
  --project "PROJECT NAME" \
  parse_junit \
  --title "Automated Test Run" \
  --run-description "CI Build: ${BUILD_URL}" \
  -f "reports/junit-report.xml"
    
# Exit with Pytest exit code
exit $pytest_exit_code

# Go to test project folder

The script starts by navigating to the Pytest sample project base folder which contains the automated tests project we want to execute.

# Setup virtual environment

The next step is creating a Python virtual environment dedicated to this project, which is a good practice to make sure you start from a consistent baseline. After the virtual environment is created, you need to activate it, which means your shell will be using it instead of the global Python installation.

# Install and run test project

After having activated the virtual environment, you can directly install the Pytest tests project and execute it. Notice the --junitxml "reports/junit-report.xml" option, which commands Pytest to generate a JUnit style report to be used by the TestRail CLI. You can find more details about this on the documentation relating to Integrating with Pytest. Another detail on this script is the || pytest_exit_code=$? after the pytest command. This tells the shell to proceed even if tests fail, which causes Pytest to return a failure exit code and would otherwise stop the whole job execution. We will later exit the script with the pytest_exit_code to either pass or fail the build according to the test results. There are multiple ways to achieve this behavior and plugins that may help you doing so, this was the simplest way to fulfil the purpose of this example.

# Install TestRail CLI and upload test results

If the test execution finished successfully (whether there were failing tests or not), you should now have a JUnit style report to be able to upload your results to TestRail. To do so, you just install the TestRail CLI using pip and then execute it with your TestRail instance details. You'll need to replace the placeholders for the -h, -u, -p and --project. Another small but useful detail on the trcli call in this script is the option --run-description "CI Build: ${BUILD_URL}". This will add the current Jenkins build url to your TestRail test run, so you can easily navigate back to it. As a final recommendation, do not use your password or API token directly on the script, make use of the Jenkins credentials manager.

# Exit with Pytest exit code

As mentioned before, we simply exit the script with the original Pytest exit code, which will inform Jenkins of the result of the actual automated tests execution.

Running the job

1. After you have saved the job with the configurations above, it's time to run it. To do so, you simply hit the Build Now action on the left side panel.

   

2. After the job is complete you should be able to see the script was executed as expected by peeking at the logs on the Console Output page of the build. If the test execution completed without any bumps and the TestRail CLI was properly configured and found the test results JUnit report in the place it was meant to be, you should see on the logs that it submitted the test results to your TestRail instance.

   

3. Finally, if you open your TestRail instance and go to the Test Runs & Results page, you should be able to find the newly created execution named Automated Test Run and see your test results, along with the link to your Jenkins job build that generated this test run.

   

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.