Integrating with JMeter

The Apache JMeter™ application is an open-source, 100% pure Java tool designed for load testing functional behavior and measuring performance. Originally developed for testing web applications, JMeter has since expanded to support various testing functions.

It can be used to assess performance across both static and dynamic resources, including web applications, simulating heavy loads on a server (or a group of servers), a network, or an object to evaluate its resilience and analyze overall performance under different load conditions.

In this tutorial, we’ll explore how to integrate JMeter test results into TestRail using the TestRail CLI. This integration enables you to centralize automated test results and leverage TestRail’s analytical and reporting capabilities.

Overview

In this tutorial, we’ll use a sample project to walk you through setting up a JMeter performance test project that integrates with the TestRail CLI to upload the generated test results.

By the end of this tutorial, you will be able to:

• Create performance tests for a JMeter sample project
• Install the TestRail CLI
• Configure your TestRail instance
• Use the CLI to execute performance tests and upload results to TestRail
• View your test cases and results in TestRail

Prerequisites

To create and run the sample project, you’ll need to install Apache JMeter, the JUnit Reporter Plugin, and Python (to install and run the TestRail CLI). These tools will be used to generate and import test results into TestRail.

Create a sample project 

After you have installed JMeter, it's time to create a sample project.

Note: If you prefer to use a preconfigured project, you can clone the sample project from the TestRail Automation Framework Sample repository.

Alternatively, if you'd like to create everything from scratch, follow these steps:

• When opening JMeter GUI, you'll have access to an area similar to the one presented in the image below. In here, you can change the name of your project.

  

• Right-click on the project and select Add → Thread Group. This serves as the starting point for any test plan. All controllers and samplers must be placed under a thread group.
• Configure the Thread Group: JMeter provides various configurable elements within the Thread Group. Elements are configurable:
  1. Number of Threads: Each thread runs the entire test plan independently of other test threads
  2. Ramp-up period: Specifies how long JMeter should take to reach the full number of threads. Example: If 10 threads are used and the ramp-up period is set to 100 seconds, JMeter will incrementally start threads over this duration.
  3. Loop Count: By default, the thread group is configured to loop once through its elements. You can adjust this setting as needed.
• Then, right click on the Thread Group to configure the action - an HTTP Request.
  1. Server Name or IP: For this sample project, simply enter the target server’s address in the HTTP request. You can rename the file as needed. Example: The selected server name is www.google.com (there's no need to include http://)
• Then, right click again the Thread Group to configure the listener - a JUnit Reporter.
  1. Filename: Specify where the performance test report file will be saved. This file will be used by TRCLI to upload execution results to TestRail.
  2. Key Performance Indicators (KPIs): For our test case, we will configure two metrics to evaluate the results, which will be treated as Test Cases. When TRCLI reads the JUnit Report, it will generate Test Case names by combining the Metric, Comparator, and Threshold fields. The value entered in the Name field in JMeter serves only as a reference to easily identify the metric being tested. Each KPI will be represented as a Test Case in TestRail.

  Name	Metric	Comparator	Threshold	Test Objective
  Average less than 1000ms	MEAN		1000	Average response time must be less than 1000 milliseconds
  Not more than 1% sample errors	ERRORS	<=	.01	Error rate must be less than 0.01% for executed test

  

Executing the sample project

Executing using JMeter GUI:

To manually execute the performance test, simply click the green play button. Once the test starts, the Start button will turn grey and become inoperable. You can monitor the execution process in the top-right corner.

To determine when the execution is complete, wait for the Start button to turn green again. At that point, the execution report will be available for analysis.

If you followed the example metrics configuration, your report will look similar to the one below. The report is stored in the folder you specified when configuring the JUnit Reporter listener.

<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0"
failures="0"
name="JUnit Reporter_ TRintegration"
skipped="0"
tests="2">
<testcase classname="Average less than 100ms" name="MEAN(.*) &lt; 1000"/>
<testcase classname="Not more than 1% sample error" name="ERRORS(.*) &lt;= .01"/>
</testsuite>

Executing using JMeter command line

./jmeter.bat --nongui --testfile "C:\path\to\your\trcli\config\file\jmeter_perf_test.jmx"

Importing results to TestRail

Installing the TestRail CLI

Given you already have Python installed on your machine, installing the TestRail CLI is as simple as executing the following command on your command line.

$ pip install trcli

Configuring TestRail

Secondly, you need to configure your TestRail instance according to the instructions below.

1. Enable the TestRail API by going to Admin > Site Settings, click on the API tab, and checking the Enable API option.
2. Create a Custom Field in order to map your automated test cases code to the actual TestRail cases. You can do so by going to Admin > Customizations and clicking Add Field. After you’ve reached the field creation screen, there are two requirements for this custom field:
   • The System Name must be automation_id
   • The Type must be Text

Sending results to TestRail

After you’re done installing the TestRail CLI and finished configuring your TestRail instance, there’s only two extra step before uploading your results.

First, you need to create your TRCLI config file (.yml file format), that should look like below

host: https://<TESTRAIL_HOST>.testrail.io
project: Performance_Integration
username: user@testrail.com
password: password
title: JMeter Performance Test Run

• host: TestRail instance link
• project: where your run will be saved
• username and password: your credentials to the instance identified above
• title: the name of your test run, where results will be stored

Execute TRCLI using a shell script. The TestRail CLI can be executed efficiently via a shell script. Below is a sample script, designed for Windows using Git Bash (saved it as a .sh):

clear
bold=$(tput bold)
TRCLI_CONFIG_PATH="C:\path\to\your\trcli\config\file\trcli-config.yml"
TRCLI_TEST_FILE="C:\path\to\your\trcli\config\file\jmeter_perf_test.jmx"
JMETER_RESULTS="C:\path\to\your\file\sample_results.xml"

echo "($bold)-------------------------------------"
echo "($bold)--- JMeter TestRail Sample Project --"
echo "($bold)-------------------------------------"

echo "($bold)... verify if TRCLI is installed:"
echo
trcli
echo

echo "($bold)... execute JMeter Load Test:"
echo
./jmeter.bat --nongui --testfile "$TRCLI_TEST_FILE"
echo

echo "($bold)... executing TestRail TRCLI to upload results in a new Test Run:"
echo
trcli -y --config "$TRCLI_CONFIG_PATH" parse_junit --file "$JMETER_RESULTS" --title "JMeter Demo"
echo

echo "($bold)... execution completed. New Test Run created."

Ensure that the file name after the -f option matches the correct path to your report file, especially if you’ve changed its default location. In this example, a variable was used to store the file path, but you can also specify it directly. All other options should be configured to match your TestRail instance and project.

To explore additional command-line options, refer to the TestRail CLI README.md (available in the project repository), the TRCLI documentation article, or the embedded CLI help by running the following commands:

$ trcli --help
$ trcli parse_junit --help

Visualizing the results on TestRail

Now, if you go to the Test Cases page in your TestRail project, you’ll notice that the TestRail CLI automatically created the test cases that were on your test results report.

On the Test Runs & Results page, we can see that a test run with the name JMeter Test Run was created. By opening it we can dig further into the details of each automated test result and perform a high level analysis of why any test is failing since the error message provided by the test automation framework is also registered on the test result, as you can see on the image below.

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.