Integrating with K6

k6 is an open-source, developer-friendly, and extensible load testing tool, that allows to prevent performance issues and proactively improve reliability. Using k6, you can test the reliability and performance of your application and infrastructure, preventing errors and SLO breaches, enabling to build resilient and high-performing applications that scale.

In this tutorial, we’ll explore how to integrate k6 test results into TestRail using the TestRail CLI. This integration enables you to centralize automated test results and leverage TestRail’s analytical and reporting capabilities.

Overview

In this tutorial, we’ll use a sample project to walk you through setting up a k6 performance test project that integrates with the TestRail CLI to upload the generated test results.

By the end of this tutorial, you will be able to:

• Execute tests from a simple k6 project
• Install the TestRail CLI
• Configure your TestRail instance
• Use the CLI to execute performance tests and upload results to TestRail
• View your test cases and results in TestRail

Prerequisites

To be able to run a k6 tests project you can install it through npm, so Node.js is considered a pre-requisite.

You will also need Python to install and run the TestRail CLI that will be used to import the test results to TestRail.

Installing the sample project

Let’s start by fetching the sample project code and installing the required dependencies.

1. Clone or download the sample project
2. Open your command prompt on the project root folder and execute the command below

$ npm install -g k6
$ npm install -g k6-to-junit

Note: k6-to-junit is a simple command line utility to convert the stdout of k6 tests into a junit xml format, ready to be used for TRCLI. Currently this just works by looking for thresholds in the output and creating a testcase for each threshold in the output xml file.

Exploring the sample project

Use your favorite IDE to open the sample project and start digging into the test files. We’ve kept the automated tests code simple so that the focus of this tutorial remains on how to import the execution results. These tests consist of simple interactions with a free fake and reliable API for testing.

import http from 'k6/http';
import { check, group } from 'k6';

const baseUrl = 'http://jsonplaceholder.typicode.com';

export const options = {
  vus: 5,
  stages: [
    { duration: '10s', target: 10 }
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration:=['p(95)="p(95)'],
    http_reqs: ['count = 1400', 'count <= 1600'],
  }
}
 export default function () {
  group('JSON Placeholder Performance Testing', function () {
    group('Posts endpoint', function () {
      const res = http.get(`${baseUrl}/posts`);
      check(res, {
        'is status code 200': (r) = r.status === 200,
      });
    });
  });
}

Understanding the configuration and test objectives:

1. vus: 5: 5 Virtual Users (VUs) will start the test.

2. stages:

   • duration: '10s': This stage (i.e., the test execution) lasts for 10 seconds.
   • target: 10: The number of VUs will gradually increase to 10 over the 10-second duration. So it will begin with 5 VUs and will achieve 10 VUs.

3. thresholds: These are performance criteria that the test must meet:

   • http_req_failed: ['rate<0.01']: Less than 1% of HTTP requests should fail (request response 4xx or 5xx).
   • http_req_duration: ['p(95)<500']: The 95th percentile of HTTP request durations should be under 500ms (i.e., 95% of requests must complete in less than 500ms).
   • http_reqs: ['count=1400', 'count<=1600']: The test should result in at least 1400 HTTP requests and no more than 1600 HTTP requests.

Executing the sample project

On the same command prompt, execute the following command to run the k6 tests in the project and save the results in JUnit XML format.

$ k6 run test_k6.js | k6-to-junit ./reports/junit.xml

If the k6 command was correctly executed, you should be able to see your test results on the reports folder.  There should also be a file called junit.xml, as per our command options, with test results in JUnit XML format. This is the file which will be parsed by the TestRail CLI in order to create the test run and upload your test results to TestRail on the next step.

Importing results to TestRail

After the tests have been executed and the JUnit report files are generated, you can easily import your test results (and test cases!) to TestRail. This will bring visibility to your automated test runs and will enable you to look at the big picture of how you’re actually testing your app all within TestRail.

Installing the TestRail CLI

Given you already have Python installed on your machine, installing the TestRail CLI is as simple as executing the following command on your command line.

$ pip install trcli

Configuring TestRail

Secondly, you need to configure your TestRail instance according to the instructions below.

1. Enable the TestRail API by going to Admin > Site Settings, click on the API tab, and checking the Enable API option.
2. Create a Custom Field in order to map your automated test cases code to the actual TestRail cases. You can do so by going to Admin > Customizations and clicking Add Field. After you’ve reached the field creation screen, there are two requirements for this custom field:
   • The System Name must be automation_id
   • The Type must be Text

Sending results to TestRail

After you’re done installing the TestRail CLI and finished configuring your TestRail instance, you can upload your test results by simply using a one-liner such as the one below.

$ trcli -y \
>  -h https://INSERT-INSTANCE-NAME.testrail.io \
>  --project "My Project" \
>  --username INSERT-EMAIL \
>  --password INSERT-PASSWORD \
>  parse_junit \
>  --title "k6 Automated Test Run" \
>  -f "./test-results/junit-report.xml"

Note that the file name after the -f option should match the path to your report file in case you change its default location. All others options should be according to your TestRail instance and project. You can check other command line options by checking the TestRail CLI README.md file on the project repository, the TRCLI documentation article, or the embedded CLI help through the commands below.

$ trcli --help
$ trcli parse_junit --help

Visualizing the results on TestRail

Now, if you go to the Test Cases page in your TestRail project, you’ll notice that the TestRail CLI automatically created the test cases that were on your test results report. 

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.