Integrating with WebdriverIO

WebdriverIO stands as a robust automation testing framework, offering a simplified approach to test web applications.

In this tutorial, we will explore how to integrate the test results generated by a WebdriverIO automated test run within TestRail using the TestRail CLI. This will allow you to centralize your automated test results and take advantage of all the analytical and reporting capabilities TestRail provides.

Overview

Throughout this tutorial, we will be using a sample project to guide you through the process of setting up a WebdriverIO automated tests project that is compatible with the TestRail CLI and uploading the generated test results.

After reading through this tutorial, you will be able to:

• Execute tests from a simple WebdriverIO project
• Install the TestRail CLI
• Configure your TestRail instance
• Run the CLI
• See your test cases and test results on TestRail

Prerequisites

To be able to run a WebdriverIO tests project you can install it through npm, or any of the other options on the WebdriverIO setup guide. However, for this tutorial, we will install it through npm.  Since WebdriverIO is installed via Node.js, you must install Node.js first.

Additionally, you will need Python to install and run the TestRail CLI, which is responsible for importing test results into TestRail.

Installing the sample project

Let’s start by fetching the sample project code and installing the required dependencies.

1. Clone or download the sample project
2. Open your command prompt on the project root folder and execute the command below

$ npm install

Exploring the sample project

Use your preferred IDE to open the sample project and start digging into the test files. We’ve kept the automated test code simple so that the focus of this tutorial remains on how to import the execution results. These tests consist of simple interactions with a  Login page.

(sample feature file)

Feature: The Internet Website Access

  Scenario: As a user, I can log into the secure area

    Given I am on the login page
    When I login with  and 
    Then I should see a flash message saying 

    Examples:
      | username | password             | message                        |
      | tomsmith | SuperSecretPassword! | You logged into a secure area! |

Scenario: As a user, I can't log into the secure area

    Given I am on the login page
    When I login with  and 
    Then I should see a flash message saying 

    Examples:
      | username | password             | message                        |
      | tomsmith | barfoo               | Your password invalid!         |

(sample step definitions)

import { Given, When, Then } from '@wdio/cucumber-framework';
import { expect, $ } from '@wdio/globals'

import LoginPage from '../pageobjects/login.page.js';
import SecurePage from '../pageobjects/secure.page.js';

const pages = {
    login: LoginPage
}

Given(/^I am on the (\w+) page$/, async (page) = {
    await pages[page].open()
});

When(/^I login with (\w+) and (.+)$/, async (username, password) = {
    await LoginPage.login(username, password)
});

Then(/^I should see a flash message saying (.*)$/, async (message) = {

    const now = new Date();
    const timestamp = now.toISOString().replace(/[^\w]/g, '-');
    const screenshotName = `screenshots-login-${timestamp}.png`;
    await browser.saveScreenshot(`./results/screenshots/${screenshotName}`);

    await expect(SecurePage.flashAlert).toBeExisting();
    await expect(SecurePage.flashAlert).toHaveText(expect.stringContaining(message));
});

Executing the sample project

On the same command prompt, execute the following command to run the WebdriverIO tests in the project and save the results in JUnit XML format.

npx wdio run wdio.conf.js

If the WebdriverIO command is correctly executed, you should be able to see your test results in the results folder. The project is configured to generate one report file.

There should also be a file called webdriverIO-test-results.xml (as defined in wdio.conf.js file) with test results in JUnit XML format. This is the file that will be parsed by the TestRail CLI to create the test run and upload your test results to TestRail in the next step.

Importing results to TestRail

Installing the TestRail CLI

Given you already have Python installed on your machine, installing the TestRail CLI is as simple as executing the following command on your command line.

$ pip install trcli

Configuring TestRail

Secondly, you need to configure your TestRail instance according to the instructions below.

1. Enable the TestRail API by going to Admin > Site Settings, click on the API tab, and checking the Enable API option.
2. Create a Custom Field in order to map your automated test cases code to the actual TestRail cases. You can do so by going to Admin > Customizations and clicking Add Field. After you’ve reached the field creation screen, there are two requirements for this custom field:
   • The System Name must be automation_id
   • The Type must be Text

After the tests have been executed and the JUnit report files are generated, you can easily import your test results (and test cases!) to TestRail. This will bring visibility to your automated test runs and will enable you to look at the big picture of how you’re actually testing your app all within TestRail.

Sending results to TestRail

Once you have installed the TestRail CLI and configured your TestRail instance, you can upload your test results using the following command:

$ trcli -y \
>  -h https://INSERT-INSTANCE-NAME.testrail.io \
>  --project "My Project" \
>  --username INSERT-EMAIL \
>  --password INSERT-PASSWORD \
>  parse_junit \
>  --title "Automation Demo - WebdriverIO" \
>  -f "results/test-results.xml"

Note that the file name after the -f option should match the path to your report file in case you change its default location. All others options should be according to your TestRail instance and project. You can check other command line options by checking the TestRail CLI README.md file on the project repository, the TRCLI documentation article, or the embedded CLI help through the commands below..

$ trcli --help
$ trcli parse_junit --help

Visualizing the results on TestRail

When you navigate to the Test Cases page in your TestRail project, you’ll see that the TestRail CLI has automatically created test cases based on your test results report.

Each test case is assigned a unique Automation ID, generated by combining the classname and name attributes from the JUnit report. This Automation ID allows TestRail CLI to match test cases each time it runs. If a test case with the same Automation ID already exists, it will be updated; otherwise, a new one will be created.

On the Test Runs & Results page, we can see that a test run with the name Automation Demo - WebdriverIO was created. By opening it we can dig further into the details of each automated test result and perform a high level analysis of why any test is failing since the error message provided by the test automation framework is also registered on the test result, along with any attachments such as the screenshot from when the test failure happened, as you can see on the image below.

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.