Integrating with Playwright

Playwright is a reliable end-to-end testing framework for modern web apps. It allows you to develop resilient test scripts that run on multiple browsers, supported by powerful tooling to help you write and debug tests.

In this tutorial, we will explore how to integrate the test results generated by a Playwright automated test run within TestRail using the TestRail CLI. This will allow you to centralize your automated test results and take advantage of all the analytical and reporting capabilities TestRail provides.

Overview

Throughout this tutorial, we will be using a sample project to guide you through the process of setting up a Playwright automated tests project that is compatible with the TestRail CLI and uploading the generated test results.

After reading through this tutorial, you will be able to:

• Execute tests from a simple Playwright project
• Install the TestRail CLI
• Configure your TestRail instance
• Run the CLI
• See your test cases and test results on TestRail

Prerequisites

To be able to run a Playwright tests project you can install it through npm, or any of the other options on the Playwright installation documentation, but for the purpose of this article we will be installing through npm. As a pre-requisite to install Playwright using npm all you need to do is install Node.js.

You will also need Python to install and run the TestRail CLI that will be used to import the test results to TestRail.

Installing the sample project

Let’s start by fetching the sample project code and installing the required dependencies.

1. Clone or download the sample project
2. Open your command prompt on the project root folder and execute the command below

$ npm install

The packages.json file contains the libraries we want to install for this project. In this case the only dependency is Playwright itself, which naturally depends on other libraries.

Exploring the sample project

Use your favorite IDE to open the sample project and start digging into the test files. We’ve kept the automated tests code simple so that the focus of this tutorial remains on how to import the execution results. These tests consist of simple interactions with a TO DO list web app, hosted by Playwright.

// @ts-check
const { test, expect } = require('@playwright/test');
const { randomUUID } = require('crypto');

test.beforeEach(async ({ page }) => {
  await page.goto('https://demo.playwright.dev/todomvc');
});

test.afterEach(async ({ page }, testInfo) => {
  if (testInfo.status !== testInfo.expectedStatus) {
    let screenshotPath = `test-results/screenshots/screenshot-${randomUUID()}.png`;
    await page.screenshot({ path: screenshotPath, fullPage: true });
    testInfo.annotations.push({ type: 'testrail_attachment', description: screenshotPath });
  }
});

const TODO_ITEMS = [
  'buy some cheese',
  'feed the cat',
  'book a doctors appointment'
];

test.describe('New Todo', () => {
  test('should allow me to add todo items', async ({ page }) => {
    // Create 1st todo.
    await page.locator('.new-todo').fill(TODO_ITEMS[0]);
    await page.locator('.new-todo').press('Enter');

    // Make sure the list only has one todo item.
    await expect(page.locator('.view label')).toHaveText([
      TODO_ITEMS[0]
    ]);

    // Create 2nd todo.
    await page.locator('.new-todo').fill(TODO_ITEMS[1]);
    await page.locator('.new-todo').press('Enter');

    // Make sure the list now has two todo items.
    await expect(page.locator('.view label')).toHaveText([
      TODO_ITEMS[0],
      TODO_ITEMS[1]
    ]);

    await checkNumberOfTodosInLocalStorage(page, 2);
  });
});

Attaching screenshots to the report

In order to easily capture and send screenshots from failed tests to TestRail, you can make use of custom JUnit reporter properties. To do so, you just need to add the testRailOptions as per the code below to your playwright config file.

// @ts-check
const { devices } = require('@playwright/test');

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */

const testRailOptions = {
  // Whether to add <properties> with all annotations; default is false
  embedAnnotationsAsProperties: true,
  // Where to put the report.
  outputFile: './test-results/junit-report.xml'
};

/**
 * @see https://playwright.dev/docs/test-configuration
 * @type {import('@playwright/test').PlaywrightTestConfig}
 */
const config = {
  /* (...) */ 
  
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: [
    ['list'],
    ['html', { outputFolder: 'test-results', open: 'never' }],
    ['junit', testRailOptions]
  ],
  
  /* (...) */
};

module.exports = config;

After having the JUnit reporter properly configured, you just need to add the logic to capture screenshots in case your test result is not the expected and add the path to the screenshot to a testrail_attachment property. You can achieve this with the code below, which was already present in the test code snippet above.

test.afterEach(async ({ page }, testInfo) => {
  if (testInfo.status !== testInfo.expectedStatus) {
    let screenshotPath = `test-results/screenshots/screenshot-${randomUUID()}.png`;
    await page.screenshot({ path: screenshotPath, fullPage: true });
    testInfo.annotations.push({ type: 'testrail_attachment', description: screenshotPath });
  }
});

Executing the sample project

On the same command prompt, execute the following command to run the Playwright tests in the project and save the results in JUnit XML format.

$ npx playwright test

If the Playwright command was correctly executed, you should be able to see your test results on the test-results folder. The playwright project is configured to generate two report files. By opening the file index.html you can see an interactive Playwright HTML report such as the one below.

There should also be a file called junit-report.xml, as per our robot command options, with test results in JUnit XML format. This is the file which will be parsed by the TestRail CLI in order to create the test run and upload your test results to TestRail on the next step.

<testsuites id="" name="" tests="5" failures="1" skipped="0" errors="0" time="11.882232000008225">
  <testsuite name="demo-todo-app.spec.js" timestamp="1664397866418" hostname="" tests="5" failures="1" skipped="0" time="15.464" errors="0">
    <testcase name="New Todo should allow me to add todo items" classname="demo-todo-app.spec.js › New Todo › should allow me to add todo items" time="2.369">
    </testcase>
    <testcase name="New Todo should clear text input field when an item is added" classname="demo-todo-app.spec.js › New Todo › should clear text input field when an item is added" time="1.906">
    </testcase>
    <testcase name="New Todo should append new items to the bottom of the list" classname="demo-todo-app.spec.js › New Todo › should append new items to the bottom of the list" time="2.634">
    </testcase>
    <testcase name="Mark all as completed should allow me to mark all items as completed" classname="demo-todo-app.spec.js › Mark all as completed › should allow me to mark all items as completed" time="7.32">
      <properties>
        <property name="testrail_attachment" value="test-results/screenshots/screenshot-f6f2df78-0eaa-487c-a7a3-97ec6bb66a9a.png">
        </property>
      </properties>
      <failure message="demo-todo-app.spec.js:73:3 should allow me to mark all items as completed" type="FAILURE">
        <![CDATA[  demo-todo-app.spec.js:73:3 › Mark all as completed › should allow me to mark all items as completed 

    Error: expect(received).toHaveClass(expected) // deep equality

    - Expected  - 3
    + Received  + 3

      Array [
    -   "complete",
    -   "complete",
    -   "complete",
    +   "completed",
    +   "completed",
    +   "completed",
      ]
    Call log:
      - expect.toHaveClass with timeout 5000ms
      - waiting for selector ".todo-list li"
      -   selector resolved to 3 elements

      76 |
      77 |     // Ensure all todos have 'completed' class.
    > 78 |     await expect(page.locator('.todo-list li')).toHaveClass(['complete', 'complete', 'complete']);
         |                                                 ^
      79 |     await checkNumberOfCompletedTodosInLocalStorage(page, 3);
      80 |   });
      81 |

        at C:\Github\d-rede\playwright-tests-new\tests\demo-todo-app.spec.js:78:49
]]>
      </failure>
    </testcase>
    <testcase name="Mark all as completed should allow me to clear the complete state of all items" classname="demo-todo-app.spec.js › Mark all as completed › should allow me to clear the complete state of all items" time="1.235">
    </testcase>
  </testsuite>
</testsuites>

Importing results to TestRail

After the tests have been executed and the JUnit report files are generated, you can easily import your test results (and test cases!) to TestRail. This will bring visibility to your automated test runs and will enable you to look at the big picture of how you’re actually testing your app all within TestRail.

Installing the TestRail CLI

Given you already have Python installed on your machine, installing the TestRail CLI is as simple as executing the following command on your command line.

$ pip install trcli

Configuring TestRail

Secondly, you need to configure your TestRail instance according to the instructions below.

1. Enable the TestRail API by going to Admin > Site Settings, click on the API tab, and checking the Enable API option.
2. Create a Custom Field in order to map your automated test cases code to the actual TestRail cases. You can do so by going to Admin > Customizations and clicking Add Field. After you’ve reached the field creation screen, there are two requirements for this custom field:
   • The System Name must be automation_id
   • The Type must be Text

Sending results to TestRail

After you’re done installing the TestRail CLI and finished configuring your TestRail instance, you can upload your test results by simply using a one-liner such as the one below.

$ trcli -y \
>  -h https://INSERT-INSTANCE-NAME.testrail.io \
>  --project "My Project" \
>  --username INSERT-EMAIL \
>  --password INSERT-PASSWORD \
>  parse_junit \
>  --title "Playwright Automated Test Run" \
>  -f "./test-results/junit-report.xml"

Note that the file name after the -f option should match the path to your report file in case you change its default location. All others options should be according to your TestRail instance and project. You can check other command line options by checking the TestRail CLI README.md file on the project repository, the TRCLI documentation article, or the embedded CLI help through the commands below.

$ trcli --help
$ trcli parse_junit --help

Visualizing the results on TestRail

Now, if you go to the Test Cases page in your TestRail project, you’ll notice that the TestRail CLI automatically created the test cases that were on your test results report. Notice that it added a unique Automation ID by combining the classname and name attributes on each test on the JUnit report. This Automation ID is used to map the tests on your automation code base to the test cases on TestRail. This means that each time you run the TestRail CLI, it first attempts to match an existing test case on TestRail and only creates a new one in case there is no test case with that Automation ID.

On the Test Runs & Results page, we can see that a test run with the name Playwright Automated Test Run was created. By opening it we can dig further into the details of each automated test result and perform a high level analysis of why any test is failing since the error message provided by the test automation framework is also registered on the test result, along with any attachments such as the screenshot from when the test failure happened, as you can see on the image below.

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.