Integrating with Postman

Postman is an API platform that helps teams design, test, and manage APIs more efficiently. It streamlines the entire API lifecycle and makes collaboration easier.

Newman is a command-line tool for running Postman Collections, enabling you to execute and test collections from the command line instead of the Postman app. Designed for extensibility, Newman can be integrated into continuous integration (CI) pipelines and build systems.

In this tutorial, we will explore how to integrate the test results generated by a Postman automated test run (using Newman CLI) within TestRail using the TestRail CLI. This will allow you to centralize your automated test results and take advantage of all the analytical and reporting capabilities TestRail provides.

Overview

This tutorial uses a sample project to guide you through setting up a Postman automated test project compatible with the TestRail CLI and uploading the generated test results.

After reading through this tutorial, you will be able to:

• Execute tests from a simple Postman project
• Install the TestRail CLI
• Configure your TestRail instance
• Run the CLI
• See your test cases and test results on TestRail

Prerequisites

To be able to run a Postman tests project you can install Newman through npm. As a pre-requisite to install Postman Newman using npm all you need to do is install Node.js.

You will also need Python to install and run the TestRail CLI that will be used to import the test results to TestRail.

Installing the sample project

Let’s start by fetching the sample project code and installing the required dependencies.

1. Clone or download the sample project
2. Open your command prompt on the project root folder and execute the command below

npm install -g newman

Exploring the sample project

To edit/manipulate/validate the project, you can do it in two moments, as explained below. We’ve kept the automated tests code simple so that the focus of this tutorial remains on how to import the execution results. The tests consist of simple interactions with a Login page.

• To edit the requests that will be executed on these tests, it will probably be easier to manage them directly into Postman, and then download the collection, guaranteeing that the file is saved on JSON format.

• Using your favourite IDE, you'll then configure all the test execution, to trigger your requests using Newman CLI and then upload results to TestRail through TRCLI

Executing the sample project

This sample project can be executed in two ways. On the code sample available, there's the downloadable project but there's also a GitHub Actions file, which allows you to add the project in a CI pipeline and be automatically executed each time that it's triggered. Let’s explore both options.

.

Executing manually

After downloading the code sample from the repository, you'll just need to execute the following script, and then verify the results, that would be save on a file named newman_junit.xml

newman run API_Test_Demo.postman_collection.json -r cli,junit --reporter-junit-export ./reports/newman_junit.xml"

As result, you'll have a similar result:

Importing results to TestRail

Installing the TestRail CLI

Given you already have Python installed on your machine, installing the TestRail CLI is as simple as executing the following command on your command line.

$ pip install trcli

Configuring TestRail

Secondly, you need to configure your TestRail instance according to the instructions below.

1. Enable the TestRail API by going to Admin > Site Settings, click on the API tab, and checking the Enable API option.
2. Create a Custom Field in order to map your automated test cases code to the actual TestRail cases. You can do so by going to Admin > Customizations and clicking Add Field. After you’ve reached the field creation screen, there are two requirements for this custom field:
   • The System Name must be automation_id
   • The Type must be Text

Sending results to TestRail

After installing the TestRail CLI and configuring your TestRail instance, only two final steps remain before uploading your results.

First, you need to create your TRCLI config file (.yml file format), that should look like below

host: https://<TESTRAIL_HOST>.testrail.io
project: PostmanNewmanCLI_Integration
username: user@testrail.com
password: password
title: Postman Newman CLI Test Run

• host: TestRail instance link
• project: where your run will be saved
• username and password: your credentials to the instance identified above
• title: the name of your test run, where results will be stored

Command line execution of the TRCLI can be organized and executed easily using a common shell script as shown in the sample below that was run on a Windows machine using Git Bash (should be saved as a .sh file format):

#!/bin/bash
clear
bold=$(tput bold)
echo "${bold}-------------------------------------------"
echo "${bold}--Postman Newman Automation TestRail Demo--"
echo "${bold}-------------------------------------------"
echo

echo "${bold}...verify TRCLI is installed"
echo

trcli
echo

echo "${bold}...install the Newman Postman project, resolve dependencies"
echo

npm install -g newman
echo

echo "${bold}...executing Newman API Postman tests"
echo
newman run API_Test_Demo.postman_collection.json -r cli,junit --reporter-junit-export ./reports/newman_junit.xml

echo

echo "${bold}...executing TestRail CLI, uploading results in new test run"
echo

trcli -y -c "trcli-config.yml" parse_junit -f "./reports/newman_junit.xml" --title "Automation Demo - Newman Postman API Tests"
echo

echo "${bold}...execution complete"
echo

Ensure that the file name after the -f option matches the correct path to your report file, especially if you’ve changed its default location. In this example, a variable was used to store the file path, but you can also specify it directly.

All other options should be configured to match your TestRail instance and project. You can check other command line options by checking out the TestRail CLI README.md file on the project repository, the TRCLI documentation article, or the embedded CLI help through the commands below.

$ trcli --help
$ trcli parse_junit --help

Visualizing the results on TestRail

Now, if you go to the Test Cases page in your TestRail project, you’ll see that the TestRail CLI automatically created the test cases based on your test results report. A unique. A unique Automation ID by combining the classname and name attributes on each test on the JUnit report.

This Automation ID is used to map the tests on your automation code base to the test cases on TestRail. This means that each time you run the TestRail CLI, it first attempts to match an existing test case on TestRail and only creates a new one in case there is no test case with that Automation ID.

On the Test Runs & Results page, we can see that a test run with the name Automation Demo - Newman Postman API Tests was created. By opening it we can dig further into the details of each automated test result and perform a high level analysis of why any test is failing since the error message provided by the test automation framework is also registered on the test result, as you can see on the image below.

What next?

Now that you have centralized your test results on TestRail, not only can you check the results of your automated test runs, along with the error messages for failed tests, but you can also aggregate both your manual and automated testing efforts on reports that show you the full test coverage surrounding your app and even track test automation progress. You can also report a bug directly from the automated test result to an issue tracker of your preference as you would do for your manual test results!

You can look into the TestRail’s Reports and Test Metrics video to learn about how you can leverage TestRail’s reporting capabilities.