Integrating with Robot Framework (using Robot parser)

Robot Framework is a generic open source test automation framework. It can be used for test automation and robotic process automation (RPA). In this tutorial, we will explore how to integrate the test results generated by a Robot Framework automated test run within TestRail using the TestRail CLI. This will allow you to centralize your automated test results and take advantage of all the analytical and reporting capabilities TestRail provides.

Overview

Throughout this tutorial, we will be using a sample project to guide you through the process of setting up a Robot Framework automated tests project that is compatible with the TestRail CLI and uploading the generated test results.

After reading through this tutorial, you will be able to:

• Execute tests from a Robot Framework project using Selenium
• Install the TestRail CLI
• Configure your TestRail instance
• Run the CLI
• See your test cases and test results on TestRail

Prerequisites

To be able to install and run a Robot Framework project that uses Selenium as a browser driver, all you need is Python and the Chromedriver to get started. Since the TestRail CLI that will be used to import the test results is also a tool developed using Python, this will also allow us to install and run it.

Installing the sample project

Let’s start by fetching the sample project code and installing the required dependencies.

1. Clone or download the sample project
2. Open your command prompt on the project root folder and execute the command below

$ pip install -r "requirements.txt"

Exploring the sample project

Use your favorite IDE to open the sample project and start digging into the test files. We’ve kept the automated tests code simple so that the focus of this tutorial remains on how to import the execution results.

In order to optimize your test steps display in TestRail, you should create top level keywords using only natural language to describe your test steps, like you can see in the sample below. Also, to enrich your test cases and test results, you can add properties to the documentation. Simply start a new line and use a pattern similar to - testrail_property_name: value and it will be parsed by the TestRail CLI. The supported properties are the same as for JUnit, which you can check in the JUnit to TestRail mapping documentation.

In case you will only know the value of the property you want to set during execution time (i.e.: attachment paths), you can dynamically add on to your documentation using a keyword such as the one in the example below.

Executing the sample project

On your command prompt, execute the command below to run the Robot Framework tests and save the reports in the reports folder.

$ robot -d reports "./tests"

If the robot command was correctly executed, you should be able to see your test results on the reports folder.

Robot framework will automatically generate three two reports files. The report.html, log.html and output.xml. By opening the file report.html you should see a nicely formatted Robot Framework HTML report such as the one below.

When it comes to integrating test results within TestRail, the file that actually matters is the output.xml. This file contains a structured and detailed model of what happened during testing, including the properties we added on the documentation, which will be reflected in TestRail. This file will be parsed by the TestRail CLI in order to create the test run and upload your test results to TestRail on the next step.

<?xml version="1.0" encoding="UTF-8"?>
<robot generator="Robot 6.1 (Python 3.10.5 on win32)" generated="20230912 10:52:39.727" rpa="false" schemaversion="4">
  <suite id="s1" name="Tests" source="tests">
    <suite id="s1-s1" name="Footer-Tests" source="tests\footer-tests.robot">
      <test id="s1-s1-t1" name="Verify Presence Of Social Links" line="6">
        (...)
        <status status="PASS" starttime="20230912 10:52:39.958" endtime="20230912 10:52:47.980"/>
      </test>
      <status status="PASS" starttime="20230912 10:52:39.743" endtime="20230912 10:52:47.980"/>
    </suite>
    <suite id="s1-s2" name="Homepage-Tests" source="tests\homepage-tests.robot">
      <test id="s1-s2-t1" name="Verify Presence Of Header Links" line="6">
        (...)
        <status status="PASS" starttime="20230912 10:52:47.994" endtime="20230912 10:52:52.521"/>
      </test>
      <test id="s1-s2-t2" name="Verify Presence Of Demo Link" line="15">
        <kw name="Open TestRail Homepage" library="keywords">
          <status status="PASS" starttime="20230912 10:52:52.521" endtime="20230912 10:52:54.445"/>
        </kw>
        <kw name="Verify Header Demo Link" library="keywords">
          <status status="FAIL" starttime="20230912 10:52:54.446" endtime="20230912 10:52:59.946"/>
        </kw>
        <kw name="Click Demo Link" library="keywords">
          <status status="NOT RUN" starttime="20230912 10:52:59.947" endtime="20230912 10:52:59.947"/>
        </kw>
        <kw name="Verify Demo Page URL" library="keywords">
          <status status="NOT RUN" starttime="20230912 10:52:59.947" endtime="20230912 10:52:59.947"/>
        </kw>
        <kw name="Close Test Browser" library="keywords" type="TEARDOWN">
          <status status="PASS" starttime="20230912 10:52:59.948" endtime="20230912 10:53:03.127"/>
        </kw>
        <doc>Verifies header contains link to request a demo - Intentionally failing
- testrail_case_field: refs:TR-1
- testrail_case_field: priority_id:2
- testrail_result_field: custom_environment:qa

- testrail_attachment: C:\Github\gurock\automation-frameworks-integration\samples\robotframework\robotframework-selenium\reports\failure-2.png</doc>
        <status status="FAIL" starttime="20230912 10:52:52.521" endtime="20230912 10:53:03.127">Element 'css=.breakdance-menu-list [href*='/invalid/']' not visible after 5 seconds.</status>
      </test>
    </suite>
  </suite>
</robot>

Importing results to TestRail

After the tests have been executed and the output.xml file is generated, you can easily import your test results (and test cases!) to TestRail by using the TestRail CLI. This will bring visibility to your automated test runs and will enable you to look at the big picture of how you’re actually testing your app all within TestRail.

Installing the TestRail CLI

Given you already have Python installed on your machine, installing the TestRail CLI is as simple as executing the following command on your command line.

$ pip install trcli

Configuring TestRail

Secondly, you need to configure your TestRail instance according to the instructions below.

1. Enable the TestRail API by going to Administration > Site Settings, click on the API tab, and checking the Enable API option.
2. Create a Custom Field in order to map your automated test cases code to the actual TestRail cases. You can do so by going to Administration > Customizations and clicking Add Field. After you’ve reached the field creation screen, there are two requirements for this custom field:
   • The System Name must be automation_id
   • The Type must be Text

Sending results to TestRail

After you’re done installing the TestRail CLI and finished configuring your TestRail instance, you can upload your test results by simply using a one-liner such as the one below.

$ trcli -y \
>  -h https://INSERT-INSTANCE-NAME.testrail.io \
>  --project "My Project" \
>  --username INSERT-EMAIL \
>  --password INSERT-PASSWORD \
>  parse_robot \
>  --title "RF Automated Tests" \
>  -f reports/output.xml

Note that the file name after the -f option should match the path to your report file in case you change its default location. All others options should be according to your TestRail instance and project. You can check other command line options by using the embedded CLI help through the commands below.

$ trcli --help
$ trcli parse_robot --help

Visualizing the results on TestRail

Now, if you go to the Test Cases page in your TestRail project, you’ll notice that the TestRail CLI automatically created the test cases that were on your test results report. Notice that it added a unique Automation ID by combining the name attributes in the suites hierarchy and the name attribute on each test on the report. This Automation ID is used to map the tests on your automation code base to the test cases on TestRail. This means that each time you run the TestRail CLI, it first attempts to match an existing test case on TestRail and only creates a new one in case there is no test case with that Automation ID.

On the Test Runs & Results page, we can see that a test run with the name RF Automated Tests was created. By opening it we can dig further into the details of each automated test result and perform a high level analysis of why any test is failing since the error message provided by the test automation framework is also registered on the test result, as you can see on the image below.