UI scripts for test automation

Most teams who integrate their automated tests with TestRail trigger and execute their automated tests outside of TestRail (e.g. as part of a continuous integration system) and use TestRail’s API to submit the test results. However, some teams prefer to implement a system to trigger test runs directly from TestRail. This document explains how to do this using practical examples with the usage of UI scripts. 

Streamlining test automation with TestRail

The following diagram illustrates the basic workflow of integrating your automated tests from trigger to results reporting on TestRail:

Using UI scripts, you can add a new button to the TestRail UI to trigger automated test runs. When clicked, it sends an HTTP request to the executor, which in turn triggers the automated tests. The script, or the test automation framework itself, then reports back the test results via TestRail’s API.

Adding UI scripts to TestRail

UI scripts allow you to customize your TestRail GUI with HTML, CSS and JS as a web developer would, meaning you can add almost any UI elements and trigger any action you regularly would when developing a web page.

To add a UI script, go to TestRail’s administration area under Administration > Customizations > Add UI Script and simply copy and paste the whole UI script to the Configuration text box. In the examples below, we have UI scripts samples you can use, but if you want to learn about how to develop your own custom scripts or further customize these samples, please check the UI scripts documentation page. 

Example 1: Jenkins integration

In this example, we add a Jenkins integration to the right side panel of the Test Runs & Results page, so that we have visibility over our latest automated tests jobs results and can trigger them directly from TestRail. Note that this example only covers triggering and fetching builds from Jenkins; if you want your test results to be reported back to TestRail as test runs, you need to configure your Jenkins job to send the results back to TestRail somehow, i.e., using the TestRail CLI. You can see the Integrating with Jenkins documentation for more information and a practical example of how to configure the TestRail CLI to be used in Jenkins.

The source code for this example can be found on GitHub:

Configuring Jenkins

To allow the TestRail UI to make direct requests to your Jenkins instance, you need to allow Cross-origin resource sharing (CORS) on the Jenkins end. This is usually considered a security risk, but with the right precautions you can leverage this mechanism to allow TestRail to access your Jenkins jobs data without an increased risk. To do so, you should allow CORS to your trusted TestRail domain by following the steps below.

1. Go to Manage Jenkins > Plugin Manager and search for and install the CORS support for Jenkins plugin
2. Go to Manage Jenkins > Configure System

3. At the bottom of the page, on the CORS Filter section, insert the required parameters

   
4. Click Save to accept the changes

UI script

As mentioned, the UI script adds a Jenkins integration section to the right hand panel which can be configured to display your automated tests jobs (or any other jobs if you wish). 

On the panel we can see a new section called Automated Test Triggers and right below it the jobs that were configured to be fetched. For each job, there's a trigger button that will start a new build and also links to the 3 latest builds using a color code that indicates the status of the build. In this case, red for failure, green for success, and blue means it's still running.

The jobs being displayed are configured at the beginning of the UI script, where the configuration variables are defined, such as in the example below. The name of the jobs can be anything and the path is the relative path you can see on your browser when you navigate to the job on Jenkins.

/* CONFIG */

const host = "JENKINS_URL";
const username = null;               // Enter username and key only if you fully trust your organization members as this may pose a security risk
const key = null;                    // Generate an API key on your Jenkins user account
const requestInterval = 10000;       // Time between requests to Jenkins
const project = "PROJECT NAME";      // Name of the project to load the triggers for (use value null to enable for all projects)
const jobs = [                       // Sample jobs config
  {
    "name": "Run Desktop Tests",
    "path": "job/Desktop%20Automated%20Tests"
  },
  {
    "name": "Run Mobile Tests",
    "path": "job/Mobile%20Automated%20Tests"
  }
];

Example 2: Trigger server-side scripts

In this example, a UI Script is used to add a new button to TestRail’s user interface and a server-side script launches the automated tests. In this case, the tests are executed on the test run level, but this can be changed to support test plans or just individual tests.

The source code for this example can be found on GitHub. Please choose the version that matches your TestRail version:

Trigger script

Assuming you're a TestRail Server user, the trigger script trigger.php needs to be placed into TestRail’s installation directory on the web server next to TestRail’s index.php file. If you plan to use a different location you also need to change the URL in the UI script accordingly.

The next step is to configure the following settings for TestRail's API:

define('TESTRAIL_API_ADDRESS', 'http:///testrail');
define('TESTRAIL_API_USER', 'user@example.com');
define('TESTRAIL_API_PASSWORD', '***');

There’s a lot of boilerplate code in the script, e.g., to handle the script arguments and submit test results back to TestRail's API. The interesting bit is the following function:

function execute_test($run_id, $case_id, $test_id)
{
    ..
}

This function is expected to trigger automated tests for a given test run and test case and should return a valid TestRail status ID. In this example, this function simply returns a random status ID, but real implementations could trigger tests by calling external command line tools, an external API, or similar.

UI script

To make UI button available on your TestRail instance, simply add a new UI script as described on the section above with the content of the trigger.ui file.  After adding the script you should see a new Start Tests button in the toolbar of every test run page.

Clicking this button will execute an HTTP request to the server-side PHP script as per the jQuery snippet below. Notice how you can access context variables from the page where the UI script is loaded, such as uiscripts.context.run.id.

$.ajax({
  url: "trigger.php?run_id=" + uiscripts.context.run.id,
  dataType: "json",
  type: "POST",
  success: function()
  {
    location.reload();
  },
  error: function()
  {
    App.Dialogs.error('An error occurred while trying to trigger the automated tests.');
  }
});

Final considerations

Using approaches similar to the ones above, it's possible to implement scripts to integrate with different tools such as GitHub Actions, GitLab CI, Travis CI, etc. However, when architecting this type of integration using UI scripts, always consider security risks and make sure you are able to handle policies such as CORS on the external tool, otherwise you will need to use server-side scripts.

You can also contribute to the user-submitted UI scripts collection by creating a PR with your code. Please see the general guidelines on the TestRail customizations GitHub repository.