Most teams who integrate their automated tests with TestRail trigger and execute their automated tests outside of TestRail (e.g. as part of a continuous integration system) and use TestRail’s API to submit the test results. However, some teams prefer to implement a system to trigger test runs directly from TestRail. This document explains how to do this using practical examples with the usage of UI scripts.
Please note that this document contains working examples, but understanding the scripts and further extending them requires basic knowledge of PHP as well as web development (HTML, CSS and JS), which this content alone can't replace.
Streamlining test automation with TestRail
The following diagram illustrates the basic workflow of integrating your automated tests from trigger to results reporting on TestRail:
Using UI scripts, you can add a new button to the TestRail UI to trigger automated test runs. When clicked, it sends an HTTP request to the executor, which in turn triggers the automated tests. The script, or the test automation framework itself, then reports back the test results via TestRail’s API.
If your test automation framework supports generating JUnit style reports, we recommend using the TestRail CLI. With one line of code in your shell script, you can easily parse the test results and send them to TestRail.
Adding UI scripts to TestRail
UI scripts allow you to customize your TestRail GUI with HTML, CSS and JS as a web developer would, meaning you can add almost any UI elements and trigger any action you regularly would when developing a web page.
To add a UI script, go to TestRail’s administration area under Administration > Customizations > Add UI Script and simply copy and paste the whole UI script to the Configuration text box. In the examples below, we have UI scripts samples you can use, but if you want to learn about how to develop your own custom scripts or further customize these samples, please check the UI scripts documentation page.
Example 1: Jenkins integration
In this example, we add a Jenkins integration to the right side panel of the Test Runs & Results page, so that we have visibility over our latest automated tests jobs results and can trigger them directly from TestRail. Note that this example only covers triggering and fetching builds from Jenkins; if you want your test results to be reported back to TestRail as test runs, you need to configure your Jenkins job to send the results back to TestRail somehow, i.e., using the TestRail CLI. You can see the Integrating with Jenkins documentation for more information and a practical example of how to configure the TestRail CLI to be used in Jenkins.
The source code for this example can be found on GitHub:
The GitHub repository with the Jenkins integration UI script (tested on TestRail v7.5.4)
To allow the TestRail UI to make direct requests to your Jenkins instance, you need to allow Cross-origin resource sharing (CORS) on the Jenkins end. This is usually considered a security risk, but with the right precautions you can leverage this mechanism to allow TestRail to access your Jenkins jobs data without an increased risk. To do so, you should allow CORS to your trusted TestRail domain by following the steps below.
- Go to Manage Jenkins > Plugin Manager and search for and install the CORS support for Jenkins plugin
- Go to Manage Jenkins > Configure System
At the bottom of the page, on the CORS Filter section, insert the required parameters
- Click Save to accept the changes
As mentioned, the UI script adds a Jenkins integration section to the right hand panel which can be configured to display your automated tests jobs (or any other jobs if you wish).
On the panel we can see a new section called Automated Test Triggers and right below it the jobs that were configured to be fetched. For each job, there's a trigger button that will start a new build and also links to the 3 latest builds using a color code that indicates the status of the build. In this case, red for failure, green for success, and blue means it's still running.
The jobs being displayed are configured at the beginning of the UI script, where the configuration variables are defined, such as in the example below. The
name of the jobs can be anything and the
path is the relative path you can see on your browser when you navigate to the job on Jenkins.
You can enter the Jenkins
key if you fully trust all the TestRail users in your organization, but be aware that this poses a security risk since the UI script (and credentials) will be available in plain text on the browser. We recommend you leave these variables as
null and let users configure their own credentials through the UI.
/* CONFIG */
const host = "JENKINS_URL";
const username = null; // Enter username and key only if you fully trust your organization members as this may pose a security risk
const key = null; // Generate an API key on your Jenkins user account
const requestInterval = 10000; // Time between requests to Jenkins
const project = "PROJECT NAME"; // Name of the project to load the triggers for (use value null to enable for all projects)
const jobs = [ // Sample jobs config
"name": "Run Desktop Tests",
"name": "Run Mobile Tests",
Example 2: Trigger server-side scripts
In this example, a UI Script is used to add a new button to TestRail’s user interface and a server-side script launches the automated tests. In this case, the tests are executed on the test run level, but this can be changed to support test plans or just individual tests.
The source code for this example can be found on GitHub. Please choose the version that matches your TestRail version:
The GitHub repository with the UI and server-side script for TestRail 4.x
The GitHub repository with the UI and server-side script for TestRail 3.x
Assuming you're a TestRail Server user, the trigger script
trigger.php needs to be placed into TestRail’s installation directory on the web server next to TestRail’s
index.php file. If you plan to use a different location you also need to change the URL in the UI script accordingly.
The next step is to configure the following settings for TestRail's API:
There’s a lot of boilerplate code in the script, e.g., to handle the script arguments and submit test results back to TestRail's API. The interesting bit is the following function:
function execute_test($run_id, $case_id, $test_id)
This function is expected to trigger automated tests for a given test run and test case and should return a valid TestRail status ID. In this example, this function simply returns a random status ID, but real implementations could trigger tests by calling external command line tools, an external API, or similar.
You would also be able to use the trigger script with a TestRail Cloud account. You would simply place the server-side trigger script on the server where you would like to execute or kick off the tests (usually on your private network). TestRail would not need to be able to connect to your server, as the connection/glue is made via the UI script in TestRail and via your web browser. You can also write the server-side script in any other programming language and our script is only an example.
To make UI button available on your TestRail instance, simply add a new UI script as described on the section above with the content of the
trigger.ui file. After adding the script you should see a new Start Tests button in the toolbar of every test run page.
Clicking this button will execute an HTTP request to the server-side PHP script as per the jQuery snippet below. Notice how you can access context variables from the page where the UI script is loaded, such as
url: "trigger.php?run_id=" + uiscripts.context.run.id,
App.Dialogs.error('An error occurred while trying to trigger the automated tests.');
Using approaches similar to the ones above, it's possible to implement scripts to integrate with different tools such as GitHub Actions, GitLab CI, Travis CI, etc. However, when architecting this type of integration using UI scripts, always consider security risks and make sure you are able to handle policies such as CORS on the external tool, otherwise you will need to use server-side scripts.
You can also contribute to the user-submitted UI scripts collection by creating a PR with your code. Please see the general guidelines on the TestRail customizations GitHub repository.