1. TestRail Support Center
  2. User Guide
  3. Reporting and analysis
  4. Results reports

Comparison for References (Results) report

Updated

Overview

The Comparison for References (Results) report shows the status of the tests you have planned or run for each, grouped by the references you have linked to those test cases. You can include all the references in your report or add specific reference IDs to include them in the report. You can also select the specific sections of test cases you want to include in your report and apply filters such as test type, priority, etc. 

Creating a Comparison for References (Results) report

Prerequisites

Please make sure you have the following permissions enabled for your user role.

Comparison for references ss1.png

For more details, please check out the managing user roles and permissions guide

Steps to create a Comparison for References (Results) report

To generate the Comparison for References (Results) in TestRail, you need to put a few configuration settings in place.

Navigate to the project for which you would like to create a test report. Then, navigate to Reports.

Comparison for References ss2.png

On the right side, you will see a pane titled Create a Report. Click on Comparison for References under the Results section.

Comparison for References ss3.png

On Add Report page, you will find 3 main sections: 

  • Name and Description
  • Report Options
  • Access & Scheduling

Follow the below steps to configure each section

  1. Name and Description

    Enter the report's name and description in their respective fields.

  2. Report Options

    On the Add Report page, scroll down to Report Options. Here, you will see three tabs, REFERENCES, SECTIONS & TEST RUNS, and TEST CASES.

    REFERENCES 

    Under the REFERENCES tab, you have two options. The first is to include all references, thereby generating a report for all the references you have linked to your tests.

    ReportOptions_ReferencesTab.png

    The second option is to add the reference IDs of the specific reference(s) you would like to include in your report. You should enter only one reference ID per line.

    ReportOptions_SpecificReferences.png

     

    SECTIONS & TEST RUNS

    You can do the following under the SECTIONS & TEST RUNS tab:

    CoverageForReferences_SectionsAndTestRuns.png

    1. Choose to include all sections of test cases or specific sections only. This allows you to show the results for specific sections of the test case that you would like to display in your report. Hold down the ctrl/cmd key and click to select multiple sections or sub-sections.

      CoverageForReferences_SelectMultipleSections.png
    2. Choose the test runs for which you would like to view test results. This can be done in two ways. The first option is to apply one or more filters to select the test runs that match the applied filter(s). You can filter the test runs using parameters like Assigned To, Completed On, Created By, Milestone, etc. This can be helpful if you want to compare results for tests across multiple test runs that fit certain criteria, e.g. all the test runs in a given test plan, or across a milestone that you are using to track testing for a specific release.

      CoverageForReferences_SelectTestRuns.png

       

      Afterwards, click Add Test Runs; then, click Apply Filters.

       ApplyFilter.png

      The second option is to make individual selections of the test runs, one by one. Click the radio button next to Only the following test runs, then click Add Test Runs and check the boxes for each test run you would like to include in your report.

      CoverageForReferences_AddSpecificRuns.png

    3. Select the maximum number of test runs to include. The maximum number can be five, 10, 25, 50, or 100. 

    TEST CASES 

    You can do the following under the TEST CASES tab:

    1. Choose the column(s) to include in the table to display information about test cases. The default columns include ID and Title. Depending on the information you want to display, you can add additional columns such as Status, Type (of the test case), Priority, etc.

      CovereageForReferences_AddColumn.png
    2. If you would like to show the comparison between the test results for each selected test run in your report, you need to check the box provided for the option as shown in the Figure below. The same applies if you want to show the latest/combined test result for the selected test runs. Afterwards, you can set the maximum number of test cases to include in the report. The maximum number can be 100, 250, 500, 1000, 2500, or 5000.

      TestCasesTab_AdditionalDetails.png

     

  3. Access & Scheduling

    This section enables users to set access and scheduling options. For more details, please refer to the General configurations guide.

Finally, scroll down and click Add Report to generate the report.

Reading a Comparison for References (Results) report

At this juncture, it is important to note that generating your report may take time, depending on the amount of data included in the report. However, once the report is fully generated, you can view it by going to Reports and clicking on the appropriate report under the list. The first chart on the report page is the stacked bar chart showing results for tests linked to the selected references and test cases. The three bars to the far left show the UI Test Runs results for Android, iOS and Windows, respectively. Similarly, the next three bars show the Smoke Test results and are followed by additional three bars displaying the results for Regression Test Runs. To the far right of the chart is a bar that shows the latest/combined test results for the tests linked to the selected references and test cases.

Hover your mouse on each of the bars to see more details.

CoverageForReferences_Read_image1.png

Next, you will see a list of the test runs linked to the selected references included in your report. Scroll to the right to see the full stats.

CoverageForReferences_Read_Image23.png

Afterwards, you will see a table that provides the configuration, test plan, and milestone associated with the test run (if applicable), the total number of tests (linked to selected references) in the run, and a summary of the number of tests with the Passed, Blocked, Untested, Retest, and Failed status for each selected test run. If you created the test run outside of a test plan or have not linked the test run to a milestone, these fields will appear blank.

ComparisonForCasesResults_Read_image2.png

 

The last table in the report is highly crucial as it provides a list of all the references included in the report and their associated test cases. You will see information about each test case configured during the report setup, like Case ID, Title, and any other columns you added. Also, you will see the current status for any tests you have added against the test cases for each test run. If a cell in the table is blank, it means you have not added a test for that case in the specific test runs included in the report.

The figure below shows that the test cases linked to the reference TRM-1 include C204, C7452, and C8414. Also, it should be noted that a test case can be linked to more than one reference. For instance, the figure below shows that test case C7452 is linked to references TRM-1 and TRM-42.

CoverageForReferences_Read_image6.png

You can click on a specific test case to get comprehensive information about the case. 

You may have test cases (and corresponding tests) linked to multiple references. For example, you can see in the screenshot below an example of a test case (with ID C7452) linked to two references (TRM-42 and TRM-1). The test case will appear under each reference grouping when displayed in the report. Aside from that, you can also see more information about the test type, priority, and status, among others.

CoverageForReferences_Read_image7.png

Was this article helpful?
1 out of 1 found this helpful