Overview
The Summary for Cases (Defects) report shows defects recorded and linked to test runs in TestRail broken down by test case. This report provides insight into the latest test status of specific test cases, the number of defects discovered during testing for each test case that was tested, and the overall quality of the corresponding system capabilities. The report includes a graphical representation of defects by test run, as well as a detailed breakdown of defects by test case over a series of test runs.
Creating a Summary for Cases (Defects) report
Prerequisites
Please make sure you have the following permissions enabled for your user role.
For more details, please check out the managing user roles and permissions guide
Steps to create a Summary for Cases (Defects) report
To generate the Summary for Cases (Defects) report in TestRail, you need to put a few configuration settings in place.
Navigate to the project for which you would like to create a test report. Then, navigate to the Report tab.
On the right side, you will see a pane titled Create a Report. Click on Summary for Cases under the Defects section.
On Add Report page, you will find 3 main sections:
- Name and Description
- Report Options
- Access & Scheduling
Follow the below steps to configure each section
- Name and Description
Enter the report's name and description in their respective fields.
- Report Options
On the Add Report page, scroll down to Report Options. Here, you will see two tabs, SECTIONS & TEST RUNS and TEST CASES.
SECTIONS & TEST RUNS
You can do the following under the sections and test runs tab:
- Choose to include all sections of test cases or specific sections only. This allows you to show the results for specific sections of your test case repository that you would like to display in your report. For instance, if you would like to focus on particular features like Login and Messaging, you can select those two sections alone. Hold down the ctrl/cmd key and click to select multiple sections or sub-sections.
- Choose the test runs for which you would like to view test results. This can be done in two ways. The first option is to apply one or more filters to select the test runs that match the applied filter(s). You can filter the test runs using parameters like Assigned To, Completed On, Created By, Created On, Is Completed, Milestone, and Test Plan. This can be helpful if you want to compare results for tests across multiple test runs that fit certain criteria, e.g. all the test runs in a given test plan, or across a milestone that you are using to track testing for a specific release.
- Afterwards, click Add Test Runs; then, click Apply Filters.
The second option is to make individual selections of the test runs, one by one.
Select the maximum number of test runs to include. The maximum number can be five, 10, 25, 50, or 100.
TEST CASES
You can do the following under the test cases tab:
- Filter the test cases to be displayed in the report by any case field—including custom case fields—except text fields, like Assigned to, Created By, Deletion Status, Estimate, Priority, etc. For instance, in the figure below, the Priority filter is applied and set to High.
- Choose the column(s) to include in the table to display information about test cases. The default columns include ID and Title. Depending on the information you want to display, you can add additional columns such as Type (of the test case), Priority, Created On, etc.
- If you would like to show the comparison between the test results for each selected test run in your report, you need to check the box provided for the option as shown in the Figure below. The same applies if you want to show the latest/combined test result for the selected test runs. Afterwards, you can set the maximum number of test cases to include in the report. The maximum number can be 100, 250, 500, 1000, 2500, or 5000.
- Filter the test cases to be displayed in the report by any case field—including custom case fields—except text fields, like Assigned to, Created By, Deletion Status, Estimate, Priority, etc. For instance, in the figure below, the Priority filter is applied and set to High.
- Choose to include all sections of test cases or specific sections only. This allows you to show the results for specific sections of your test case repository that you would like to display in your report. For instance, if you would like to focus on particular features like Login and Messaging, you can select those two sections alone. Hold down the ctrl/cmd key and click to select multiple sections or sub-sections.
- Access & Scheduling
This section enables users to set access and scheduling options. For more details, please refer to the General configurations guide.
Finally, scroll down and click Add Report to generate the report.
Reading a Summary for Cases (Defects) report
It may take some minutes to generate your report depending on the amount of data included. However, once the report is fully generated, the next thing is to view and interpret it.
To view the report, go to your Reports tab and click on the appropriate report under the list.
The first chart on the report page is a bar graph that illustrates the number of defects found during applicable test runs. The Combined (Summary) column on the far right shows the total number of linked defects in the test runs included in the report. An example of this graph is shown below.
Next, you will see a list of test runs that have defects and meet the criteria provided in the report options. This list includes high-level information about each test run including the corresponding test section and milestone. An example is shown below. The note at the bottom indicates the number of test runs that met the search criteria but did not have any defects.
Immediately following that is a list of the test cases and additional information about applicable test runs based on the report options chosen. If a case was tested in a given run, the status of that test run is indicated by color. For example, red indicates the test failed. The column includes links to the defects from that test run. A blank space in the table indicates that the test case was not executed as part of that run or it did not have any linked defects.
If you selected the comparison report option, columns for each test run show the execution status of that test indicated by color.
If you selected the summary report option, a summary column on the far right shows the most recent test status across all test runs and links to defects from that test run. Defects are hyperlinked to the corresponding entries in an integrated defect tracking system, if applicable.
