Frequently Asked Questions
1. How can we build traceability and measure test coverage in TestRail?
TestRail provides traceability and coverage reports like Comparison for References, Coverage for References, and Summary for References to track the coverage.
If you want to know which references have test cases associated with them and the test cases with/without references, you should opt for a Coverage for References report(especially during the test planning phase of your project).
If your goal is to see some additional details like which references and their test cases have defects associated with them, then you should create a Summary for References report.
And if you’re looking for details like how many test cases (associated with/without references) are in Passed, Failed, and other statuses, you should create a Comparison for References report.
You can check this guide if you want to see a quick overview of the use cases provided by TestRail reports.
2. How do I track test coverage or other test activities for a specific release or sprint?
You can use a milestone in TestRail to collect all test runs/plans involved in your release or sprint. You can also copy/paste the issue IDs related to the requirements/defects you're working on in that sprint into TestRail.
3. How do I report on test automation results in TestRail?
Firstly, you can use TestRail APIs to report your automation testing results to TestRail. You can check out this guide for Results API which will enable you to send automation results to TestRail. You can also use TestRail CLI, a command line interface tool that allows you to effortlessly upload test automation results from any JUnit-style XML file to TestRail.
Secondly, once you send your automated test results to TestRail, you can report on them using any of the reports in TestRail. For example, using the Property Distribution report, you can filter cases by automation type if you want to report only with automated test cases.
4. Can I customize the type of chart displayed in one of the built-in reports?
It's not currently possible to customize the chart types for the built-in reports under Reports but we have some chart customization for Dashboard charts and Project Overview charts in our 7.8 Early Access release!
If you’re interested in checking that out you can request access to the Early Access release track here: https://www.testrail.com/change-release-track/
5. Can we create a report across multiple TestRail projects?
Yes, on the Dashboard page, you can do one of the following
a) Under the "All Projects" tab, use the chart selecting multiple projects;
b) Under the "Cross-Project Reports" tab, create one of the available reports - available for Enterprise license plans only.
6. How can I report on requirements or defects tracked in external tools using TestRail?
You can link issue IDs for requirements to test cases and defects to test results. As long as you have not closed a test run yet, you can edit the previous test result to add the defect ID if you choose to create the defect in your external tool outside of TestRail.
7. What's the best practice to generate reports for projects having 1000s of test cases?
TestRail's built-in reports currently limit how many test results/cases you can display.
One way to overcome this is to create multiple reports dedicated to different test case sections, test runs, or plans. Depending on your use case, you could also run a print report for each test run or plan under your milestone, which will provide you with the complete list of tests you ran in each run or plan with the latest status. However, you can't just run the Milestone print report itself because TestRail doesn't support Milestone print reports if you're using a test plan in your milestone.