How to get the best results with Prioritize with AI

Prioritize with AI analyses the information available in your TestRail environment - your tests content, execution history, defect data, labels, and the instructions you provide - to decide which tests matter most for your current release.

The quality of the results you get out is directly shaped by the quality of the information you put in. This guide explains what that means in practice: what to have in place before you run prioritization, and what habits will help you get consistently strong results over time.

Write a specific and meaningful prompt

Your Additional Instructions prompt is the single most important input to prioritization. It tells the AI what your release is about, what areas carry the most risk, and what you need to focus on right now. The more specific and context-rich your prompt, the more targeted and accurate your results will be.

The AI uses your prompt to understand:

• Which areas of the application matter for this release

• What risk signals to look for, such as recent changes, known defects, or unstable features

• Which types of tests should be prioritized or deprioritized

• Whether any tests should be excluded entirely from the results

Strong prompt examples:

• “We've made significant changes to the checkout and payment flows this sprint. Focus on tests covering price calculation, discount logic, and payment gateway integrations. Deprioritize mobile-only and negative-flow tests.”

• “This is a hotfix release targeting a known login regression. Prioritize authentication, session management, and MFA flows. Everything unrelated to login can be deprioritized.”

• “Black Friday regression. Focus on high-traffic flows: search, cart, checkout, and order confirmation. Prefer tests that have failed or had linked defects in the past 30 days.”

Make sure your Test Cases have meaningful content

The AI reads the text content of your test cases — including titles, steps, expected results, BDD scenarios, and other description fields — and semantically matches that information against your prompt and labels to understand what each test actually covers.

For best results, ensure your test cases include:

• Descriptive titles that clearly identify the feature or flow being tested

• Detailed steps or scenarios describing the actions being performed

• Clear expected results explaining what successful behaviour looks like

Only fields containing data on the selected tests are available in the field selector.

Use Labels consistently and meaningfully

Labels are one of the strongest signals available to the prioritization AI. They help the AI quickly identify which tests belong to specific features, components, or testing categories.

The AI uses labels in two ways:

• Explicitly, when you select labels in the prioritization modal to increase the importance of matching tests

• Semantically, when labels align with terms used in your Additional Instructions prompt

Build and maintain execution history

When historical data is enabled, the AI analyses execution history to identify patterns that may indicate higher release risk, such as frequent failures, instability, or linked defects.

The richer your execution history, the stronger the AI signal becomes.

To improve historical analysis:

• Run tests regularly across multiple release cycles

• Record execution results directly in TestRail

• Select a look-back period that matches your release cadence

Link defects to your Test Results

Tests that have historically uncovered real defects provide valuable prioritization signals. The AI uses linked defect information to identify areas that may represent higher release risk.

For best results:

• Link defects to failing test results whenever possible

• Keep defect links accurate and current

Over time, consistent defect linkage improves the AI’s ability to identify high-risk tests more accurately.

Set Priority on your Test Cases

Native test case priority in TestRail acts as an additional signal for the AI when evaluating relative importance.

For best results:

• Assign priorities that reflect actual business criticality

• Review and maintain priorities as the application evolves

If all tests remain at the same default priority level, this signal becomes less meaningful to the AI.

Choose the right scope for your Run

AI priorities are always relative to the tests included in the specific run being prioritized.

To improve prioritization quality:

• Include a meaningful and relevant set of tests

• Avoid adding large volumes of unrelated tests to the run

• Use the Selected tests option for targeted prioritization scenarios

A “Very High” AI Priority in one run does not necessarily mean the same test would receive the same priority in a different run. Priorities are relative to the specific set of tests included in the run.

Keep your Test Suite healthy

The AI prioritizes based on the data it receives. Outdated or poorly maintained test cases introduce noise that can reduce prioritization quality.

Recommended practices include:

• Archive or remove obsolete test cases

• Keep titles, steps, and expected results current

• Apply labels and priorities consistently

• Review stale or rarely executed tests regularly