Getting started with Prioritize tests with AI

QA teams often face the same challenge before every release: there are more tests available than time to execute them all. As applications grow, releases accelerate, and regression suites expand, it becomes increasingly difficult to identify which tests are most critical for the current release.

Prioritize with AI helps solve this problem by intelligently ranking tests in a test run based on likely risk, relevance, and importance. Instead of treating every test equally, the AI helps your team focus first on the tests most likely to uncover critical defects.

Using historical execution data, linked defects, labels, and your own release-specific instructions, the feature automatically assigns each test an:

• AI Priority - Very High, High, Medium, or Low
• AI Reason - A clear explanation describing why the test received its ranking

This allows QA teams to:

• Focus testing effort where it matters most
• Reduce release risk during time-constrained cycles
• Accelerate regression testing
• Improve confidence in release readiness
• Prioritize newly changed or high-risk areas faster
• Make smarter testing decisions using historical trends and AI insights

Prioritize with AI is especially valuable for:

• Large regression suites
• Fast-moving Agile teams
• Short release windows
• Risk-based testing strategies
• Teams with limited testing capacity
• High-frequency release environments

Before you begin

Prioritize with AI is enabled by default for TestRail Cloud instances. To use the feature, the following conditions must be met:

• Prioritize with AI remains enabled at the instance level (AI Hub → AI Settings)

• Prioritize with AI remains enabled for your project (Project Settings)

• Your user role or project access role includes permission to use Prioritize with AI

• The test run is currently open (not archived or closed)

Step 1: Open a Test Run

Navigate to the open test run you want to prioritize. In the test run toolbar, click Prioritize (BETA). Choose one of the available options:

• All tests - Prioritizes every test in the run. This option becomes unavailable if the number of tests exceeds the maximum allowed.
• Selected tests - Prioritizes only the tests you manually selected. This option becomes unavailable if your selection falls outside the allowed minimum or maximum range.

Step 2: Configure Prioritization

The prioritization modal allows you to control the information the AI uses when analysing your tests, helping the AI make more accurate and context-aware prioritization decisions for your release.

Use Historical Data

When Historical Data is enabled, the AI analyses historical testing activity related to the test cases included in the current test run.

This may include:

• Previous test results
• Historical failures
• Linked defects
• Execution patterns
• Frequently failing or unstable areas

You can choose how much historical execution data the AI should consider by selecting a look-back period of: 7, 30 and 60 days.

The AI uses this information to better identify:

• High-risk functionality
• Tests associated with recurring failures
• Areas with recent instability
• Tests linked to known defects
• Functionality that may require additional attention before release

Labels (Optional)

Use labels to tell the AI which areas, features, or types of tests are most important for the current release.

Examples include:

• Smoke

• Checkout

• Payments

• Security

• Critical Path

You can select up to 6 test case and test labels using the type-ahead search.

Additional Instructions (Required)

Provide instructions describing what the AI should focus on for the current release. Examples:

• Areas with recent code changes

• High-risk workflows

• Known instability

• Recently reported defects

• Business-critical functionality

Example prompts:

• “Focus on checkout, payment processing, and recently modified authentication workflows.”

• “Prioritize tests related to mobile responsiveness and known login issues.”

• “This release contains major API changes. Focus on integration and regression risk.”

Test Fields

After you begin entering instructions, a Test Fields selector appears. Choose up to 6 fields for the AI to analyse when evaluating each test.

Common examples include: Title, Steps, Expected Result, Descriptions, etc. Only fields available on the selected tests appear in the selector. The following case field types are supported:

• Text
• Steps
• Scenario
• String

Step 3: Run Prioritization

Once all required fields are complete, you can start prioritization and TestRail will begin analysing your tests using AI to generate prioritization results.

Only one prioritization process can run per test run at any given time. If another user has already started prioritization for the same run, you will see a notification and must wait until processing completes before starting a new prioritization.

Step 4: Review and work with the Results

Once prioritization is complete, open the test run to review the generated AI Priority results and AI Reasons applied to your selected tests. Each prioritized test receives one of the following AI Priority values:

• Very High

• High

• Medium

• Low

• Unprioritized (tests not included in the current prioritization run)

Tests are automatically grouped and sorted by priority level, with the highest priority displayed first, making it easier to focus on the areas considered highest risk or most important for the release.

The AI Reason column explains why a test received its assigned priority. If the AI Reason column is hidden, you can still view the explanation by hovering over the AI Priority value.

A summary banner appears at the top of the test run showing when prioritization was last run, who ran it, and under what conditions. This helps teams maintain visibility and transparency into how prioritization decisions were generated, making it easier to review testing strategy, understand AI context, and align on release risk across the team.

You can also manually override any AI-assigned priority by clicking the value in the AI Priority column and selecting a different priority level. The test immediately moves into the corresponding priority group. Hovering over the updated value displays the original AI-assigned priority for reference.

Managing Access to Prioritize with AI

Prioritize with AI is enabled by default for all TestRail Cloud instances. User roles with the relevant permissions can control feature availability at the instance, project, and user role level to match their organisation’s governance and testing workflows.

• To enable or disable the feature across the entire instance, navigate to: AI Hub → AI Settings

• To manage availability for a specific project, navigate to: Project Settings → Edit Project

• To control which users can access the feature, navigate to: Administration → Users & Roles → Roles

• You can also manage access using project access roles, which override a user’s global role within a specific project.

Audit Logs

Audit logging for Prioritize with AI is available for customers on the Enterprise license with auditing enabled on their TestRail instance.

When enabled, TestRail records key Prioritize with AI activities under: Site Settings → Auditing Logs

These audit logs help organisations maintain visibility, governance, and traceability over AI-related activity within TestRail.