Getting the most out of AI-Powered Automation in TestRail

AI-powered automation in TestRail helps you move from manual test cases to executable automation faster. It generates a structured first draft based on your test case content and selected settings, while explaining what it created and why.

The output is designed to be reviewed, tested, and refined. It accelerates your work, but you remain in control.

The quality of what you receive depends directly on the quality of what you provide. This guide explains how to write better test cases, prompt more effectively, and provide the right context so the generated automation aligns with your real project.

Understand what the AI actually uses

When generating automation, only text-based information from the test case is sent to the AI. This may include:

• Title
• Description
• Preconditions
• Steps
• Expected results
• BDD scenarios
• Other custom text fields

The AI uses whichever of these fields are present in the test case. Structured metadata, attachments, and non-text fields are not used.

Write test cases for automation, not just for humans

Regardless of the test case template you use, the more precise and structured the information in your title, preconditions, steps, and expected results, the better the generated automation will be.

AI does not interpret intent the way a human tester does. It translates written instructions into executable logic. The clearer the instructions, the stronger the result. Even something as simple as a more specific title helps the AI better understand the test objective.

Let’s look at a simple comparison.

🟡 Weak example

Title: user can update profile

Steps:

• Log in
• Update profile
• Save
• Verify changes

This leaves too much open to interpretation:

• Which field is being updated?
• What data should be entered?
• How is login performed?
• Which selectors are involved?
• What confirms success?
• What should the automation wait for?

🟢 Stronger example

Title: update phone number in profile settings

Notice how the title already clarifies the specific objective.

Preconditions:

• User account exists
• Base URL: https://staging.example.com
• User is logged out

Steps:

1. Navigate to /login
2. Enter email into field with ID email
3. Enter password into field with ID password
4. Click button with ID login-submit
5. Wait until URL contains /dashboard
6. Navigate to /profile
7. Clear field with ID phone-number
8. Enter value +44 7700 900123
9. Click button with CSS selector .save-profile
10. Wait for success toast with text Profile updated successfully

Expected results:

• Success toast is visible
• Toast text equals Profile updated successfully
• Field phone-number contains +44 7700 900123

This version provides:

• A clearly defined objective
• Explicit navigation
• Concrete element references
• Defined test data
• Verifiable assertions

Be explicit about validations

Automation only works when the system can clearly determine whether a test has passed or failed.

Unlike a human tester, an automated test cannot rely on judgement or visual interpretation. It must check specific, measurable conditions in the application to decide if the behaviour is correct.

In automation, these checks are called assertions. An assertion is simply a rule in the code that verifies something is true, such as:

• A message is visible
• A button is enabled
• A URL has changed
• A field contains a specific value

If the condition is not met, the test fails.

Validation details can appear in:

• Preconditions
• Steps
• Expected results
• Or in additional prompting instructions

What matters is that the expected outcome is clearly defined somewhere in the information sent to the AI.

Select the correct language and framework

Before generating automation, you select the language and framework, such as Java or Python, Selenium or Playwright.

If your test case uses a BDD-style template, the AI aligns automatically:

• Cucumber for Java
• Behave for Python 

For best results with BDD, keep one scenario per test case. Multiple scenarios can lead to bloated or less focused output.

Use direct prompting strategically

When generating automation in TestRail, you can provide additional instructions in the AI automation pop-up before clicking generate. This is called prompting.

A prompt is simply a structured set of instructions that tells the AI how to behave during code generation. It can include preferences, constraints, rules, and project-specific guidance.

Think of it as briefing a new team member before they start writing automation. Stronger prompts lead to stronger results.

The most effective prompts include two elements:

1. Context-building instructions
2. Boundary-defining instructions

Both are essential.

What are context-building prompts?

For non-technical readers, context-building prompts are instructions that explain how your project works. They help the AI understand:

• What framework you use
• How your automation is structured
• What naming conventions you follow
• How you prefer validations and waits to be written

They improve quality by reducing generic output and aligning the generated code with your existing framework. Examples of context-building instructions:

• Preferred locator strategy
• Page Object Model usage
• Custom assertion wrappers
• Naming conventions
• Wait strategies
• Architectural patterns

What are boundary-defining prompts?

Boundary-defining prompts tell the AI what it must not do.

AI systems try to fill in gaps. If something is unclear, they may invent structure, selectors, utilities, or test data.

Boundaries protect consistency.

They prevent:

• Architectural overreach
• Invented data
• Unapproved patterns
• Unnecessary refactoring

They keep the generated code stable and predictable.

Why both matter

Context-building prompts improve quality.
Boundary-defining prompts protect consistency.

Together, they give you control.

When you send a prompt that combines both, the AI is guided not only on what to use during code generation, but also on what to avoid.

This significantly reduces generic outputs and increases the likelihood that the generated code will:

• Align with your framework
• Respect your architectural decisions
• Use the correct patterns
• Meet your expectations

Example of a good prompt

Here is an example that combines both context and boundaries in a single structured instruction:

Generate automation using Playwright with Python.
Follow our existing Page Object Model structure as shown in the uploaded files.
Use data-testid selectors wherever available.
Use our custom assert_text_equals() wrapper for validations.
Use explicit waits from our wait utility and do not use fixed sleeps.
Do not introduce new helper classes or base classes.
Do not invent URLs, credentials, or test data.
If required information is missing, insert a clearly marked TODO instead of guessing.
Only implement the steps defined in this test case. Do not extend behaviour beyond what is described.

When you provide a prompt like this in the AI automation pop-up:

• The AI understands your preferred framework and patterns.
• It avoids inventing architecture.
• It inserts placeholders instead of making unsafe assumptions.
• The generated script is far more aligned with your real project.

This is what good prompting looks like.

What makes prompts more effective

The prompts that produce better results usually:

• Are specific rather than generic
• Reflect your actual project structure
• Combine guidance and constraints
• Avoid vague instructions like “write clean code”
• Clearly define how missing information should be handled

Prompting is not about adding more text.
It is about adding the right clarity.

When used well, prompting transforms AI automation from a generic generator into a framework-aligned assistant.

Use file uploads as a quality multiplier

You can upload supporting files before generating automation to provide additional context. Supported formats include common code files, configuration files, and documentation formats. Uploading real project files significantly improves the relevance and alignment of the generated automation.

Useful files to upload:

• Page Object classes
• Base test classes
• Utility methods
• Example test files
• Centralised locator repositories
• Test setup files
• Runner configuration
• Framework snippets 
• Custom assertion wrappers
• Logging utilities
• Wait helpers 
• etc.

Review, refine, and iterate

The AI response includes:

• An explanation of what was generated
• Inline comments mapping code to test steps
• Clearly marked placeholders where input is missing.

Always:

• Review the script
• Run it in your environment
• Validate selectors and waits
• Refine for maintainability

Automation generation is traceable and recorded in history and audit logs, but human validation remains essential.