Getting started with Automate Test Cases with AI

Sembi IQ-powered AI automation in TestRail helps shorten the time it takes to write automation code.

Instead of starting from a blank file, you get a solid first draft generated from your manual test case and selected settings. Repetitive setup and boilerplate are handled for you, while clear inline comments explain what the code is doing and why.

The generated automation is meant to be reviewed, tested, and refined by a human. It gives you a faster starting point, not a finished solution, so you stay in control of how the automation works in your project.

The goal is to help you move faster with confidence, without taking ownership or judgment out of your hands.

AI Hub overview

The AI Hub provides a central place to understand how AI features are being used across your TestRail instance.

From the AI Hub → Overview page, users with the relevant permissions can see:

• Test cases generated with AI
  Displays the all-time number of test cases that have been created using AI-powered test case generation.
• Test cases automated with AI
  Displays the all-time number of test cases where automation scripts were generated using AI.

These metrics give teams a quick, high-level view of how AI is being adopted across their instance and how it supports both manual and automated testing workflows.

Feature controls and permissions

Sembi IQ-powered AI Test Automation in TestRail is managed through multiple configuration levels. Based on their permissions, users can enable or disable the feature at the following levels:

• Instance level
  Instance Admins can enable or disable AI automation for the entire TestRail instance. When disabled, the feature is not available to any user roles or projects.
• User role level
  Instance Admins can control which user roles are allowed to access AI automation. This determines who can generate, download, and attach AI-generated automation scripts.
• Project level
  Project Admins can enable or disable AI automation for individual projects. This allows teams to limit AI usage to selected projects only.

Access points

You can start generating automation from multiple places:

• From an individual test case view
• From the test case table.

At this stage, automation can be generated for one test case at a time.

AI automation settings (user-level)

When opening the AI automation modal, users can configure in Settings:

Supported languages

• Java
• Python

Supported frameworks

• Selenium
• Playwright

Support for more languages and frameworks is coming soon!

BDD support

• If the test case uses a BDD-style template, TestRail will generate:
  • Cucumber for Java
  • Behave for Python
• This is automatically aligned with the selected language.

How the Sembi IQ uses test case context

For best results:

• Make sure test steps are up to date and written clearly enough for an AI to translate them into automation actions.
• Be explicit in preconditions, steps, expected results, and validations. What is clear to a human executing a test may be too ambiguous for automation.
• Use consistent terminology across steps and expected results to avoid ambiguity.

Writing test cases with clear actions and verifiable outcomes helps the AI generate more accurate automation and reduces the need for follow-up changes.

Providing additional context

You can improve results by uploading supporting files, such as:

• Existing automation code from your project
• A list of selectors or page object references
• Framework configuration examples
• And include any other relevant files that could provide additional context to the AI and improve the quality of its output.

You can also prompt the Sembi IQ AI directly with instructions, for example:

• Preferred locator strategies
• Naming conventions
• Project-specific patterns
• And include any additional instructions that may provide further context or clearer direction to the AI.

This helps the AI adapt to your real-world setup rather than generate generic code. 

Providing additional context with files

You can upload files to give the AI more context before generating automation. This is optional, but it can significantly improve the quality and relevance of the generated script.

Providing real project context helps the AI align with your existing structure, naming conventions, and implementation patterns.

Examples of useful files to upload

1. Existing automation code
Upload files from your current automation project, such as:

• Page Object classes
• Base test classes
• Utility or helper methods
• Example test files

This helps the Sembi IQ AI:

• Follow your project structure
• Reuse naming conventions
• Align with your architectural patterns

2. Selectors or element references
If your team maintains:

• A selectors file
• A UI map
• A centralised locator repository

Uploading these files helps the AI use the correct element identifiers instead of generating generic ones.

3. Framework configuration examples
You may upload:

• Example test setup files
• Framework configuration snippets
• Test runner configuration files

This provides additional clarity about how your automation is structured and executed.

4. Coding standards or conventions
If your team follows specific patterns, such as:

• Custom assertion wrappers
• Specific naming conventions
• Particular wait strategies

Including sample files helps the AI align with those standards. If no files are uploaded, the AI will generate automation based solely on the test case content and selected settings.

Understanding the generated output

Sembi IQ's AI response includes:

• A clear explanation of what was generated and why
• Helpful inline comments that explain the purpose of each part of the code
• A logical structure that maps back to the steps defined in the test case

The quality of the generated automation depends on the quality of the input provided. The AI relies on the test case details, additional context, and user instructions to produce meaningful results.

For best outcomes:

• Review the generated code before using it
• Test the automation in your environment to ensure it behaves as expected
• Refine or extend the code where necessary to match your application and framework setup

Understanding AI explanations and code annotations

Automation generation is not limited to producing code. The AI also explains what it did and highlights where your attention may be required.

In the chat response, the Sembi IQ:

• Describes the structure of the generated files
• Explains how test steps were translated into automation methods
• Clearly points out placeholders that require user input, such as base URLs, credentials, or element identifiers
• Highlights areas that may need adjustment before execution

For example, if login credentials or environment-specific values were not provided, the AI will insert placeholders and explicitly call them out in the chat. This helps you quickly identify what needs to be configured before running the script.

Inside the generated code, inline comments reinforce this guidance. Placeholders are clearly marked and annotated, making it easy to see which values must be replaced and which sections may require project-specific refinement.

This combination of explanation and annotation makes the output transparent and easier to review.

Quickly locating logic related to specific test steps

Sembi IQ's AI structures the generated code to reflect your test case steps and expected results.

Inline comments within the code:

• Indicate which part of the test case a method or assertion corresponds to
• Clearly label navigation steps, actions, and validations
• Separate logical sections of the automation flow

This makes it easy to:

• Locate the code that implements a particular step
• Adjust a specific assertion tied to an expected result
• Understand how the test case maps to executable automation

Instead of scanning through a large script to understand what it does, you can quickly identify relevant sections and make targeted updates.

AI guidance in the chat

In addition to generating automation code, the Sembi IQ AI chat provides helpful guidance alongside the script.

The chat response:

• Explains what the AI generated and how it maps to the test case steps
• Highlights key decisions made during generation, such as selectors, waits, or assertions
• Provides recommendations for next steps, for example:
  • Where the script may need adjustment for your application
  • What to review or configure before running the test
  • How to extend or improve the automation structure

This makes it easier to understand, review, and refine the generated automation, especially when adapting it to an existing project or framework setup.

The chat is intended to support iteration. You can continue prompting the AI to clarify, adjust, or improve parts of the script as you work through testing and refinement.

Using and storing the generated automation

Once automation is generated, you can:

• Copy and paste or download the script and use it in an existing automation project, or as a starting point for a new one
• Attach the generated automation to the test case to keep manual and automated testing linked
• Update the attached automation using AI if the test case context changes, such as updated steps or expected behaviour

This keeps manual test cases and their associated automation connected in one place.

Updating automation scripts with AI

Automation scripts generated and attached to a test case can be updated using AI as the test case evolves.

When a test case is modified, users can:

• Reopen the attached AI-generated automation script
• Use AI to update or regenerate the script based on the latest test case content
• Refine the automation to reflect new steps, validations, or changes in behaviour

This helps keep manual test cases and their associated automation in sync over time, without having to start from scratch.

As with initial generation, users should review and test updated automation scripts to ensure they behave as expected in their environment.

Filtering test cases by AI status

The test case table includes new filters:

• AI Generated
• AI Automated

This makes it easy to:

• Track automation coverage
• Identify which test cases already have AI-generated scripts
• Plan further automation work

Traceability

All automation script generation is fully traceable:

• Changes appear in test case version history
• Events are recorded in Audit Logs

This provides transparency, accountability, and confidence for teams operating in regulated environments.