TestRail’s new Automate with AI (BETA) feature leverages Large Language Models (LLMs) to automatically generate test code directly from your manual test cases. By writing well-structured test steps and providing the correct context files, you can instantly translate your manual efforts into functional automated test artifacts.
Feature Capabilities & Scope
LLMs are trained on massive amounts of data to learn patterns, but they do not actually write or execute code independently; instead, they learn what valid, useful code tends to look like. Therefore, understanding the scope and limitations of the tool is vital:
- Supported Frameworks: Generate code in Java (using JUnit) or Python. Supported frameworks include Selenium and Playwright. You can also choose to generate BDD-style tests (like Cucumber or Behave).
- Project Structure: Generated Selenium projects automatically use the Page Object Model (POM) structure, and Java projects will include JavaDoc comments.
- Supported Use Cases: This feature is exclusively for standalone Web-based UI end-to-end automated test cases.
- Not Supported: Mobile, desktop, backend testing, or other programming languages are not currently supported.
Optimizing Your Test Case for AI
Because LLMs do not inherently know your application's URLs or routing, you must write your test cases using Pseudocode—precise, detailed wording that reads like plain text but acts like computer code. Assure the AI has zero outside context.
Writing Pseudocode Steps
- Use Exact Navigation: Instead of "Go to Settings", use: "Navigate to 'url.com', Click on My Account link, Click Settings Button".
- Use Precise Assertions: Computer code differentiates between "visible to the user" and "present on the page". Use phrases like "Wait up to X seconds for the {element} to be visible" instead of the vague "Should be there".
- Specify Text Matching Strictness: Distinguish between exact matches and partial matches. Use "The error text is an exact match for 'Incorrect password'" or "The error text contains the words 'incorrect password'".
Expected Results are Required
The "Expected Result" field cannot be empty. If this field lacks pass/fail criteria, the Automate with AI button will remain disabled. Any text placed in the Pre-Conditions field will be generated as commented-out code, so you should omit manual-only setup steps.
Generating the Code
- Navigate to your saved Test Case in TestRail. (The button will be disabled until the test is saved).
- Click the Automate with AI (BETA) button in the left panel.
- In the AI Tool Window, click Settings to choose your Programming Language and Framework.
- Click Add Context Files and upload your prepared Selectors, Test Data, and Coding Rules files.
- Click Generate and wait for the AI to prepare your automation code.
Reviewing and Refining
Once generated, the results window will display:
- Imports.java (1): The library and framework imports for your code.
- Code.java(2): The test script, broken down by class using comment strings.
- AI Chat Window(3): Use this to interactively iterate and refine your code.
If the code isn't perfect, use the chat box to request edits. A strong prompt includes the Goal, Context, Rules/Style, and Output Format. For complex tasks, ask the AI to "Think step by step". If something is unclear, you can ask "Why did you do X on line 123?".
Saving and Execution
Once you are satisfied with the code, you have two options:
- Click Download to export the project as a package to run in your local IDE.
- Click Add to Test Case to save the generated code natively within your TestRail test case.
Important Limitations & Security
Because this feature is in BETA, the AI treats each test case as a completely isolated project; it does not retain data or instructions outside of the test case they were given to. LLMs can occasionally generate code that looks technically correct but does not perform as expected. Never skip human review and oversight before implementing generated code.
This script includes code generated using LLM. Because this code is produced by an automated system, please note the following:
- No Official Support. We do not provide technical support, troubleshooting, or bug fixes for code generated by AI.
- Review Required. You must manually audit and test all output before use. Automated systems may produce errors, security flaws, or outdated syntax.
- Environmental Context. Generated steps may be incomplete or incorrect due to missing context. This includes unmentioned browser requirements, missing dependencies, or specific environment configurations.
- User Responsibility. You are solely responsible for the performance, security, and maintenance of any generated code you choose to implement.
Additional TestRail AI Resources
For more specific articles on AI functionality and security, please explore the User Guide in the TestRail Support Center: