Parallel Pagination Performance Guide for Large Test Suite

This guide provides comprehensive information about using the --parallel-pagination option in TestRail CLI, especially when combined with --batch-size for optimal performance when working with large test suites.

Understanding the options

1) --parallel-pagination (Experimental)

Purpose: Accelerates the process of fetching test cases from TestRail by retrieving multiple pages concurrently.

How it works:

1. Fetches the first page to determine the total number of pages
2. Spawns multiple worker threads (default: 10) to fetch remaining pages in parallel
3. Intelligently batches requests (100 pages at a time) to avoid overwhelming the server
4. Combines all results for processing

When to use:

• Projects with 1,000+ test cases
• When case matching during result upload is slow
• When initial case fetching takes significant time

Configuration:

• CLI flag: --parallel-pagination
• Settings file: ENABLE_PARALLEL_PAGINATION = True in trcli/settings.py
• Worker threads: MAX_WORKERS_PARALLEL_PAGINATION = 10 in trcli/settings.py

Performance impact: Can reduce case fetching time by 5-10x for large test suites.

2) --batch-size or -b

Purpose: Controls how many test results are bundled together in each API request when uploading results to TestRail.

How it works:

1. Groups test results into batches (default: 50 results per batch)
2. Uploads each batch as a single API call using the add_results_for_cases endpoint
3. Uses multithreading (default: 10 workers) to upload multiple batches concurrently

When to use:

• Always! This is a standard option that affects all result uploads
• Increase for faster uploads (with sufficient bandwidth and server capacity)
• Decrease if experiencing timeout issues or server overload

Configuration:

• CLI flag: -b 100 or --batch-size 100
• Settings file: DEFAULT_BATCH_SIZE = 50 in trcli/settings.py
• Valid range: 2 to unlimited (recommended: 50-250)
• Upload workers: MAX_WORKERS_ADD_RESULTS = 10 in trcli/settings.py

Performance impact: Larger batches reduce total API calls but may increase individual call time.

These options complement each other:

• --parallel-pagination: Speeds up Stage 1 (fetching existing test cases)
• --batch-size: Optimizes Stage 3 (uploading results)

Using both together provides end-to-end performance optimization.

Performance Optimization

Scenario Analysis

Small Projects [500 test cases | 100 results]

# Default settings are optimal
trcli parse_junit -f results.xml \
  -h https://yourinstance.testrail.io \
  -u user@example.com -p password \
  --project "Small Project"

Performance:

• Fetching: 2-5 seconds
• Uploading: 1-3 seconds
• Recommendation: No optimization needed

Medium Projects [500 - 2000 test cases | 100 - 500 results]

# Enable parallel pagination for faster case fetching
trcli parse_junit -f results.xml \
  --parallel-pagination \
  -h https://yourinstance.testrail.io \
  -u user@example.com -p password \
  --project "Medium Project"

Performance:

• Fetching: 5-15s (sequential) → 2-4s (parallel) 3-4x faster
• Uploading: 5-10 seconds (default batch size is fine)
• Recommendation: Use --parallel-pagination

Large Projects [2000 - 10.000 test cases | 500 - 2000 results]

# Optimize both fetching and uploading
trcli parse_junit -f results.xml \
  --parallel-pagination \
  --batch-size 100 \
  -h https://yourinstance.testrail.io \
  -u user@example.com -p password \
  --project "Large Project"

Performance:

• Fetching: 30-60s (sequential) → 5-10s (parallel) 5-6x faster
• Uploading: 60-90s (batch=50) → 30-45s (batch=100) 2x faster
• Recommendation: Use both --parallel-pagination and increased --batch-size

Very Large Projects [>10.000 test cases | >2000 results]

# Maximum optimization with configuration tuning
trcli parse_junit -f results.xml \
  --parallel-pagination \
  --batch-size 200 \
  --timeout 60 \
  -h https://yourinstance.testrail.io \
  -u user@example.com -p password \
  --project "Enterprise Project"

Additional configuration in trcli/settings.py:

# Increase workers for parallel pagination
MAX_WORKERS_PARALLEL_PAGINATION = 15

# Increase workers for result uploads
MAX_WORKERS_ADD_RESULTS = 15

Performance:

• Fetching: 120-180s (sequential) → 15-30s (parallel) 6-8x faster
• Uploading: 180-240s (batch=50) → 45-60s (batch=200) 4x faster
• Recommendation: Use aggressive optimization + settings tuning

Troubleshooting

#1 Parallel Pagination Causes Timeout Errors

Symptoms:

Error: Request timeout when fetching pages
Failed to retrieve test cases: Connection timeout

Solutions:

1 - Reduce worker count in trcli/settings.py:

MAX_WORKERS_PARALLEL_PAGINATION = 5  # Reduced from 10

2 - Increase timeout:

trcli parse_junit -f results.xml \
  --parallel-pagination \
  --timeout 60 \
  ...

3 - Check network/server capacity:

• TestRail server may be rate-limiting
• Network bandwidth may be insufficient

• Try disabling parallel pagination temporarily

   

#2 Large Batch Sizes Cause Upload Failures

Symptoms:

Error: 413 Request Entity Too Large
Failed to add results: Payload too large

Solutions:

1 - Reduce batch size:

trcli parse_junit -f results.xml \
  --batch-size 50 \  # Reduced from 200
  ...

2 - Check TestRail server limits:

• TestRail may have max request size limits
• Contact your TestRail administrator

3 - Split large result files:

# Split results into smaller chunks
split -l 1000 large_results.xml results_chunk_

# Upload each chunk separately
for chunk in results_chunk_*; do
  trcli parse_junit -f "$chunk" --run-id 123 ...
done

#3 Memory Usage Too High

Symptoms:

Out of memory error
Python process consuming excessive RAM

Solutions:

1 - Reduce concurrent operations:

# In trcli/settings.py
MAX_WORKERS_PARALLEL_PAGINATION = 3
MAX_WORKERS_ADD_RESULTS = 3

2 - Reduce batch size:

trcli parse_junit -f results.xml \
  --batch-size 25 \
  ...

3 - Process results in chunks (for very large runs)

#4 Inconsistent Performance

Symptoms:

• Some runs are fast, others are slow
• Unpredictable upload times

Solutions:

1. Check TestRail server load:
   • Server may be handling multiple concurrent requests
   • Consider scheduling uploads during off-peak hours
2. Monitor network conditions:
   • VPN or proxy may introduce latency
   • Use --proxy settings if needed
3. Standardize configuration:
   • Use config files to ensure consistent settings
   • Document optimal settings for your environment

#5 Parallel Pagination Shows No Improvement

Symptoms:

• Upload times similar with or without --parallel-pagination

Likely causes:

1. Project has few test cases (1000)
   • Parallel pagination overhead isn't worth it
   • Solution: Don't use the flag for small projects
2. Network is the bottleneck
   • Slow internet connection limits parallel benefit
   • Solution: Improve network connection or reduce workers
3. TestRail server is slow
   • Server response time dominates over parallelization
   • Solution: Contact TestRail administrator about performance

Best Practices

1. Start with Baseline Measurements

Before optimizing, measure your current performance:

# Measure without optimizations
time trcli parse_junit -f results.xml -h ... --project "..." -y

# Measure with parallel pagination
time trcli parse_junit -f results.xml --parallel-pagination -h ... --project "..." -y

# Measure with both optimizations
time trcli parse_junit -f results.xml --parallel-pagination --batch-size 150 -h ... --project "..." -y

2. Gradual Optimization

Don't jump to maximum settings immediately:

# Step 1: Enable parallel pagination
--parallel-pagination

# Step 2: If upload is still slow, increase batch size gradually
--batch-size 75   # Test
--batch-size 100  # Test
--batch-size 150  # Test

# Step 3: If needed, tune worker counts in settings.py

3. Use Configuration Files

Create environment-specific configs:

# config/dev.yml - Development (small dataset)
batch_size: 50

# config/staging.yml - Staging (medium dataset)
batch_size: 100

# config/prod.yml - Production (large dataset)
batch_size: 200

4. Monitor and Alert

In CI/CD pipelines, track upload times:

#!/bin/bash
START_TIME=$(date +%s)

trcli parse_junit -f results.xml --parallel-pagination --batch-size 150 ... -y

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

echo "Upload took ${DURATION} seconds"

# Alert if upload takes too long
if [ $DURATION -gt 300 ]; then
  echo "WARNING: Upload took longer than expected (${DURATION}s > 300s)"
  # Send alert to monitoring system
fi

5. Document Your Configuration

Create a README for your team:

## TestRail Upload Configuration

Our project has ~5000 test cases. Optimal settings:

- `--parallel-pagination`: Always use
- `--batch-size`: 150
- `--timeout`: 45

Typical upload time: 60-90 seconds

If uploads fail:
1. Check TestRail status page
2. Reduce batch-size to 100
3. Contact DevOps if issues persist

6. Consider Server Capacity

Coordinate with your TestRail administrator:

• What's the server's maximum concurrent request limit?

• Are there rate limits per user/API key?

• What's the recommended max batch size?

• Are there scheduled maintenance windows to avoid?

Additional Resources

• TestRail CLI Documentation: Getting Started with the TestRail CLI

• TestRail API Documentation: API Manual – TestRail Support Center

• GitHub Repository: gurock/trcli

• Performance Testing Results: See CONTRIBUTING.md for benchmark data

Feedback and Support

Please report issues or share performance results:

• GitHub Issues: gurock/trcli: 11 Issues

• TestRail Support: TestRail Support Center

When reporting issues, please include:

• Number of test cases in your project

• Number of results being uploaded

• Command used (with sensitive info redacted)

• Error messages or unexpected behavior

• Operating system and Python version

• TestRail server version (if known)