An enterprise-grade, AI-driven Autonomous QA platform that automatically generates, executes, and analyzes test cases using LLM agents. Built with LangChain, OpenAI, and Playwright.
Author: Ashwin Kulkarni
Email: ashwin.ak21@gmail.com
License: MIT
The Autonomous QA Platform automates the entire QA process:
- Test Planning: Parse feature descriptions and generate structured test plans
- Test Generation: Convert test plans into executable Playwright tests
- Test Execution: Run tests and capture detailed results
- Root Cause Analysis: Analyze failures and provide actionable insights
- Python 3.11+
- OpenAI API key
- Node.js 18+ (for Playwright)
# Clone the repository
git clone <repo-url>
cd autonomous-qa-platform
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Setup (choose based on your OS)
# macOS/Linux:
make setup
# Windows PowerShell:
.\scripts\windows\setup-windows.ps1 -Task setup
# Windows Command Prompt:
scripts\windows\setup-windows.bat setupOr install manually:
# Install dependencies
pip install -r requirements/requirements.txt
# Install Playwright browsers
playwright install
# Configure environment
cp .env.example .env
# Edit .env and add your OpenAI API keyfrom orchestration.agent_workflow import AutonomousQAWorkflow
# Initialize workflow
workflow = AutonomousQAWorkflow()
# Run QA on a feature
feature_description = """
Implement a user login feature with:
- Email/password authentication
- Remember me functionality
- Password reset via email
- Account lockout after 5 failed attempts
"""
result = workflow.run_qa_workflow(feature_description)
# Access results
print(f"Test Plan: {result.test_plan}")
print(f"Success: {result.success}")
if result.rca_results:
print(f"RCA: {result.rca_results}")See examples/fastapi_server.py for running as a web service.
The platform consists of four main components:
- TestPlanAgent: Generates structured test plans from feature descriptions
- TestGenerationAgent: Creates Playwright test code from test plans
- RCAAgent: Performs root cause analysis on test failures
- PlaywrightRunner: Executes Playwright tests and captures results
- LogParser: Extracts and structures error information from test logs
- AutonomousQAWorkflow: Main workflow orchestrator
- QAResult: Structured output containing test plan, results, and RCA
- RAG (
rag/): Vector database and embeddings for context retrieval - Prompts (
prompts/): LLM prompt templates - Evaluation (
evaluation/): Performance metrics and evaluation tools - API (
api/): FastAPI server for HTTP endpoints
See docs/architecture.md for detailed architecture.
Create a .env file (copy from .env.example):
OPENAI_API_KEY=sk-...
PLAYWRIGHT_TIMEOUT=30000
LOG_LEVEL=INFOSee docs/configuration.md for all options.
autonomous-qa-platform/
├── agents/ # LLM agents
│ ├── test_plan_agent.py
│ ├── test_generation_agent.py
│ └── rca_agent.py
├── tools/ # Utilities and runners
│ ├── playwright_runner.py
│ └── log_parser.py
├── orchestration/ # Workflow orchestration
│ └── agent_workflow.py
├── api/ # API package
│ └── __init__.py
├── rag/ # Vector DB and embeddings
│ ├── embeddings.py
│ └── vector_store.py
├── prompts/ # LLM prompt templates
│ ├── test_plan_prompt.txt
│ └── rca_prompt.txt
├── evaluation/ # Evaluation metrics
│ └── agent_metrics.py
├── playwright-tests/ # Test files
│ └── tests/
│ └── generated/ # Auto-generated tests
├── config/ # Configuration templates
│ └── config.yaml
├── examples/ # Usage examples
│ ├── basic_workflow.py
│ └── fastapi_server.py
├── tests/ # Unit tests
├── docs/ # Documentation
│ ├── architecture.md
│ ├── configuration.md
│ ├── api.md
│ └── development.md
├── scripts/ # Setup and utility scripts
│ ├── Makefile # Development tasks
│ └── windows/ # Windows-specific scripts
│ ├── setup-windows.ps1
│ └── setup-windows.bat
├── docker/ # Docker configuration
│ ├── Dockerfile
│ ├── docker-compose.yml
│ └── .dockerignore
├── requirements/ # Python dependencies
│ ├── requirements.txt
│ └── requirements-dev.txt
├── .env.example # Environment template
├── README.md # This file
└── LICENSE
- Architecture - System design and data flow
- Configuration - Environment and settings
- API Reference - HTTP API documentation
- Development Guide - Contributing guidelines
- Examples - Code samples and usage patterns
- ✅ AI-Powered Test Planning: Intelligent test scenario generation
- ✅ Automatic Test Generation: TypeScript/JavaScript Playwright tests
- ✅ Test Execution: Concurrent test runs with detailed reporting
- ✅ Failure Analysis: Root cause analysis powered by LLMs
- ✅ JSON Structured Output: Machine-readable results
- ✅ Retry Logic: Automatic retries with exponential backoff
- ✅ Error Handling: Comprehensive error handling and logging
- ✅ Extensible Architecture: Easy to add custom agents and tools
- Python 3.11: Backend language
- LangChain: LLM framework and orchestration
- OpenAI GPT-4: Intelligence engine
- Playwright: Browser automation
- FastAPI: REST API framework
- ChromaDB: Vector database for RAG
- Pydantic: Data validation
- Average test plan generation: < 10 seconds
- Average test generation: < 30 seconds
- Parallel test execution: Depends on test count and complexity
- RCA analysis: < 5 seconds per failure
See docs/development.md for setup and contribution guidelines.
MIT License - see LICENSE file
For issues, questions, or contributions, please open a GitHub issue.