diff --git a/README.md b/README.md index 13e7d825f..425c0a929 100644 --- a/README.md +++ b/README.md @@ -216,6 +216,7 @@ Fine-tuning workspace for training Aria's language understanding using LoRA and - `datasets/chat/aria_movement/` - Movement command training data - `datasets/chat/aria_expanded/` - Extended Aria interactions - `datasets/chat/aria_simple/` - Basic Aria commands +- `datasets/chat/github_actions/` - GitHub Actions & CI/CD workflows training data **Quick Training:** diff --git a/datasets/chat/github_actions/README.md b/datasets/chat/github_actions/README.md new file mode 100644 index 000000000..85a573395 --- /dev/null +++ b/datasets/chat/github_actions/README.md @@ -0,0 +1,178 @@ +# GitHub Actions Training Dataset + +This dataset contains synthetic Q&A pairs for training AI models to understand GitHub Actions, CI/CD workflows, and DevOps best practices. + +## Overview + +- **Train samples**: 42 +- **Test samples**: 8 +- **Total samples**: 50 +- **Format**: JSONL (newline-delimited JSON) +- **Generated from**: Real workflow files in `.github/workflows/` + +## Content + +The dataset covers the following topics: + +### Workflow-Specific Content +Generated from 6 actual workflows in this repository: +1. **Auto Validation** - Automated validation workflows +2. **E2E + Integration Tests** - End-to-end and integration testing +3. **Aria E2E Tests** - Aria character system tests +4. **QAI CI Pipeline** - Main CI/CD pipeline with validate/train/deploy stages +5. **Quantum Automation** - Quantum computing workflow orchestration +6. **AzureML LoRA Train** - Azure ML model training workflows + +### Q&A Templates +For each workflow, the dataset includes: +- **Overview**: What the workflow does and its purpose +- **Triggers**: When and how the workflow runs +- **Jobs**: Detailed job descriptions and dependencies +- **Actions Used**: GitHub Actions and their purposes +- **Best Practices**: CI/CD best practices demonstrated +- **Modification Guide**: How to add or change steps +- **Troubleshooting**: Debugging workflow failures + +### General GitHub Actions Knowledge +Includes foundational Q&A pairs about: +- What GitHub Actions is and how it works +- Workflow file structure (YAML format) +- Common triggers (push, pull_request, schedule, etc.) +- Using secrets and environment variables +- Difference between `run` and `uses` +- Parallel vs sequential job execution +- Matrix strategies for testing multiple configurations +- Dependency caching for faster workflows + +## File Format + +Each line in the JSONL files is a JSON object with this structure: + +```json +{ + "messages": [ + {"role": "user", "content": "Question about GitHub Actions"}, + {"role": "assistant", "content": "Detailed answer with examples"} + ], + "source_workflow": "Workflow Name", + "source_path": ".github/workflows/example.yml", + "template": "overview", + "hash": "abc123" +} +``` + +## Generation + +This dataset was generated using `scripts/generate_github_actions_dataset.py`: + +```bash +python scripts/generate_github_actions_dataset.py --max-records 200 +``` + +To regenerate with different parameters: + +```bash +# Generate more samples +python scripts/generate_github_actions_dataset.py --max-records 500 + +# Use different seed for variation +python scripts/generate_github_actions_dataset.py --seed 123 + +# Different train/test split +python scripts/generate_github_actions_dataset.py --train-ratio 0.9 +``` + +## Training Usage + +This dataset is automatically discovered by the autonomous training orchestrator and can be used with any of the training scripts: + +### Quick Training +```bash +# Using the automated pipeline +python scripts/automated_training_pipeline.py --quick --models tinyllama + +# Using the aria quick train (adapted for this dataset) +python scripts/aria_quick_train.py +``` + +### Manual Training with LoRA +```bash +cd AI/microsoft_phi-silica-3.6_v1 +python scripts/train_lora.py \ + --dataset ../../datasets/chat/github_actions \ + --hf-model-id TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ + --epochs 25 \ + --max-train-samples 42 \ + --max-eval-samples 8 +``` + +### Autonomous Training +The dataset is automatically included in autonomous training cycles: + +```bash +# Start autonomous training (runs every 30 minutes) +nohup python scripts/autonomous_training_orchestrator.py > data_out/autonomous_training.log 2>&1 & + +# Trigger immediate cycle +pkill -USR1 -f autonomous_training +``` + +## Validation + +To validate the dataset: + +```bash +# Manual validation +python -c " +import json +with open('datasets/chat/github_actions/train.json') as f: + samples = [json.loads(line) for line in f] + print(f'✅ Loaded {len(samples)} valid training samples') +" +``` + +## Use Cases + +This dataset trains models to: +1. **Explain workflows**: Describe what GitHub Actions workflows do +2. **Answer CI/CD questions**: Provide guidance on workflow triggers, jobs, and steps +3. **Debug workflows**: Help troubleshoot common workflow failures +4. **Best practices**: Teach CI/CD best practices and patterns +5. **Modification guidance**: Guide developers on how to extend workflows +6. **General GitHub Actions knowledge**: Understand core concepts like matrix strategies, caching, secrets + +## Dataset Quality + +✅ **Strengths**: +- Based on real, working workflows from this repository +- Covers diverse CI/CD patterns (testing, training, deployment) +- Includes both specific and general knowledge +- Consistent format compatible with existing training infrastructure +- Multiple question templates for varied learning + +⚠️ **Limitations**: +- Limited to 50 samples (can be expanded by adjusting `--max-records`) +- Focused on workflows in this repository (Python, ML, quantum computing) +- May need supplementation with workflows from other domains (e.g., web frontend, mobile) + +## Future Enhancements + +Potential improvements: +1. Parse more workflow files from other repositories +2. Add examples of advanced GitHub Actions features (reusable workflows, custom actions) +3. Include workflow run history and failure patterns +4. Add examples of GitHub Actions API usage +5. Cover GitHub Actions security best practices +6. Include composite actions and action development + +## Related Datasets + +Other chat datasets in this repository: +- `app_repo/` - Repository code and documentation +- `aria_movement/` - Aria character movement commands +- `comprehensive/` - Mixed general chat topics +- `dolly/` - Instruction-following dataset + +## License + +This synthetic dataset is generated from the repository's own workflow files and is part of the Aria project. diff --git a/datasets/chat/github_actions/metadata.json b/datasets/chat/github_actions/metadata.json new file mode 100644 index 000000000..628811cda --- /dev/null +++ b/datasets/chat/github_actions/metadata.json @@ -0,0 +1,16 @@ +{ + "total_records": 50, + "train_records": 42, + "test_records": 8, + "generation_seed": 42, + "workflows_parsed": 6, + "workflow_names": [ + "Auto Validation", + "E2E + Integration Tests", + "Aria E2E Tests", + "QAI CI Pipeline", + "Quantum Automation", + "AzureML LoRA Train" + ], + "generation_date": "2026-01-22T15:34:36.150978" +} \ No newline at end of file diff --git a/datasets/chat/github_actions/test.json b/datasets/chat/github_actions/test.json new file mode 100644 index 000000000..645a7e4b2 --- /dev/null +++ b/datasets/chat/github_actions/test.json @@ -0,0 +1,8 @@ +{"messages": [{"role": "user", "content": "What is a matrix strategy in GitHub Actions?"}, {"role": "assistant", "content": "A matrix strategy in GitHub Actions allows you to run a job with multiple configurations automatically:\n\n```yaml\njobs:\n test:\n runs-on: ubuntu-latest\n strategy:\n matrix:\n python-version: ['3.9', '3.10', '3.11', '3.12']\n os: [ubuntu-latest, windows-latest, macos-latest]\n steps:\n - uses: actions/setup-python@v5\n with:\n python-version: ${{ matrix.python-version }}\n - run: python --version\n - run: pytest\n```\n\nThis creates 12 jobs (4 Python versions × 3 OS) automatically.\n\n**Key features**:\n- `${{ matrix.variable }}` accesses matrix values\n- `fail-fast: false` continues other jobs if one fails\n- `max-parallel` limits concurrent jobs\n- `include` adds specific combinations\n- `exclude` removes specific combinations\n\n**Use cases**:\n- Testing across multiple Python/Node versions\n- Cross-platform testing (Linux, Windows, macOS)\n- Testing with different dependency versions\n- Different configuration combinations"}], "source": "general_knowledge", "template": "matrix"} +{"messages": [{"role": "user", "content": "When does the 'Auto Validation' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "triggers", "hash": "1cd6da4993cb8a3f"} +{"messages": [{"role": "user", "content": "How would I modify the 'Auto Validation' workflow to add a new step?"}, {"role": "assistant", "content": "To add a new step to the 'Auto Validation' workflow:\n\n1. Choose which job to modify (available: dry-run)\n2. Locate the `steps` section within that job\n3. Add a new step with:\n - `name`: A descriptive name for the step\n - `run`: Command(s) to execute, or\n - `uses`: An action to use (e.g., actions/checkout@v4)\n4. Optionally add conditions with `if:` to control when the step runs\n\nExample step structure:\n```yaml\n- name: My new step\n run: |\n echo \"Running my command\"\n python my_script.py\n```\n\nOr using an action:\n```yaml\n- name: Upload results\n uses: actions/upload-artifact@v4\n with:\n name: my-artifact\n path: results/\n```\n\nRemember to test your changes in a feature branch before merging to main."}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "modification", "hash": "d328c4f201a13de3"} +{"messages": [{"role": "user", "content": "What GitHub Actions are used in the 'E2E + Integration Tests' workflow?"}, {"role": "assistant", "content": "The workflow uses 2 different actions:\n\n- **actions/checkout@v4**: Checks out the repository code\n- **actions/setup-python@v4**: Sets up a Python environment"}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "actions", "hash": "40ba1e664aeb4659"} +{"messages": [{"role": "user", "content": "What jobs are defined in the 'E2E + Integration Tests' workflow?"}, {"role": "assistant", "content": "The workflow defines 3 job(s):\n\n\n**integration**:\n - Runs on: ubuntu-latest\n - Contains 5 step(s)\n\n**e2e_playwright**:\n - Runs on: ubuntu-latest\n - Contains 5 step(s)\n\n**containerized_chrome**:\n - Runs on: ubuntu-latest\n - Contains 5 step(s)"}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "jobs", "hash": "9b45784bd38ac82f"} +{"messages": [{"role": "user", "content": "What is the structure of a GitHub Actions workflow file?"}, {"role": "assistant", "content": "A GitHub Actions workflow file is a YAML file with the following main sections:\n\n1. **name**: Optional workflow name displayed in the Actions tab\n2. **on**: Defines triggers (push, pull_request, schedule, etc.)\n3. **jobs**: Contains one or more jobs to run\n - Each job has:\n - **runs-on**: Runner type (ubuntu-latest, windows-latest, etc.)\n - **steps**: Sequential actions or commands to execute\n - **needs**: Optional dependencies on other jobs\n4. **env**: Optional environment variables\n5. **permissions**: Optional permissions settings\n\nExample:\n```yaml\nname: CI\non: [push, pull_request]\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - run: npm test\n```"}], "source": "general_knowledge", "template": "structure"} +{"messages": [{"role": "user", "content": "How do I cache dependencies in GitHub Actions?"}, {"role": "assistant", "content": "Caching dependencies speeds up workflow runs by reusing previously downloaded packages:\n\n**Using actions/cache**:\n```yaml\nsteps:\n - uses: actions/checkout@v4\n \n - uses: actions/cache@v4\n with:\n path: ~/.npm\n key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}\n restore-keys: |\n ${{ runner.os }}-node-\n \n - run: npm install\n```\n\n**Language-specific setup actions** with built-in caching:\n```yaml\n# Python with pip cache\n- uses: actions/setup-python@v5\n with:\n python-version: '3.11'\n cache: 'pip'\n\n# Node.js with npm cache\n- uses: actions/setup-node@v4\n with:\n node-version: '20'\n cache: 'npm'\n```\n\n**Benefits**:\n- Reduces workflow time by 30-70%\n- Decreases network load\n- Cache expires after 7 days of no access\n- Maximum cache size: 10 GB per repository\n\n**Best practices**:\n- Use hash of lock files in cache key for version tracking\n- Include OS in key if caching OS-specific files\n- Use restore-keys for partial cache hits"}], "source": "general_knowledge", "template": "caching"} +{"messages": [{"role": "user", "content": "What does the GitHub Actions workflow 'Quantum Automation' do?"}, {"role": "assistant", "content": "The workflow 'Quantum Automation' is a CI/CD pipeline that:\n- Contains 1 job(s): run-quantum\n- Uses 2 different GitHub Actions"}], "source_workflow": "Quantum Automation", "source_path": ".github/workflows/quantum-orchestration.yml", "template": "overview", "hash": "928bae82eae9d3dd"} diff --git a/datasets/chat/github_actions/train.json b/datasets/chat/github_actions/train.json new file mode 100644 index 000000000..c4683b437 --- /dev/null +++ b/datasets/chat/github_actions/train.json @@ -0,0 +1,42 @@ +{"messages": [{"role": "user", "content": "When does the 'Aria E2E Tests' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "Aria E2E Tests", "source_path": ".github/workflows/aria-tests.yml", "template": "triggers", "hash": "0b108737ae091f7e"} +{"messages": [{"role": "user", "content": "What jobs are defined in the 'Aria E2E Tests' workflow?"}, {"role": "assistant", "content": "The workflow defines 5 job(s):\n\n\n**unit-integration-tests**:\n - Runs on: ubuntu-latest\n - Contains 5 step(s)\n\n**playwright-e2e**:\n - Runs on: ubuntu-latest\n - Depends on: unit-integration-tests\n - Contains 9 step(s)\n\n**pyppeteer-e2e**:\n - Runs on: ubuntu-latest\n - Depends on: unit-integration-tests\n - Contains 10 step(s)\n\n**containerized-chrome-e2e**:\n - Runs on: ubuntu-latest\n - Depends on: unit-integration-tests\n - Contains 9 step(s)\n\n**test-summary**:\n - Runs on: ubuntu-latest\n - Depends on: unit-integration-tests, playwright-e2e, pyppeteer-e2e, containerized-chrome-e2e\n - Contains 2 step(s)\n - Conditional: always()"}], "source_workflow": "Aria E2E Tests", "source_path": ".github/workflows/aria-tests.yml", "template": "jobs", "hash": "1477073b6344d5e1"} +{"messages": [{"role": "user", "content": "What does the GitHub Actions workflow 'E2E + Integration Tests' do?"}, {"role": "assistant", "content": "The workflow 'E2E + Integration Tests' is a CI/CD pipeline that:\n- Contains 3 job(s): integration, e2e_playwright, containerized_chrome\n- Uses 2 different GitHub Actions\n\nThis workflow is primarily focused on running automated tests."}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "overview", "hash": "c8bb7a05e6e884c8"} +{"messages": [{"role": "user", "content": "What jobs are defined in the 'Auto Validation' workflow?"}, {"role": "assistant", "content": "The workflow defines 1 job(s):\n\n\n**dry-run**:\n - Runs on: ubuntu-latest\n - Contains 5 step(s)"}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "jobs", "hash": "510dd0820fe98f3b"} +{"messages": [{"role": "user", "content": "What is the difference between 'run' and 'uses' in a workflow step?"}, {"role": "assistant", "content": "In GitHub Actions workflow steps:\n\n**run**: Executes shell commands directly\n```yaml\n- name: Install dependencies\n run: |\n npm install\n npm run build\n```\n- Runs arbitrary commands in the runner's shell\n- Can use multiple lines with `|` or `>`\n- Good for simple commands or scripts\n- Has access to environment variables\n\n**uses**: Invokes a reusable action\n```yaml\n- name: Checkout code\n uses: actions/checkout@v4\n with:\n fetch-depth: 0\n```\n- References a pre-built action from GitHub Marketplace or your repository\n- Actions are versioned (using @v4, @main, or commit SHA)\n- Can accept parameters via `with:`\n- Promotes code reuse and maintainability\n- Examples: actions/checkout, actions/setup-python, azure/login\n\n**When to use**:\n- Use `run` for simple commands and scripts\n- Use `uses` for complex operations, third-party integrations, or reusable logic"}], "source": "general_knowledge", "template": "concepts"} +{"messages": [{"role": "user", "content": "What GitHub Actions are used in the 'AzureML LoRA Train' workflow?"}, {"role": "assistant", "content": "The workflow uses 3 different actions:\n\n- **actions/checkout@v4**: Checks out the repository code\n- **azure/cli@v2**: Runs Azure CLI commands\n- **azure/login@v2**: Authenticates with Azure"}], "source_workflow": "AzureML LoRA Train", "source_path": ".github/workflows/azureml-train.yml", "template": "actions", "hash": "e24c3f258c9bd065"} +{"messages": [{"role": "user", "content": "How would I modify the 'Aria E2E Tests' workflow to add a new step?"}, {"role": "assistant", "content": "To add a new step to the 'Aria E2E Tests' workflow:\n\n1. Choose which job to modify (available: unit-integration-tests, playwright-e2e, pyppeteer-e2e, containerized-chrome-e2e, test-summary)\n2. Locate the `steps` section within that job\n3. Add a new step with:\n - `name`: A descriptive name for the step\n - `run`: Command(s) to execute, or\n - `uses`: An action to use (e.g., actions/checkout@v4)\n4. Optionally add conditions with `if:` to control when the step runs\n\nExample step structure:\n```yaml\n- name: My new step\n run: |\n echo \"Running my command\"\n python my_script.py\n```\n\nOr using an action:\n```yaml\n- name: Upload results\n uses: actions/upload-artifact@v4\n with:\n name: my-artifact\n path: results/\n```\n\nRemember to test your changes in a feature branch before merging to main."}], "source_workflow": "Aria E2E Tests", "source_path": ".github/workflows/aria-tests.yml", "template": "modification", "hash": "f9d15bc576018586"} +{"messages": [{"role": "user", "content": "What GitHub Actions are used in the 'Auto Validation' workflow?"}, {"role": "assistant", "content": "The workflow uses 3 different actions:\n\n- **actions/checkout@v4**: Checks out the repository code\n- **actions/setup-python@v5**: Sets up a Python environment\n- **actions/upload-artifact@v4**: Uploads build artifacts for later use"}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "actions", "hash": "d47db55294222c9e"} +{"messages": [{"role": "user", "content": "What jobs are defined in the 'QAI CI Pipeline' workflow?"}, {"role": "assistant", "content": "The workflow defines 3 job(s):\n\n\n**validate**:\n - Runs on: ubuntu-latest\n - Contains 7 step(s)\n\n**train**:\n - Runs on: ubuntu-latest\n - Depends on: validate\n - Contains 5 step(s)\n - Conditional: github.event_name == 'schedule'\n\n**deploy**:\n - Runs on: ubuntu-latest\n - Depends on: train\n - Contains 5 step(s)\n - Conditional: github.event_name == 'schedule'"}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "jobs", "hash": "4489d346cee7d100"} +{"messages": [{"role": "user", "content": "How do I debug failures in the 'E2E + Integration Tests' workflow?"}, {"role": "assistant", "content": "To debug failures in the 'E2E + Integration Tests' workflow:\n\n1. **Check the Actions tab**: Go to GitHub Actions tab to see run history\n2. **Review logs**: Click on the failed run and expand failed steps to see detailed logs\n3. **Look for error messages**: Red text usually indicates the failure point\n4. **Verify triggers**: Ensure the workflow triggered as expected (check triggers: none)\n5. **Test locally**: Try running the commands from failed steps on your local machine\n6. **Check dependencies**: Ensure all required dependencies are installed\n7. **Review permissions**: Verify the workflow has necessary permissions\n8. **Use workflow_dispatch**: Add manual trigger for easier testing\n\nCommon issues:\n- Missing environment variables or secrets\n- Incorrect file paths\n- Dependency version conflicts\n- Insufficient permissions\n- Timeout issues (increase with `timeout-minutes`)\n\nFor persistent issues, you can enable debug logging by setting these secrets:\n- ACTIONS_STEP_DEBUG: true\n- ACTIONS_RUNNER_DEBUG: true"}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "troubleshooting", "hash": "43a40bc9f26393e1"} +{"messages": [{"role": "user", "content": "What does the GitHub Actions workflow 'QAI CI Pipeline' do?"}, {"role": "assistant", "content": "The workflow 'QAI CI Pipeline' is a CI/CD pipeline that:\n- Contains 3 job(s): validate, train, deploy\n- Uses 3 different GitHub Actions\n\nThis workflow is primarily focused on continuous integration checks."}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "overview", "hash": "2cba8f17f3a50cef"} +{"messages": [{"role": "user", "content": "When does the 'E2E + Integration Tests' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "triggers", "hash": "8ff662bf7a6b8058"} +{"messages": [{"role": "user", "content": "How do I debug failures in the 'Auto Validation' workflow?"}, {"role": "assistant", "content": "To debug failures in the 'Auto Validation' workflow:\n\n1. **Check the Actions tab**: Go to GitHub Actions tab to see run history\n2. **Review logs**: Click on the failed run and expand failed steps to see detailed logs\n3. **Look for error messages**: Red text usually indicates the failure point\n4. **Verify triggers**: Ensure the workflow triggered as expected (check triggers: none)\n5. **Test locally**: Try running the commands from failed steps on your local machine\n6. **Check dependencies**: Ensure all required dependencies are installed\n7. **Review permissions**: Verify the workflow has necessary permissions\n8. **Use workflow_dispatch**: Add manual trigger for easier testing\n\nCommon issues:\n- Missing environment variables or secrets\n- Incorrect file paths\n- Dependency version conflicts\n- Insufficient permissions\n- Timeout issues (increase with `timeout-minutes`)\n\nFor persistent issues, you can enable debug logging by setting these secrets:\n- ACTIONS_STEP_DEBUG: true\n- ACTIONS_RUNNER_DEBUG: true"}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "troubleshooting", "hash": "ebcc9778f0cf397d"} +{"messages": [{"role": "user", "content": "How do I use secrets in GitHub Actions?"}, {"role": "assistant", "content": "To use secrets in GitHub Actions:\n\n1. **Add secrets**: Go to repository Settings > Secrets and variables > Actions\n2. **Click 'New repository secret'** and add your secret (e.g., API keys, passwords)\n3. **Reference in workflow**: Use `${{ secrets.SECRET_NAME }}` syntax\n\nExample:\n```yaml\nsteps:\n - name: Deploy\n env:\n API_KEY: ${{ secrets.API_KEY }}\n DATABASE_URL: ${{ secrets.DATABASE_URL }}\n run: |\n echo \"Deploying with credentials...\"\n ./deploy.sh\n```\n\n**Best practices**:\n- Never commit secrets to code\n- Use environment-specific secrets for dev/staging/prod\n- Secrets are masked in logs\n- Use GitHub's OIDC for cloud provider authentication when possible\n- Consider using environment protection rules for sensitive deployments"}], "source": "general_knowledge", "template": "secrets"} +{"messages": [{"role": "user", "content": "When does the 'Quantum Automation' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "Quantum Automation", "source_path": ".github/workflows/quantum-orchestration.yml", "template": "triggers", "hash": "069b335a4a39fa8a"} +{"messages": [{"role": "user", "content": "How would I modify the 'Quantum Automation' workflow to add a new step?"}, {"role": "assistant", "content": "To add a new step to the 'Quantum Automation' workflow:\n\n1. Choose which job to modify (available: run-quantum)\n2. Locate the `steps` section within that job\n3. Add a new step with:\n - `name`: A descriptive name for the step\n - `run`: Command(s) to execute, or\n - `uses`: An action to use (e.g., actions/checkout@v4)\n4. Optionally add conditions with `if:` to control when the step runs\n\nExample step structure:\n```yaml\n- name: My new step\n run: |\n echo \"Running my command\"\n python my_script.py\n```\n\nOr using an action:\n```yaml\n- name: Upload results\n uses: actions/upload-artifact@v4\n with:\n name: my-artifact\n path: results/\n```\n\nRemember to test your changes in a feature branch before merging to main."}], "source_workflow": "Quantum Automation", "source_path": ".github/workflows/quantum-orchestration.yml", "template": "modification", "hash": "fa8d9a4f9e30e539"} +{"messages": [{"role": "user", "content": "What are the best practices demonstrated in the 'Auto Validation' workflow?"}, {"role": "assistant", "content": "The 'Auto Validation' workflow demonstrates several best practices:\n\n- Uses actions/checkout to ensure clean code access\n- Preserves build artifacts for debugging and deployment"}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "best_practices", "hash": "92f618023e1490b1"} +{"messages": [{"role": "user", "content": "What are the best practices demonstrated in the 'Aria E2E Tests' workflow?"}, {"role": "assistant", "content": "The 'Aria E2E Tests' workflow demonstrates several best practices:\n\n- Uses actions/checkout to ensure clean code access\n- Preserves build artifacts for debugging and deployment\n- Uses job dependencies to create a proper pipeline flow\n- Uses conditional job execution to optimize resource usage\n- Uses matrix strategy for testing across multiple configurations"}], "source_workflow": "Aria E2E Tests", "source_path": ".github/workflows/aria-tests.yml", "template": "best_practices", "hash": "2524595bc41cd145"} +{"messages": [{"role": "user", "content": "What are the best practices demonstrated in the 'QAI CI Pipeline' workflow?"}, {"role": "assistant", "content": "The 'QAI CI Pipeline' workflow demonstrates several best practices:\n\n- Uses actions/checkout to ensure clean code access\n- Preserves build artifacts for debugging and deployment\n- Uses job dependencies to create a proper pipeline flow\n- Uses conditional job execution to optimize resource usage\n- Uses matrix strategy for testing across multiple configurations"}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "best_practices", "hash": "ff4be11d7d91c9f7"} +{"messages": [{"role": "user", "content": "What is GitHub Actions?"}, {"role": "assistant", "content": "GitHub Actions is a CI/CD (Continuous Integration/Continuous Deployment) platform built into GitHub that allows you to automate your software development workflows. You can:\n\n- Run tests automatically on every push or pull request\n- Build and deploy applications\n- Automate code quality checks and linting\n- Schedule regular tasks\n- Respond to GitHub events (issues, releases, etc.)\n\nWorkflows are defined in YAML files stored in `.github/workflows/` directory."}], "source": "general_knowledge", "template": "concept"} +{"messages": [{"role": "user", "content": "How would I modify the 'AzureML LoRA Train' workflow to add a new step?"}, {"role": "assistant", "content": "To add a new step to the 'AzureML LoRA Train' workflow:\n\n1. Choose which job to modify (available: submit-job)\n2. Locate the `steps` section within that job\n3. Add a new step with:\n - `name`: A descriptive name for the step\n - `run`: Command(s) to execute, or\n - `uses`: An action to use (e.g., actions/checkout@v4)\n4. Optionally add conditions with `if:` to control when the step runs\n\nExample step structure:\n```yaml\n- name: My new step\n run: |\n echo \"Running my command\"\n python my_script.py\n```\n\nOr using an action:\n```yaml\n- name: Upload results\n uses: actions/upload-artifact@v4\n with:\n name: my-artifact\n path: results/\n```\n\nRemember to test your changes in a feature branch before merging to main."}], "source_workflow": "AzureML LoRA Train", "source_path": ".github/workflows/azureml-train.yml", "template": "modification", "hash": "2b2126b6480a8d03"} +{"messages": [{"role": "user", "content": "What are the best practices demonstrated in the 'E2E + Integration Tests' workflow?"}, {"role": "assistant", "content": "The 'E2E + Integration Tests' workflow demonstrates several best practices:\n\n- Uses actions/checkout to ensure clean code access"}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "best_practices", "hash": "72e024d63251cb1e"} +{"messages": [{"role": "user", "content": "How do I debug failures in the 'QAI CI Pipeline' workflow?"}, {"role": "assistant", "content": "To debug failures in the 'QAI CI Pipeline' workflow:\n\n1. **Check the Actions tab**: Go to GitHub Actions tab to see run history\n2. **Review logs**: Click on the failed run and expand failed steps to see detailed logs\n3. **Look for error messages**: Red text usually indicates the failure point\n4. **Verify triggers**: Ensure the workflow triggered as expected (check triggers: none)\n5. **Test locally**: Try running the commands from failed steps on your local machine\n6. **Check dependencies**: Ensure all required dependencies are installed\n7. **Review permissions**: Verify the workflow has necessary permissions\n8. **Use workflow_dispatch**: Add manual trigger for easier testing\n\nCommon issues:\n- Missing environment variables or secrets\n- Incorrect file paths\n- Dependency version conflicts\n- Insufficient permissions\n- Timeout issues (increase with `timeout-minutes`)\n\nFor persistent issues, you can enable debug logging by setting these secrets:\n- ACTIONS_STEP_DEBUG: true\n- ACTIONS_RUNNER_DEBUG: true"}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "troubleshooting", "hash": "6bc31608704d660b"} +{"messages": [{"role": "user", "content": "How do I debug failures in the 'AzureML LoRA Train' workflow?"}, {"role": "assistant", "content": "To debug failures in the 'AzureML LoRA Train' workflow:\n\n1. **Check the Actions tab**: Go to GitHub Actions tab to see run history\n2. **Review logs**: Click on the failed run and expand failed steps to see detailed logs\n3. **Look for error messages**: Red text usually indicates the failure point\n4. **Verify triggers**: Ensure the workflow triggered as expected (check triggers: none)\n5. **Test locally**: Try running the commands from failed steps on your local machine\n6. **Check dependencies**: Ensure all required dependencies are installed\n7. **Review permissions**: Verify the workflow has necessary permissions\n8. **Use workflow_dispatch**: Add manual trigger for easier testing\n\nCommon issues:\n- Missing environment variables or secrets\n- Incorrect file paths\n- Dependency version conflicts\n- Insufficient permissions\n- Timeout issues (increase with `timeout-minutes`)\n\nFor persistent issues, you can enable debug logging by setting these secrets:\n- ACTIONS_STEP_DEBUG: true\n- ACTIONS_RUNNER_DEBUG: true"}], "source_workflow": "AzureML LoRA Train", "source_path": ".github/workflows/azureml-train.yml", "template": "troubleshooting", "hash": "ea8bf43a39364d05"} +{"messages": [{"role": "user", "content": "What jobs are defined in the 'Quantum Automation' workflow?"}, {"role": "assistant", "content": "The workflow defines 1 job(s):\n\n\n**run-quantum**:\n - Runs on: windows-latest\n - Contains 3 step(s)"}], "source_workflow": "Quantum Automation", "source_path": ".github/workflows/quantum-orchestration.yml", "template": "jobs", "hash": "4170e602e624397e"} +{"messages": [{"role": "user", "content": "How would I modify the 'QAI CI Pipeline' workflow to add a new step?"}, {"role": "assistant", "content": "To add a new step to the 'QAI CI Pipeline' workflow:\n\n1. Choose which job to modify (available: validate, train, deploy)\n2. Locate the `steps` section within that job\n3. Add a new step with:\n - `name`: A descriptive name for the step\n - `run`: Command(s) to execute, or\n - `uses`: An action to use (e.g., actions/checkout@v4)\n4. Optionally add conditions with `if:` to control when the step runs\n\nExample step structure:\n```yaml\n- name: My new step\n run: |\n echo \"Running my command\"\n python my_script.py\n```\n\nOr using an action:\n```yaml\n- name: Upload results\n uses: actions/upload-artifact@v4\n with:\n name: my-artifact\n path: results/\n```\n\nRemember to test your changes in a feature branch before merging to main."}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "modification", "hash": "b1c701c8d4ebb30b"} +{"messages": [{"role": "user", "content": "What does the GitHub Actions workflow 'Aria E2E Tests' do?"}, {"role": "assistant", "content": "The workflow 'Aria E2E Tests' is a CI/CD pipeline that:\n- Contains 5 job(s): unit-integration-tests, playwright-e2e, pyppeteer-e2e, containerized-chrome-e2e, test-summary\n- Uses 4 different GitHub Actions\n\nThis workflow is primarily focused on running automated tests."}], "source_workflow": "Aria E2E Tests", "source_path": ".github/workflows/aria-tests.yml", "template": "overview", "hash": "444c93c7ec75f5ab"} +{"messages": [{"role": "user", "content": "How would I modify the 'E2E + Integration Tests' workflow to add a new step?"}, {"role": "assistant", "content": "To add a new step to the 'E2E + Integration Tests' workflow:\n\n1. Choose which job to modify (available: integration, e2e_playwright, containerized_chrome)\n2. Locate the `steps` section within that job\n3. Add a new step with:\n - `name`: A descriptive name for the step\n - `run`: Command(s) to execute, or\n - `uses`: An action to use (e.g., actions/checkout@v4)\n4. Optionally add conditions with `if:` to control when the step runs\n\nExample step structure:\n```yaml\n- name: My new step\n run: |\n echo \"Running my command\"\n python my_script.py\n```\n\nOr using an action:\n```yaml\n- name: Upload results\n uses: actions/upload-artifact@v4\n with:\n name: my-artifact\n path: results/\n```\n\nRemember to test your changes in a feature branch before merging to main."}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "modification", "hash": "49fa9de55378dcbd"} +{"messages": [{"role": "user", "content": "What GitHub Actions are used in the 'Aria E2E Tests' workflow?"}, {"role": "assistant", "content": "The workflow uses 4 different actions:\n\n- **actions/checkout@v4**: Checks out the repository code\n- **actions/download-artifact@v4**: Downloads previously uploaded artifacts\n- **actions/setup-python@v5**: Sets up a Python environment\n- **actions/upload-artifact@v4**: Uploads build artifacts for later use"}], "source_workflow": "Aria E2E Tests", "source_path": ".github/workflows/aria-tests.yml", "template": "actions", "hash": "696899fa4621c11f"} +{"messages": [{"role": "user", "content": "What are the best practices demonstrated in the 'Quantum Automation' workflow?"}, {"role": "assistant", "content": "The 'Quantum Automation' workflow demonstrates several best practices:\n\n- Uses actions/checkout to ensure clean code access"}], "source_workflow": "Quantum Automation", "source_path": ".github/workflows/quantum-orchestration.yml", "template": "best_practices", "hash": "8500996bf5337190"} +{"messages": [{"role": "user", "content": "What are the best practices demonstrated in the 'AzureML LoRA Train' workflow?"}, {"role": "assistant", "content": "The 'AzureML LoRA Train' workflow demonstrates several best practices:\n\n- Uses actions/checkout to ensure clean code access"}], "source_workflow": "AzureML LoRA Train", "source_path": ".github/workflows/azureml-train.yml", "template": "best_practices", "hash": "aa6e37f522471169"} +{"messages": [{"role": "user", "content": "How do I debug failures in the 'Quantum Automation' workflow?"}, {"role": "assistant", "content": "To debug failures in the 'Quantum Automation' workflow:\n\n1. **Check the Actions tab**: Go to GitHub Actions tab to see run history\n2. **Review logs**: Click on the failed run and expand failed steps to see detailed logs\n3. **Look for error messages**: Red text usually indicates the failure point\n4. **Verify triggers**: Ensure the workflow triggered as expected (check triggers: none)\n5. **Test locally**: Try running the commands from failed steps on your local machine\n6. **Check dependencies**: Ensure all required dependencies are installed\n7. **Review permissions**: Verify the workflow has necessary permissions\n8. **Use workflow_dispatch**: Add manual trigger for easier testing\n\nCommon issues:\n- Missing environment variables or secrets\n- Incorrect file paths\n- Dependency version conflicts\n- Insufficient permissions\n- Timeout issues (increase with `timeout-minutes`)\n\nFor persistent issues, you can enable debug logging by setting these secrets:\n- ACTIONS_STEP_DEBUG: true\n- ACTIONS_RUNNER_DEBUG: true"}], "source_workflow": "Quantum Automation", "source_path": ".github/workflows/quantum-orchestration.yml", "template": "troubleshooting", "hash": "a0a8fa698f2931f9"} +{"messages": [{"role": "user", "content": "What GitHub Actions are used in the 'QAI CI Pipeline' workflow?"}, {"role": "assistant", "content": "The workflow uses 3 different actions:\n\n- **actions/checkout@v4**: Checks out the repository code\n- **actions/setup-python@v5**: Sets up a Python environment\n- **actions/upload-artifact@v4**: Uploads build artifacts for later use"}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "actions", "hash": "5baacd58b1e6417e"} +{"messages": [{"role": "user", "content": "What does the GitHub Actions workflow 'AzureML LoRA Train' do?"}, {"role": "assistant", "content": "The workflow 'AzureML LoRA Train' is a CI/CD pipeline that:\n- Contains 1 job(s): submit-job\n- Uses 3 different GitHub Actions\n\nThis workflow is primarily focused on training machine learning models."}], "source_workflow": "AzureML LoRA Train", "source_path": ".github/workflows/azureml-train.yml", "template": "overview", "hash": "645956756614b55b"} +{"messages": [{"role": "user", "content": "What does the GitHub Actions workflow 'Auto Validation' do?"}, {"role": "assistant", "content": "The workflow 'Auto Validation' is a CI/CD pipeline that:\n- Contains 1 job(s): dry-run\n- Uses 3 different GitHub Actions"}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "overview", "hash": "defc1d033c53a858"} +{"messages": [{"role": "user", "content": "What jobs are defined in the 'AzureML LoRA Train' workflow?"}, {"role": "assistant", "content": "The workflow defines 1 job(s):\n\n\n**submit-job**:\n - Runs on: ubuntu-latest\n - Contains 4 step(s)"}], "source_workflow": "AzureML LoRA Train", "source_path": ".github/workflows/azureml-train.yml", "template": "jobs", "hash": "fac4b5bf3be54d3c"} +{"messages": [{"role": "user", "content": "When does the 'AzureML LoRA Train' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "AzureML LoRA Train", "source_path": ".github/workflows/azureml-train.yml", "template": "triggers", "hash": "f1489d015fef797c"} +{"messages": [{"role": "user", "content": "What are the most common GitHub Actions triggers?"}, {"role": "assistant", "content": "The most common GitHub Actions triggers are:\n\n1. **push**: Triggers on code pushes to specified branches\n ```yaml\n on:\n push:\n branches: [main, dev]\n ```\n\n2. **pull_request**: Triggers on PR events (opened, synchronized, etc.)\n ```yaml\n on:\n pull_request:\n branches: [main]\n ```\n\n3. **schedule**: Triggers on a cron schedule\n ```yaml\n on:\n schedule:\n - cron: '0 2 * * *' # Daily at 2 AM UTC\n ```\n\n4. **workflow_dispatch**: Allows manual workflow runs\n ```yaml\n on:\n workflow_dispatch:\n inputs:\n environment:\n description: 'Environment to deploy to'\n required: true\n ```\n\n5. **release**: Triggers on release events\n ```yaml\n on:\n release:\n types: [published]\n ```"}], "source": "general_knowledge", "template": "triggers"} +{"messages": [{"role": "user", "content": "How do I debug failures in the 'Aria E2E Tests' workflow?"}, {"role": "assistant", "content": "To debug failures in the 'Aria E2E Tests' workflow:\n\n1. **Check the Actions tab**: Go to GitHub Actions tab to see run history\n2. **Review logs**: Click on the failed run and expand failed steps to see detailed logs\n3. **Look for error messages**: Red text usually indicates the failure point\n4. **Verify triggers**: Ensure the workflow triggered as expected (check triggers: none)\n5. **Test locally**: Try running the commands from failed steps on your local machine\n6. **Check dependencies**: Ensure all required dependencies are installed\n7. **Review permissions**: Verify the workflow has necessary permissions\n8. **Use workflow_dispatch**: Add manual trigger for easier testing\n\nCommon issues:\n- Missing environment variables or secrets\n- Incorrect file paths\n- Dependency version conflicts\n- Insufficient permissions\n- Timeout issues (increase with `timeout-minutes`)\n\nFor persistent issues, you can enable debug logging by setting these secrets:\n- ACTIONS_STEP_DEBUG: true\n- ACTIONS_RUNNER_DEBUG: true"}], "source_workflow": "Aria E2E Tests", "source_path": ".github/workflows/aria-tests.yml", "template": "troubleshooting", "hash": "75fb6f3000d1b4ca"} +{"messages": [{"role": "user", "content": "What GitHub Actions are used in the 'Quantum Automation' workflow?"}, {"role": "assistant", "content": "The workflow uses 2 different actions:\n\n- **actions/checkout@v4**: Checks out the repository code\n- **azure/login@v2**: Authenticates with Azure"}], "source_workflow": "Quantum Automation", "source_path": ".github/workflows/quantum-orchestration.yml", "template": "actions", "hash": "a578b5cbbce18bf6"} +{"messages": [{"role": "user", "content": "How do I run jobs in parallel vs sequentially in GitHub Actions?"}, {"role": "assistant", "content": "GitHub Actions supports both parallel and sequential job execution:\n\n**Parallel (default)**: Jobs run simultaneously by default\n```yaml\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n \n lint:\n runs-on: ubuntu-latest\n steps:\n - run: npm run lint\n```\nBoth `test` and `lint` run at the same time.\n\n**Sequential**: Use `needs` to create dependencies\n```yaml\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - run: npm build\n \n test:\n needs: build # Waits for build to complete\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n \n deploy:\n needs: [build, test] # Waits for both\n runs-on: ubuntu-latest\n steps:\n - run: ./deploy.sh\n```\n\n**Benefits**:\n- Parallel: Faster total execution time\n- Sequential: Ensures correct order (e.g., build before test, test before deploy)\n- Mix both: Some jobs parallel, others sequential"}], "source": "general_knowledge", "template": "parallelism"} +{"messages": [{"role": "user", "content": "When does the 'QAI CI Pipeline' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "triggers", "hash": "021d60314cdee68f"} diff --git a/scripts/generate_github_actions_dataset.py b/scripts/generate_github_actions_dataset.py new file mode 100644 index 000000000..f4bfb2bd5 --- /dev/null +++ b/scripts/generate_github_actions_dataset.py @@ -0,0 +1,546 @@ +"""Generate synthetic GitHub Actions training dataset from workflow files. + +Creates a dataset suitable for chat fine-tuning about GitHub Actions, CI/CD, and workflows. +Output format: + datasets/chat/github_actions/train.json (newline-delimited JSON objects) + datasets/chat/github_actions/test.json (newline-delimited JSON objects) +Each line: {"messages": [ {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."} ]} + +Design principles: +- Generate Q&A pairs about workflow structure, triggers, jobs, steps +- Include CI/CD best practices and explanations +- Cover common GitHub Actions patterns and use cases +- Deterministic with --seed for reproducibility + +Usage: + python scripts/generate_github_actions_dataset.py --max-records 200 +""" +from __future__ import annotations +import argparse +import hashlib +import json +import random +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path +from typing import List, Dict, Any +import yaml + + +DEFAULT_OUTPUT = Path("datasets/chat/github_actions") + + +@dataclass +class WorkflowInfo: + name: str + path: str + content: str + parsed_yaml: Dict[str, Any] + triggers: List[str] + jobs: List[str] + uses_actions: List[str] + + +def parse_workflow(path: Path) -> WorkflowInfo | None: + """Parse a GitHub Actions workflow file""" + try: + content = path.read_text(encoding="utf-8") + parsed = yaml.safe_load(content) + + if not isinstance(parsed, dict): + return None + + # Extract key information + name = parsed.get("name", path.stem) + + # Extract triggers + triggers = [] + if "on" in parsed: + on_config = parsed["on"] + if isinstance(on_config, str): + triggers.append(on_config) + elif isinstance(on_config, list): + triggers.extend(on_config) + elif isinstance(on_config, dict): + triggers.extend(on_config.keys()) + + # Extract jobs + jobs = [] + if "jobs" in parsed and isinstance(parsed["jobs"], dict): + jobs = list(parsed["jobs"].keys()) + + # Extract used actions + uses_actions = [] + if "jobs" in parsed and isinstance(parsed["jobs"], dict): + for job_name, job_config in parsed["jobs"].items(): + if isinstance(job_config, dict) and "steps" in job_config: + for step in job_config.get("steps", []): + if isinstance(step, dict) and "uses" in step: + uses_actions.append(step["uses"]) + + return WorkflowInfo( + name=name, + path=str(path.relative_to(path.parents[2])), # relative to repo root + content=content, + parsed_yaml=parsed, + triggers=triggers, + jobs=jobs, + uses_actions=list(set(uses_actions)) + ) + except Exception as e: + print(f"Warning: Could not parse {path}: {e}") + return None + + +def generate_workflow_overview_qa(workflow: WorkflowInfo) -> tuple[str, str]: + """Generate Q&A about workflow overview""" + user = f"What does the GitHub Actions workflow '{workflow.name}' do?" + + assistant_parts = [f"The workflow '{workflow.name}' is a CI/CD pipeline that:"] + + if workflow.triggers: + trigger_desc = ", ".join(workflow.triggers) + assistant_parts.append(f"\n- Triggers on: {trigger_desc}") + + if workflow.jobs: + job_desc = ", ".join(workflow.jobs) + assistant_parts.append(f"- Contains {len(workflow.jobs)} job(s): {job_desc}") + + if workflow.uses_actions: + assistant_parts.append(f"- Uses {len(workflow.uses_actions)} different GitHub Actions") + + # Add purpose based on name/jobs + purpose_hints = { + "test": "running automated tests", + "ci": "continuous integration checks", + "deploy": "deploying code to production or staging", + "train": "training machine learning models", + "validate": "validating code quality and correctness", + "build": "building and compiling the application", + "lint": "checking code style and quality" + } + + for keyword, purpose in purpose_hints.items(): + if keyword in workflow.name.lower() or any(keyword in job.lower() for job in workflow.jobs): + assistant_parts.append(f"\nThis workflow is primarily focused on {purpose}.") + break + + return user, "\n".join(assistant_parts) + + +def generate_trigger_qa(workflow: WorkflowInfo) -> tuple[str, str]: + """Generate Q&A about workflow triggers""" + user = f"When does the '{workflow.name}' workflow run?" + + if not workflow.triggers: + return user, "This workflow has no explicit triggers defined." + + assistant_parts = [f"The '{workflow.name}' workflow is triggered by:"] + + trigger_explanations = { + "push": "code pushes to specified branches", + "pull_request": "pull request events (opened, synchronized, etc.)", + "workflow_dispatch": "manual workflow runs via GitHub UI or API", + "schedule": "scheduled cron jobs at specific times", + "release": "release events (published, created, etc.)", + "issues": "issue events (opened, labeled, etc.)", + "issue_comment": "comments on issues or pull requests" + } + + for trigger in workflow.triggers: + explanation = trigger_explanations.get(trigger, f"{trigger} events") + assistant_parts.append(f"\n- {trigger}: {explanation}") + + # Add schedule details if present + if "schedule" in workflow.triggers and "on" in workflow.parsed_yaml: + on_config = workflow.parsed_yaml["on"] + if isinstance(on_config, dict) and "schedule" in on_config: + schedule_list = on_config["schedule"] + if schedule_list and len(schedule_list) > 0: + cron = schedule_list[0].get("cron", "") + if cron: + assistant_parts.append(f"\nSchedule: {cron}") + + return user, "\n".join(assistant_parts) + + +def generate_jobs_qa(workflow: WorkflowInfo) -> tuple[str, str]: + """Generate Q&A about workflow jobs""" + user = f"What jobs are defined in the '{workflow.name}' workflow?" + + if not workflow.jobs: + return user, "This workflow has no jobs defined." + + assistant_parts = [f"The workflow defines {len(workflow.jobs)} job(s):\n"] + + jobs_config = workflow.parsed_yaml.get("jobs", {}) + for job_name in workflow.jobs: + job_config = jobs_config.get(job_name, {}) + assistant_parts.append(f"\n**{job_name}**:") + + # Runner + runs_on = job_config.get("runs-on", "unknown") + assistant_parts.append(f" - Runs on: {runs_on}") + + # Dependencies + if "needs" in job_config: + needs = job_config["needs"] + if isinstance(needs, str): + assistant_parts.append(f" - Depends on: {needs}") + elif isinstance(needs, list): + assistant_parts.append(f" - Depends on: {', '.join(needs)}") + + # Steps count + steps = job_config.get("steps", []) + assistant_parts.append(f" - Contains {len(steps)} step(s)") + + # Conditions + if "if" in job_config: + assistant_parts.append(f" - Conditional: {job_config['if']}") + + return user, "\n".join(assistant_parts) + + +def generate_actions_used_qa(workflow: WorkflowInfo) -> tuple[str, str]: + """Generate Q&A about actions used in workflow""" + user = f"What GitHub Actions are used in the '{workflow.name}' workflow?" + + if not workflow.uses_actions: + return user, "This workflow doesn't use any external GitHub Actions." + + assistant_parts = [f"The workflow uses {len(workflow.uses_actions)} different actions:\n"] + + action_descriptions = { + "actions/checkout": "Checks out the repository code", + "actions/setup-python": "Sets up a Python environment", + "actions/setup-node": "Sets up a Node.js environment", + "actions/upload-artifact": "Uploads build artifacts for later use", + "actions/download-artifact": "Downloads previously uploaded artifacts", + "actions/cache": "Caches dependencies to speed up workflows", + "azure/login": "Authenticates with Azure", + "azure/cli": "Runs Azure CLI commands" + } + + for action in sorted(workflow.uses_actions): + # Extract action name without version + action_name = action.split("@")[0] if "@" in action else action + description = action_descriptions.get(action_name, f"Performs {action_name} functionality") + assistant_parts.append(f"- **{action}**: {description}") + + return user, "\n".join(assistant_parts) + + +def generate_best_practices_qa(workflow: WorkflowInfo) -> tuple[str, str]: + """Generate Q&A about best practices in the workflow""" + user = f"What are the best practices demonstrated in the '{workflow.name}' workflow?" + + practices = [] + + # Check for specific patterns + if "actions/checkout" in " ".join(workflow.uses_actions): + practices.append("- Uses actions/checkout to ensure clean code access") + + if any("cache" in action.lower() for action in workflow.uses_actions): + practices.append("- Implements caching to improve workflow performance") + + if any("upload-artifact" in action.lower() for action in workflow.uses_actions): + practices.append("- Preserves build artifacts for debugging and deployment") + + jobs_config = workflow.parsed_yaml.get("jobs", {}) + for job_name, job_config in jobs_config.items(): + if "needs" in job_config: + practices.append("- Uses job dependencies to create a proper pipeline flow") + break + + # Check for conditional execution + for job_name, job_config in jobs_config.items(): + if "if" in job_config: + practices.append("- Uses conditional job execution to optimize resource usage") + break + + if "schedule" in workflow.triggers: + practices.append("- Implements scheduled runs for regular automated tasks") + + if "workflow_dispatch" in workflow.triggers: + practices.append("- Allows manual workflow triggering for flexibility") + + # Check for matrix strategy + for job_name, job_config in jobs_config.items(): + if "strategy" in job_config and "matrix" in job_config["strategy"]: + practices.append("- Uses matrix strategy for testing across multiple configurations") + break + + if not practices: + practices.append("- Follows standard GitHub Actions structure with jobs and steps") + + assistant = f"The '{workflow.name}' workflow demonstrates several best practices:\n\n" + "\n".join(practices) + return user, assistant + + +def generate_modification_qa(workflow: WorkflowInfo) -> tuple[str, str]: + """Generate Q&A about how to modify the workflow""" + user = f"How would I modify the '{workflow.name}' workflow to add a new step?" + + assistant = f"""To add a new step to the '{workflow.name}' workflow: + +1. Choose which job to modify (available: {', '.join(workflow.jobs) if workflow.jobs else 'none'}) +2. Locate the `steps` section within that job +3. Add a new step with: + - `name`: A descriptive name for the step + - `run`: Command(s) to execute, or + - `uses`: An action to use (e.g., actions/checkout@v4) +4. Optionally add conditions with `if:` to control when the step runs + +Example step structure: +```yaml +- name: My new step + run: | + echo "Running my command" + python my_script.py +``` + +Or using an action: +```yaml +- name: Upload results + uses: actions/upload-artifact@v4 + with: + name: my-artifact + path: results/ +``` + +Remember to test your changes in a feature branch before merging to main.""" + + return user, assistant + + +def generate_troubleshooting_qa(workflow: WorkflowInfo) -> tuple[str, str]: + """Generate Q&A about troubleshooting workflows""" + user = f"How do I debug failures in the '{workflow.name}' workflow?" + + assistant = f"""To debug failures in the '{workflow.name}' workflow: + +1. **Check the Actions tab**: Go to GitHub Actions tab to see run history +2. **Review logs**: Click on the failed run and expand failed steps to see detailed logs +3. **Look for error messages**: Red text usually indicates the failure point +4. **Verify triggers**: Ensure the workflow triggered as expected (check triggers: {', '.join(workflow.triggers) if workflow.triggers else 'none'}) +5. **Test locally**: Try running the commands from failed steps on your local machine +6. **Check dependencies**: Ensure all required dependencies are installed +7. **Review permissions**: Verify the workflow has necessary permissions +8. **Use workflow_dispatch**: Add manual trigger for easier testing + +Common issues: +- Missing environment variables or secrets +- Incorrect file paths +- Dependency version conflicts +- Insufficient permissions +- Timeout issues (increase with `timeout-minutes`) + +For persistent issues, you can enable debug logging by setting these secrets: +- ACTIONS_STEP_DEBUG: true +- ACTIONS_RUNNER_DEBUG: true""" + + return user, assistant + + +# Template functions for generating Q&A pairs +PROMPT_TEMPLATES = { + "overview": generate_workflow_overview_qa, + "triggers": generate_trigger_qa, + "jobs": generate_jobs_qa, + "actions": generate_actions_used_qa, + "best_practices": generate_best_practices_qa, + "modification": generate_modification_qa, + "troubleshooting": generate_troubleshooting_qa, +} + + +def generate_general_qa_pairs() -> List[Dict[str, Any]]: + """Generate general GitHub Actions Q&A pairs""" + general_qa = [ + { + "messages": [ + {"role": "user", "content": "What is GitHub Actions?"}, + {"role": "assistant", "content": "GitHub Actions is a CI/CD (Continuous Integration/Continuous Deployment) platform built into GitHub that allows you to automate your software development workflows. You can:\n\n- Run tests automatically on every push or pull request\n- Build and deploy applications\n- Automate code quality checks and linting\n- Schedule regular tasks\n- Respond to GitHub events (issues, releases, etc.)\n\nWorkflows are defined in YAML files stored in `.github/workflows/` directory."} + ], + "source": "general_knowledge", + "template": "concept", + }, + { + "messages": [ + {"role": "user", "content": "What is the structure of a GitHub Actions workflow file?"}, + {"role": "assistant", "content": "A GitHub Actions workflow file is a YAML file with the following main sections:\n\n1. **name**: Optional workflow name displayed in the Actions tab\n2. **on**: Defines triggers (push, pull_request, schedule, etc.)\n3. **jobs**: Contains one or more jobs to run\n - Each job has:\n - **runs-on**: Runner type (ubuntu-latest, windows-latest, etc.)\n - **steps**: Sequential actions or commands to execute\n - **needs**: Optional dependencies on other jobs\n4. **env**: Optional environment variables\n5. **permissions**: Optional permissions settings\n\nExample:\n```yaml\nname: CI\non: [push, pull_request]\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - run: npm test\n```"} + ], + "source": "general_knowledge", + "template": "structure", + }, + { + "messages": [ + {"role": "user", "content": "What are the most common GitHub Actions triggers?"}, + {"role": "assistant", "content": "The most common GitHub Actions triggers are:\n\n1. **push**: Triggers on code pushes to specified branches\n ```yaml\n on:\n push:\n branches: [main, dev]\n ```\n\n2. **pull_request**: Triggers on PR events (opened, synchronized, etc.)\n ```yaml\n on:\n pull_request:\n branches: [main]\n ```\n\n3. **schedule**: Triggers on a cron schedule\n ```yaml\n on:\n schedule:\n - cron: '0 2 * * *' # Daily at 2 AM UTC\n ```\n\n4. **workflow_dispatch**: Allows manual workflow runs\n ```yaml\n on:\n workflow_dispatch:\n inputs:\n environment:\n description: 'Environment to deploy to'\n required: true\n ```\n\n5. **release**: Triggers on release events\n ```yaml\n on:\n release:\n types: [published]\n ```"} + ], + "source": "general_knowledge", + "template": "triggers", + }, + { + "messages": [ + {"role": "user", "content": "How do I use secrets in GitHub Actions?"}, + {"role": "assistant", "content": "To use secrets in GitHub Actions:\n\n1. **Add secrets**: Go to repository Settings > Secrets and variables > Actions\n2. **Click 'New repository secret'** and add your secret (e.g., API keys, passwords)\n3. **Reference in workflow**: Use `${{ secrets.SECRET_NAME }}` syntax\n\nExample:\n```yaml\nsteps:\n - name: Deploy\n env:\n API_KEY: ${{ secrets.API_KEY }}\n DATABASE_URL: ${{ secrets.DATABASE_URL }}\n run: |\n echo \"Deploying with credentials...\"\n ./deploy.sh\n```\n\n**Best practices**:\n- Never commit secrets to code\n- Use environment-specific secrets for dev/staging/prod\n- Secrets are masked in logs\n- Use GitHub's OIDC for cloud provider authentication when possible\n- Consider using environment protection rules for sensitive deployments"} + ], + "source": "general_knowledge", + "template": "secrets", + }, + { + "messages": [ + {"role": "user", "content": "What is the difference between 'run' and 'uses' in a workflow step?"}, + {"role": "assistant", "content": "In GitHub Actions workflow steps:\n\n**run**: Executes shell commands directly\n```yaml\n- name: Install dependencies\n run: |\n npm install\n npm run build\n```\n- Runs arbitrary commands in the runner's shell\n- Can use multiple lines with `|` or `>`\n- Good for simple commands or scripts\n- Has access to environment variables\n\n**uses**: Invokes a reusable action\n```yaml\n- name: Checkout code\n uses: actions/checkout@v4\n with:\n fetch-depth: 0\n```\n- References a pre-built action from GitHub Marketplace or your repository\n- Actions are versioned (using @v4, @main, or commit SHA)\n- Can accept parameters via `with:`\n- Promotes code reuse and maintainability\n- Examples: actions/checkout, actions/setup-python, azure/login\n\n**When to use**:\n- Use `run` for simple commands and scripts\n- Use `uses` for complex operations, third-party integrations, or reusable logic"} + ], + "source": "general_knowledge", + "template": "concepts", + }, + { + "messages": [ + {"role": "user", "content": "How do I run jobs in parallel vs sequentially in GitHub Actions?"}, + {"role": "assistant", "content": "GitHub Actions supports both parallel and sequential job execution:\n\n**Parallel (default)**: Jobs run simultaneously by default\n```yaml\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n \n lint:\n runs-on: ubuntu-latest\n steps:\n - run: npm run lint\n```\nBoth `test` and `lint` run at the same time.\n\n**Sequential**: Use `needs` to create dependencies\n```yaml\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - run: npm build\n \n test:\n needs: build # Waits for build to complete\n runs-on: ubuntu-latest\n steps:\n - run: npm test\n \n deploy:\n needs: [build, test] # Waits for both\n runs-on: ubuntu-latest\n steps:\n - run: ./deploy.sh\n```\n\n**Benefits**:\n- Parallel: Faster total execution time\n- Sequential: Ensures correct order (e.g., build before test, test before deploy)\n- Mix both: Some jobs parallel, others sequential"} + ], + "source": "general_knowledge", + "template": "parallelism", + }, + { + "messages": [ + {"role": "user", "content": "What is a matrix strategy in GitHub Actions?"}, + {"role": "assistant", "content": "A matrix strategy in GitHub Actions allows you to run a job with multiple configurations automatically:\n\n```yaml\njobs:\n test:\n runs-on: ubuntu-latest\n strategy:\n matrix:\n python-version: ['3.9', '3.10', '3.11', '3.12']\n os: [ubuntu-latest, windows-latest, macos-latest]\n steps:\n - uses: actions/setup-python@v5\n with:\n python-version: ${{ matrix.python-version }}\n - run: python --version\n - run: pytest\n```\n\nThis creates 12 jobs (4 Python versions × 3 OS) automatically.\n\n**Key features**:\n- `${{ matrix.variable }}` accesses matrix values\n- `fail-fast: false` continues other jobs if one fails\n- `max-parallel` limits concurrent jobs\n- `include` adds specific combinations\n- `exclude` removes specific combinations\n\n**Use cases**:\n- Testing across multiple Python/Node versions\n- Cross-platform testing (Linux, Windows, macOS)\n- Testing with different dependency versions\n- Different configuration combinations"} + ], + "source": "general_knowledge", + "template": "matrix", + }, + { + "messages": [ + {"role": "user", "content": "How do I cache dependencies in GitHub Actions?"}, + {"role": "assistant", "content": "Caching dependencies speeds up workflow runs by reusing previously downloaded packages:\n\n**Using actions/cache**:\n```yaml\nsteps:\n - uses: actions/checkout@v4\n \n - uses: actions/cache@v4\n with:\n path: ~/.npm\n key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}\n restore-keys: |\n ${{ runner.os }}-node-\n \n - run: npm install\n```\n\n**Language-specific setup actions** with built-in caching:\n```yaml\n# Python with pip cache\n- uses: actions/setup-python@v5\n with:\n python-version: '3.11'\n cache: 'pip'\n\n# Node.js with npm cache\n- uses: actions/setup-node@v4\n with:\n node-version: '20'\n cache: 'npm'\n```\n\n**Benefits**:\n- Reduces workflow time by 30-70%\n- Decreases network load\n- Cache expires after 7 days of no access\n- Maximum cache size: 10 GB per repository\n\n**Best practices**:\n- Use hash of lock files in cache key for version tracking\n- Include OS in key if caching OS-specific files\n- Use restore-keys for partial cache hits"} + ], + "source": "general_knowledge", + "template": "caching", + }, + ] + + return general_qa + + +def build_records(workflows: List[WorkflowInfo], max_records: int, seed: int) -> List[Dict[str, Any]]: + """Build Q&A records from workflows""" + random.seed(seed) + records: List[Dict[str, Any]] = [] + + # Add general Q&A pairs + general_records = generate_general_qa_pairs() + records.extend(general_records) + + # Generate workflow-specific Q&A + for workflow in workflows: + # Randomize template order per workflow + template_keys = list(PROMPT_TEMPLATES.keys()) + random.shuffle(template_keys) + + for template_key in template_keys: + if len(records) >= max_records: + break + + try: + template_func = PROMPT_TEMPLATES[template_key] + user_prompt, assistant_answer = template_func(workflow) + + h = hashlib.sha256((workflow.name + template_key + assistant_answer[:100]).encode("utf-8")).hexdigest()[:16] + rec = { + "messages": [ + {"role": "user", "content": user_prompt}, + {"role": "assistant", "content": assistant_answer}, + ], + "source_workflow": workflow.name, + "source_path": workflow.path, + "template": template_key, + "hash": h, + } + records.append(rec) + except Exception as e: + print(f"Warning: Could not generate {template_key} for {workflow.name}: {e}") + + if len(records) >= max_records: + break + + return records + + +def write_jsonl(path: Path, records: List[Dict[str, Any]]): + """Write records to JSONL file""" + with path.open("w", encoding="utf-8") as f: + for rec in records: + f.write(json.dumps(rec, ensure_ascii=False) + "\n") + + +def main(): + ap = argparse.ArgumentParser(description="Generate GitHub Actions training dataset") + ap.add_argument("--workflow-dir", default=".github/workflows", help="Directory containing workflow files") + ap.add_argument("--output-dir", default=str(DEFAULT_OUTPUT), help="Output dataset directory") + ap.add_argument("--max-records", type=int, default=200, help="Maximum total records (train+test)") + ap.add_argument("--seed", type=int, default=42, help="Random seed") + ap.add_argument("--train-ratio", type=float, default=0.85, help="Train split ratio") + args = ap.parse_args() + + # Find repository root (go up from scripts dir) + repo_root = Path(__file__).resolve().parents[1] + workflow_dir = repo_root / args.workflow_dir + out_dir = repo_root / args.output_dir + out_dir.mkdir(parents=True, exist_ok=True) + + # Parse all workflow files + workflows: List[WorkflowInfo] = [] + if workflow_dir.exists(): + for pattern in ("*.yml", "*.yaml"): + for workflow_file in workflow_dir.glob(pattern): + workflow_info = parse_workflow(workflow_file) + if workflow_info: + workflows.append(workflow_info) + print(f"Parsed workflow: {workflow_info.name}") + + if not workflows: + print(f"Warning: No workflow files found in {workflow_dir}") + print("Generating dataset with only general GitHub Actions Q&A...") + + # Generate Q&A records + records = build_records(workflows, max_records=args.max_records, seed=args.seed) + + # Train/test split + random.seed(args.seed) + random.shuffle(records) + n_train = int(len(records) * args.train_ratio) + train_recs = records[:n_train] + test_recs = records[n_train:] or records[:max(1, len(records) // 10)] + + # Write datasets + write_jsonl(out_dir / "train.json", train_recs) + write_jsonl(out_dir / "test.json", test_recs) + + # Write metadata + meta = { + "total_records": len(records), + "train_records": len(train_recs), + "test_records": len(test_recs), + "generation_seed": args.seed, + "workflows_parsed": len(workflows), + "workflow_names": [w.name for w in workflows], + "generation_date": datetime.now().isoformat(), + } + with (out_dir / "metadata.json").open("w", encoding="utf-8") as f: + json.dump(meta, f, indent=2) + + print(f"\n{'='*60}") + print(f"Dataset generated successfully!") + print(f"{'='*60}") + print(json.dumps(meta, indent=2)) + print(f"\nOutput directory: {out_dir}") + print(f"Train dataset: {out_dir / 'train.json'}") + print(f"Test dataset: {out_dir / 'test.json'}") + + +if __name__ == "__main__": + main() diff --git a/tests/test_github_actions_dataset.py b/tests/test_github_actions_dataset.py new file mode 100644 index 000000000..804a8653e --- /dev/null +++ b/tests/test_github_actions_dataset.py @@ -0,0 +1,160 @@ +"""Test GitHub Actions dataset generation and validation.""" +import json +import pytest +from pathlib import Path +import subprocess +import sys + +REPO_ROOT = Path(__file__).resolve().parents[1] +DATASET_DIR = REPO_ROOT / "datasets" / "chat" / "github_actions" +TRAIN_FILE = DATASET_DIR / "train.json" +TEST_FILE = DATASET_DIR / "test.json" +METADATA_FILE = DATASET_DIR / "metadata.json" +GENERATOR_SCRIPT = REPO_ROOT / "scripts" / "generate_github_actions_dataset.py" + + +def test_dataset_files_exist(): + """Test that all required dataset files exist.""" + assert DATASET_DIR.exists(), f"Dataset directory not found: {DATASET_DIR}" + assert TRAIN_FILE.exists(), f"Train file not found: {TRAIN_FILE}" + assert TEST_FILE.exists(), f"Test file not found: {TEST_FILE}" + assert METADATA_FILE.exists(), f"Metadata file not found: {METADATA_FILE}" + + +def test_train_dataset_format(): + """Test that train dataset has correct format.""" + with open(TRAIN_FILE) as f: + samples = [json.loads(line) for line in f] + + assert len(samples) > 0, "Train dataset is empty" + + for i, sample in enumerate(samples): + assert "messages" in sample, f"Sample {i} missing 'messages' field" + assert isinstance(sample["messages"], list), f"Sample {i} 'messages' is not a list" + assert len(sample["messages"]) >= 2, f"Sample {i} needs at least 2 messages" + + # Check first message is from user + assert sample["messages"][0]["role"] == "user", f"Sample {i} first message should be from user" + assert "content" in sample["messages"][0], f"Sample {i} first message missing content" + + # Check second message is from assistant + assert sample["messages"][1]["role"] == "assistant", f"Sample {i} second message should be from assistant" + assert "content" in sample["messages"][1], f"Sample {i} second message missing content" + + +def test_test_dataset_format(): + """Test that test dataset has correct format.""" + with open(TEST_FILE) as f: + samples = [json.loads(line) for line in f] + + assert len(samples) > 0, "Test dataset is empty" + + for i, sample in enumerate(samples): + assert "messages" in sample, f"Sample {i} missing 'messages' field" + assert isinstance(sample["messages"], list), f"Sample {i} 'messages' is not a list" + assert len(sample["messages"]) >= 2, f"Sample {i} needs at least 2 messages" + + +def test_metadata_content(): + """Test that metadata contains expected fields.""" + with open(METADATA_FILE) as f: + metadata = json.load(f) + + assert "total_records" in metadata + assert "train_records" in metadata + assert "test_records" in metadata + assert "generation_seed" in metadata + assert "workflows_parsed" in metadata + + # Verify counts match actual files + with open(TRAIN_FILE) as f: + train_count = sum(1 for _ in f) + with open(TEST_FILE) as f: + test_count = sum(1 for _ in f) + + assert metadata["train_records"] == train_count, "Metadata train count mismatch" + assert metadata["test_records"] == test_count, "Metadata test count mismatch" + assert metadata["total_records"] == train_count + test_count, "Metadata total count mismatch" + + +def test_dataset_content_quality(): + """Test that dataset contains meaningful content.""" + with open(TRAIN_FILE) as f: + samples = [json.loads(line) for line in f] + + # Check that questions and answers are substantive + for i, sample in enumerate(samples[:10]): # Check first 10 samples + user_content = sample["messages"][0]["content"] + assistant_content = sample["messages"][1]["content"] + + # Questions should be at least 10 characters + assert len(user_content) >= 10, f"Sample {i} question too short" + + # Answers should be at least 20 characters + assert len(assistant_content) >= 20, f"Sample {i} answer too short" + + # Check for GitHub Actions related keywords in the dataset + combined = (user_content + " " + assistant_content).lower() + github_keywords = ["workflow", "github", "action", "job", "step", "trigger", "ci", "cd"] + has_keyword = any(keyword in combined for keyword in github_keywords) + assert has_keyword, f"Sample {i} doesn't appear to be about GitHub Actions" + + +def test_dataset_diversity(): + """Test that dataset covers multiple templates/topics.""" + with open(TRAIN_FILE) as f: + samples = [json.loads(line) for line in f] + + templates = set() + for sample in samples: + if "template" in sample: + templates.add(sample["template"]) + + # Should have at least 5 different templates + assert len(templates) >= 5, f"Dataset should have diverse templates, found: {templates}" + + +def test_generator_script_exists(): + """Test that the generator script exists and is executable.""" + assert GENERATOR_SCRIPT.exists(), f"Generator script not found: {GENERATOR_SCRIPT}" + + +@pytest.mark.slow +def test_dataset_regeneration(): + """Test that dataset can be regenerated successfully.""" + # Run the generator with a small number of records for quick test + result = subprocess.run( + [sys.executable, str(GENERATOR_SCRIPT), "--max-records", "20", "--seed", "99"], + cwd=REPO_ROOT, + capture_output=True, + text=True, + timeout=30 + ) + + assert result.returncode == 0, f"Generator failed: {result.stderr}" + assert "Dataset generated successfully" in result.stdout + + # Verify files still exist and are valid + assert TRAIN_FILE.exists() + assert TEST_FILE.exists() + with open(TRAIN_FILE) as f: + samples = [json.loads(line) for line in f] + assert len(samples) > 0 + + +def test_no_duplicate_hashes(): + """Test that samples have unique hashes (no exact duplicates).""" + with open(TRAIN_FILE) as f: + train_samples = [json.loads(line) for line in f] + with open(TEST_FILE) as f: + test_samples = [json.loads(line) for line in f] + + all_samples = train_samples + test_samples + hashes = [s.get("hash") for s in all_samples if "hash" in s] + + # Should have no duplicate hashes + assert len(hashes) == len(set(hashes)), "Found duplicate hashes in dataset" + + +if __name__ == "__main__": + pytest.main([__file__, "-v"])