Add GitHub Actions training dataset for CI/CD workflow understanding

README.md

@@ -216,6 +216,7 @@ Fine-tuning workspace for training Aria's language understanding using LoRA and
 - `datasets/chat/aria_movement/` - Movement command training data
 - `datasets/chat/aria_expanded/` - Extended Aria interactions
 - `datasets/chat/aria_simple/` - Basic Aria commands
+- `datasets/chat/github_actions/` - GitHub Actions & CI/CD workflows training data
 
 **Quick Training:**
 

datasets/chat/github_actions/README.md

@@ -0,0 +1,178 @@
+# GitHub Actions Training Dataset
+
+This dataset contains synthetic Q&A pairs for training AI models to understand GitHub Actions, CI/CD workflows, and DevOps best practices.
+
+## Overview
+
+- **Train samples**: 42
+- **Test samples**: 8
+- **Total samples**: 50
+- **Format**: JSONL (newline-delimited JSON)
+- **Generated from**: Real workflow files in `.github/workflows/`
+
+## Content
+
+The dataset covers the following topics:
+
+### Workflow-Specific Content
+Generated from 6 actual workflows in this repository:
+1. **Auto Validation** - Automated validation workflows
+2. **E2E + Integration Tests** - End-to-end and integration testing
+3. **Aria E2E Tests** - Aria character system tests
+4. **QAI CI Pipeline** - Main CI/CD pipeline with validate/train/deploy stages
+5. **Quantum Automation** - Quantum computing workflow orchestration
+6. **AzureML LoRA Train** - Azure ML model training workflows
+
+### Q&A Templates
+For each workflow, the dataset includes:
+- **Overview**: What the workflow does and its purpose
+- **Triggers**: When and how the workflow runs
+- **Jobs**: Detailed job descriptions and dependencies
+- **Actions Used**: GitHub Actions and their purposes
+- **Best Practices**: CI/CD best practices demonstrated
+- **Modification Guide**: How to add or change steps
+- **Troubleshooting**: Debugging workflow failures
+
+### General GitHub Actions Knowledge
+Includes foundational Q&A pairs about:
+- What GitHub Actions is and how it works
+- Workflow file structure (YAML format)
+- Common triggers (push, pull_request, schedule, etc.)
+- Using secrets and environment variables
+- Difference between `run` and `uses`
+- Parallel vs sequential job execution
+- Matrix strategies for testing multiple configurations
+- Dependency caching for faster workflows
+
+## File Format
+
+Each line in the JSONL files is a JSON object with this structure:
+
+```json
+{
+  "messages": [
+    {"role": "user", "content": "Question about GitHub Actions"},
+    {"role": "assistant", "content": "Detailed answer with examples"}
+  ],
+  "source_workflow": "Workflow Name",
+  "source_path": ".github/workflows/example.yml",
+  "template": "overview",
+  "hash": "abc123"
+}
+```
+
+## Generation
+
+This dataset was generated using `scripts/generate_github_actions_dataset.py`:
+
+```bash
+python scripts/generate_github_actions_dataset.py --max-records 200
+```
+
+To regenerate with different parameters:
+
+```bash
+# Generate more samples
+python scripts/generate_github_actions_dataset.py --max-records 500
+
+# Use different seed for variation
+python scripts/generate_github_actions_dataset.py --seed 123
+
+# Different train/test split
+python scripts/generate_github_actions_dataset.py --train-ratio 0.9
+```
+
+## Training Usage
+
+This dataset is automatically discovered by the autonomous training orchestrator and can be used with any of the training scripts:
+
+### Quick Training
+```bash
+# Using the automated pipeline
+python scripts/automated_training_pipeline.py --quick --models tinyllama
+
+# Using the aria quick train (adapted for this dataset)
+python scripts/aria_quick_train.py
+```
+
+### Manual Training with LoRA
+```bash
+cd AI/microsoft_phi-silica-3.6_v1
+python scripts/train_lora.py \
+  --dataset ../../datasets/chat/github_actions \
+  --hf-model-id TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
+  --epochs 25 \
+  --max-train-samples 42 \
+  --max-eval-samples 8
+```
+
+### Autonomous Training
+The dataset is automatically included in autonomous training cycles:
+
+```bash
+# Start autonomous training (runs every 30 minutes)
+nohup python scripts/autonomous_training_orchestrator.py > data_out/autonomous_training.log 2>&1 &
+
+# Trigger immediate cycle
+pkill -USR1 -f autonomous_training
+```
+
+## Validation
+
+To validate the dataset:
+
+```bash
+# Manual validation
+python -c "
+import json
+with open('datasets/chat/github_actions/train.json') as f:
+    samples = [json.loads(line) for line in f]
+    print(f'✅ Loaded {len(samples)} valid training samples')
+"
+```
+
+## Use Cases
+
+This dataset trains models to:
+1. **Explain workflows**: Describe what GitHub Actions workflows do
+2. **Answer CI/CD questions**: Provide guidance on workflow triggers, jobs, and steps
+3. **Debug workflows**: Help troubleshoot common workflow failures
+4. **Best practices**: Teach CI/CD best practices and patterns
+5. **Modification guidance**: Guide developers on how to extend workflows
+6. **General GitHub Actions knowledge**: Understand core concepts like matrix strategies, caching, secrets
+
+## Dataset Quality
+
+✅ **Strengths**:
+- Based on real, working workflows from this repository
+- Covers diverse CI/CD patterns (testing, training, deployment)
+- Includes both specific and general knowledge
+- Consistent format compatible with existing training infrastructure
+- Multiple question templates for varied learning
+
+⚠️ **Limitations**:
+- Limited to 50 samples (can be expanded by adjusting `--max-records`)
+- Focused on workflows in this repository (Python, ML, quantum computing)
+- May need supplementation with workflows from other domains (e.g., web frontend, mobile)
+
+## Future Enhancements
+
+Potential improvements:
+1. Parse more workflow files from other repositories
+2. Add examples of advanced GitHub Actions features (reusable workflows, custom actions)
+3. Include workflow run history and failure patterns
+4. Add examples of GitHub Actions API usage
+5. Cover GitHub Actions security best practices
+6. Include composite actions and action development
+
+## Related Datasets
+
+Other chat datasets in this repository:
+- `app_repo/` - Repository code and documentation
+- `aria_movement/` - Aria character movement commands
+- `comprehensive/` - Mixed general chat topics
+- `dolly/` - Instruction-following dataset
+
+## License
+
+This synthetic dataset is generated from the repository's own workflow files and is part of the Aria project.

datasets/chat/github_actions/metadata.json

@@ -0,0 +1,16 @@
+{
+  "total_records": 50,
+  "train_records": 42,
+  "test_records": 8,
+  "generation_seed": 42,
+  "workflows_parsed": 6,
+  "workflow_names": [
+    "Auto Validation",
+    "E2E + Integration Tests",
+    "Aria E2E Tests",
+    "QAI CI Pipeline",
+    "Quantum Automation",
+    "AzureML LoRA Train"
+  ],
+  "generation_date": "2026-01-22T15:34:36.150978"
+}

datasets/chat/github_actions/test.json

@@ -0,0 +1,8 @@
+{"messages": [{"role": "user", "content": "What is a matrix strategy in GitHub Actions?"}, {"role": "assistant", "content": "A matrix strategy in GitHub Actions allows you to run a job with multiple configurations automatically:\n\n```yaml\njobs:\n  test:\n    runs-on: ubuntu-latest\n    strategy:\n      matrix:\n        python-version: ['3.9', '3.10', '3.11', '3.12']\n        os: [ubuntu-latest, windows-latest, macos-latest]\n    steps:\n      - uses: actions/setup-python@v5\n        with:\n          python-version: ${{ matrix.python-version }}\n      - run: python --version\n      - run: pytest\n```\n\nThis creates 12 jobs (4 Python versions × 3 OS) automatically.\n\n**Key features**:\n- `${{ matrix.variable }}` accesses matrix values\n- `fail-fast: false` continues other jobs if one fails\n- `max-parallel` limits concurrent jobs\n- `include` adds specific combinations\n- `exclude` removes specific combinations\n\n**Use cases**:\n- Testing across multiple Python/Node versions\n- Cross-platform testing (Linux, Windows, macOS)\n- Testing with different dependency versions\n- Different configuration combinations"}], "source": "general_knowledge", "template": "matrix"}
+{"messages": [{"role": "user", "content": "When does the 'Auto Validation' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "triggers", "hash": "1cd6da4993cb8a3f"}
+{"messages": [{"role": "user", "content": "How would I modify the 'Auto Validation' workflow to add a new step?"}, {"role": "assistant", "content": "To add a new step to the 'Auto Validation' workflow:\n\n1. Choose which job to modify (available: dry-run)\n2. Locate the `steps` section within that job\n3. Add a new step with:\n   - `name`: A descriptive name for the step\n   - `run`: Command(s) to execute, or\n   - `uses`: An action to use (e.g., actions/checkout@v4)\n4. Optionally add conditions with `if:` to control when the step runs\n\nExample step structure:\n```yaml\n- name: My new step\n  run: |\n    echo \"Running my command\"\n    python my_script.py\n```\n\nOr using an action:\n```yaml\n- name: Upload results\n  uses: actions/upload-artifact@v4\n  with:\n    name: my-artifact\n    path: results/\n```\n\nRemember to test your changes in a feature branch before merging to main."}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "modification", "hash": "d328c4f201a13de3"}
+{"messages": [{"role": "user", "content": "What GitHub Actions are used in the 'E2E + Integration Tests' workflow?"}, {"role": "assistant", "content": "The workflow uses 2 different actions:\n\n- **actions/checkout@v4**: Checks out the repository code\n- **actions/setup-python@v4**: Sets up a Python environment"}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "actions", "hash": "40ba1e664aeb4659"}
+{"messages": [{"role": "user", "content": "What jobs are defined in the 'E2E + Integration Tests' workflow?"}, {"role": "assistant", "content": "The workflow defines 3 job(s):\n\n\n**integration**:\n  - Runs on: ubuntu-latest\n  - Contains 5 step(s)\n\n**e2e_playwright**:\n  - Runs on: ubuntu-latest\n  - Contains 5 step(s)\n\n**containerized_chrome**:\n  - Runs on: ubuntu-latest\n  - Contains 5 step(s)"}], "source_workflow": "E2E + Integration Tests", "source_path": ".github/workflows/e2e-tests.yml", "template": "jobs", "hash": "9b45784bd38ac82f"}
+{"messages": [{"role": "user", "content": "What is the structure of a GitHub Actions workflow file?"}, {"role": "assistant", "content": "A GitHub Actions workflow file is a YAML file with the following main sections:\n\n1. **name**: Optional workflow name displayed in the Actions tab\n2. **on**: Defines triggers (push, pull_request, schedule, etc.)\n3. **jobs**: Contains one or more jobs to run\n   - Each job has:\n     - **runs-on**: Runner type (ubuntu-latest, windows-latest, etc.)\n     - **steps**: Sequential actions or commands to execute\n     - **needs**: Optional dependencies on other jobs\n4. **env**: Optional environment variables\n5. **permissions**: Optional permissions settings\n\nExample:\n```yaml\nname: CI\non: [push, pull_request]\njobs:\n  test:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - run: npm test\n```"}], "source": "general_knowledge", "template": "structure"}
+{"messages": [{"role": "user", "content": "How do I cache dependencies in GitHub Actions?"}, {"role": "assistant", "content": "Caching dependencies speeds up workflow runs by reusing previously downloaded packages:\n\n**Using actions/cache**:\n```yaml\nsteps:\n  - uses: actions/checkout@v4\n  \n  - uses: actions/cache@v4\n    with:\n      path: ~/.npm\n      key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}\n      restore-keys: |\n        ${{ runner.os }}-node-\n  \n  - run: npm install\n```\n\n**Language-specific setup actions** with built-in caching:\n```yaml\n# Python with pip cache\n- uses: actions/setup-python@v5\n  with:\n    python-version: '3.11'\n    cache: 'pip'\n\n# Node.js with npm cache\n- uses: actions/setup-node@v4\n  with:\n    node-version: '20'\n    cache: 'npm'\n```\n\n**Benefits**:\n- Reduces workflow time by 30-70%\n- Decreases network load\n- Cache expires after 7 days of no access\n- Maximum cache size: 10 GB per repository\n\n**Best practices**:\n- Use hash of lock files in cache key for version tracking\n- Include OS in key if caching OS-specific files\n- Use restore-keys for partial cache hits"}], "source": "general_knowledge", "template": "caching"}
+{"messages": [{"role": "user", "content": "What does the GitHub Actions workflow 'Quantum Automation' do?"}, {"role": "assistant", "content": "The workflow 'Quantum Automation' is a CI/CD pipeline that:\n- Contains 1 job(s): run-quantum\n- Uses 2 different GitHub Actions"}], "source_workflow": "Quantum Automation", "source_path": ".github/workflows/quantum-orchestration.yml", "template": "overview", "hash": "928bae82eae9d3dd"}