Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions datasets/chat/github_actions/metadata.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,19 @@
"generation_seed": 99,
"workflows_parsed": 13,
"workflow_names": [
"Aria E2E Tests",
"Auto Validation",
"AzureML LoRA Train",
"QAI CI Pipeline",
"Code Quality",
"CodeQL Security Scanning",
"E2E + Integration Tests",
"Workflow Validation",
"Stale Issues and PRs",
"Deploy to GitHub Pages",
"PR Checks",
"Aria E2E Tests",
"E2E + Integration Tests",
"AzureML LoRA Train",
"Quantum Automation",
"Auto Validation",
"PR Checks",
"Release",
"Stale Issues and PRs",
"Workflow Validation"
"CodeQL Security Scanning",
"Code Quality"
],
"generation_date": "2026-03-26T17:47:40.871477"
}
4 changes: 2 additions & 2 deletions datasets/chat/github_actions/test.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{"messages": [{"role": "user", "content": "What is a matrix strategy in GitHub Actions?"}, {"role": "assistant", "content": "A matrix strategy in GitHub Actions allows you to run a job with multiple configurations automatically:\n\n```yaml\njobs:\n test:\n runs-on: ubuntu-latest\n strategy:\n matrix:\n python-version: ['3.9', '3.10', '3.11', '3.12']\n os: [ubuntu-latest, windows-latest, macos-latest]\n steps:\n - uses: actions/setup-python@v5\n with:\n python-version: ${{ matrix.python-version }}\n - run: python --version\n - run: pytest\n```\n\nThis creates 12 jobs (4 Python versions × 3 OS) automatically.\n\n**Key features**:\n- `${{ matrix.variable }}` accesses matrix values\n- `fail-fast: false` continues other jobs if one fails\n- `max-parallel` limits concurrent jobs\n- `include` adds specific combinations\n- `exclude` removes specific combinations\n\n**Use cases**:\n- Testing across multiple Python/Node versions\n- Cross-platform testing (Linux, Windows, macOS)\n- Testing with different dependency versions\n- Different configuration combinations"}], "source": "general_knowledge", "template": "matrix"}
{"messages": [{"role": "user", "content": "What jobs are defined in the 'Auto Validation' workflow?"}, {"role": "assistant", "content": "The workflow defines 1 job(s):\n\n\n**dry-run**:\n - Runs on: ubuntu-latest\n - Contains 5 step(s)"}], "source_workflow": "Auto Validation", "source_path": ".github/workflows/auto-validation.yml", "template": "jobs", "hash": "510dd0820fe98f3b"}
{"messages": [{"role": "user", "content": "When does the 'Aria E2E Tests' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "Aria E2E Tests", "source_path": ".github/workflows/aria-tests.yml", "template": "triggers", "hash": "0b108737ae091f7e"}
{"messages": [{"role": "user", "content": "What jobs are defined in the 'Workflow Validation' workflow?"}, {"role": "assistant", "content": "The workflow defines 2 job(s):\n\n\n**validate-workflows**:\n - Runs on: ubuntu-latest\n - Contains 5 step(s)\n\n**test-workflows**:\n - Runs on: ubuntu-latest\n - Depends on: validate-workflows\n - Contains 3 step(s)"}], "source_workflow": "Workflow Validation", "source_path": ".github/workflows/workflow-validation.yml", "template": "jobs", "hash": "cb9797af5c503cf7"}
{"messages": [{"role": "user", "content": "When does the 'QAI CI Pipeline' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "triggers", "hash": "021d60314cdee68f"}

Copilot AI Mar 22, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR modifies an existing file under datasets/, but the repo’s documented convention is that datasets/ is read-only/immutable. Please revert these dataset edits (or add new data as a new versioned dataset directory or under data_out/ and update the relevant training config to point at it).

Suggested change
{"messages": [{"role": "user", "content": "When does the 'QAI CI Pipeline' workflow run?"}, {"role": "assistant", "content": "This workflow has no explicit triggers defined."}], "source_workflow": "QAI CI Pipeline", "source_path": ".github/workflows/ci-pipeline.yml", "template": "triggers", "hash": "021d60314cdee68f"}

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove false trigger label for QAI CI Pipeline sample

.github/workflows/ci-pipeline.yml:14-20 declares push, pull_request, and schedule, but this newly added held-out example answers “This workflow has no explicit triggers defined.” Because datasets/chat/github_actions/test.json is the evaluation set, this introduces a bad gold label that will mark correct trigger answers as wrong and skew any metrics derived from the dataset.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Replace false trigger answer for QAI CI Pipeline

This new held-out sample says the workflow has no triggers, but .github/workflows/ci-pipeline.yml explicitly defines push, pull_request, and schedule under on (lines 14-20). Because this row is in test.json, correct model answers about that workflow’s trigger behavior will be graded as wrong, which directly corrupts evaluation results for the GitHub Actions dataset.

Useful? React with 👍 / 👎.

20 changes: 10 additions & 10 deletions datasets/chat/github_actions/train.json

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions tests/test_quantum_provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@

torch = pytest.importorskip("torch")

pytest.importorskip("torch", reason="torch is not installed")
import torch # noqa: E402

REPO_ROOT = Path(__file__).resolve().parents[1]
CHAT_CLI_SRC = REPO_ROOT / "ai-projects" / "chat-cli" / "src"
Expand Down
2 changes: 1 addition & 1 deletion tests/test_sql_integration.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# Check if SQLAlchemy is available for these tests

try:
# import sqlalchemy (unused)
import sqlalchemy as _sqlalchemy # noqa: F401
_sqlalchemy_available = True
except ImportError:
_sqlalchemy_available = False
Expand Down
2 changes: 2 additions & 0 deletions tests/test_ui_playwright.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@

from pathlib import Path

pytest.importorskip("playwright", reason="playwright is not installed")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid skipping the dedicated Playwright jobs on import failure

Both .github/workflows/aria-tests.yml:93-118 and .github/workflows/e2e-tests.yml:51-64 explicitly install Playwright and then run this file, so a top-level pytest.importorskip("playwright") turns any future Playwright import regression in CI into a green skipped test instead of a failing E2E job. In that case those workflows stop exercising browser coverage without surfacing the breakage they were added to catch.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove module-level importorskip from Playwright E2E test

I checked .github/workflows/aria-tests.yml (playwright job runs pytest tests/test_ui_playwright.py after installing Playwright, lines 93-118) and .github/workflows/e2e-tests.yml (same pattern, lines 51-64): these jobs are meant to fail if Playwright is broken. A top-level pytest.importorskip("playwright") turns that failure mode into a green skip, so dependency regressions in those dedicated Playwright pipelines can silently pass without browser coverage.

Useful? React with 👍 / 👎.


REPO_ROOT = Path(__file__).resolve().parents[1]
ARIA_WEB = REPO_ROOT / 'aria_web'
SERVER_URL = 'http://127.0.0.1:8080'
Expand Down
2 changes: 2 additions & 0 deletions tests/test_vision_inference.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
Tests cover:
""" # noqa: F401

from __future__ import annotations

from typing import Any
import importlib.util
import base64
Expand Down
4 changes: 3 additions & 1 deletion tests/test_web_app_security.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
from typing import Any, Generator

import pytest
from flask.testing import FlaskClient

pytest.importorskip("flask", reason="flask is not installed")
from flask.testing import FlaskClient # noqa: E402

# Import the module under test
REPO_ROOT = Path(__file__).resolve().parents[1]
Expand Down
Loading