From ba747d9c80c4e25a6f50567ce7ec0336f2a3032f Mon Sep 17 00:00:00 2001 From: "Tobias.Mikula" Date: Tue, 20 Jan 2026 16:15:09 +0100 Subject: [PATCH 1/4] Copilot configuration --- .github/agents/devops-engineer.agent.md | 36 +++++++++++ .github/agents/reviewer.agent.md | 35 +++++++++++ .github/agents/sdet.agent.md | 36 +++++++++++ .github/agents/senior-developer.agent.md | 36 +++++++++++ .github/agents/specification-master.agent.md | 43 +++++++++++++ .github/copilot-instructions.md | 63 ++++++++++++++++++++ ci_local.sh | 29 +++++++++ 7 files changed, 278 insertions(+) create mode 100644 .github/agents/devops-engineer.agent.md create mode 100644 .github/agents/reviewer.agent.md create mode 100644 .github/agents/sdet.agent.md create mode 100644 .github/agents/senior-developer.agent.md create mode 100644 .github/agents/specification-master.agent.md create mode 100644 .github/copilot-instructions.md create mode 100755 ci_local.sh diff --git a/.github/agents/devops-engineer.agent.md b/.github/agents/devops-engineer.agent.md new file mode 100644 index 0000000..cd303a9 --- /dev/null +++ b/.github/agents/devops-engineer.agent.md @@ -0,0 +1,36 @@ +--- +name: DevOps Engineer +description: Keeps CI/CD fast and reliable aligned with EventGate's AWS Lambda deployment and quality gates. +--- + +DevOps Engineer + +Mission +- Keep CI/CD fast, reliable, and aligned with EventGate's AWS Lambda architecture. + +Inputs +- Source code (`src/`, `tests/`), `requirements.txt`, `Dockerfile`, GitHub Actions workflows. + +Outputs +- GitHub Actions workflows (lint, type check, test, security scan). +- Docker image build pipeline for AWS Lambda. +- Coverage reports, CI badges. + +Responsibilities +- Maintain CI: Black, Pylint ≥9.5, mypy, pytest ≥80% coverage. +- Build Lambda Docker images (ARM64, `public.ecr.aws/lambda/python:3.13-arm64`). +- Optimize: parallel jobs, pip caching, conditional execution on changed files. +- Ensure `conftest.py` mocks work in CI (no real Kafka/EventBridge/Postgres). + +EventGate CI Requirements +- Python 3.13 Lambda runtime compatibility. +- Mocked external services (Kafka, EventBridge, Postgres, S3). +- Writers inherit from `Writer(ABC)` with lazy initialization. +- Config validation: `conf/config.json`, `conf/access.json`, `conf/topic_schemas/*.json`. + +Collaboration +- Align with SDET on pytest execution and coverage. +- Work with Senior Developer on CI failures. + +Definition of Done +- CI green, fast, all gates pass, Docker builds. diff --git a/.github/agents/reviewer.agent.md b/.github/agents/reviewer.agent.md new file mode 100644 index 0000000..f799727 --- /dev/null +++ b/.github/agents/reviewer.agent.md @@ -0,0 +1,35 @@ +--- +name: Reviewer +description: Guards correctness, performance, and contract stability; approves only when all gates pass. +--- + +Reviewer + +Mission +- Guard correctness, security, performance, and API contract stability in EventGate PRs. + +Inputs +- PR diffs, CI results (Black, Pylint, mypy, pytest), coverage reports. + +Outputs +- Review comments, approvals or change requests with clear rationale. + +Responsibilities +- Verify changes follow EventGate patterns (handlers, writers, route dispatch). +- Check quality gates: Black, Pylint ≥9.5, mypy clean, pytest ≥80% coverage. +- Ensure API Gateway responses remain stable (`statusCode`, `headers`, `body` structure). +- Verify new routes added to `ROUTE_HANDLERS` dict in `event_gate_lambda.py`. +- Check handler `__init__` methods are exception-free. +- Spot mocking issues in tests (Kafka, EventBridge, Postgres, S3 must be mocked). + +EventGate Contract Stability +- API routes: `/api`, `/token`, `/health`, `/topics`, `/topics/{topic_name}`, `/terminate`. +- Response format: `{"statusCode": int, "headers": {"Content-Type": "..."}, "body": "..."}`. +- Error format: `{"success": false, "statusCode": int, "errors": [{"type": "...", "message": "..."}]}`. + +Collaboration +- Coordinate with Specification Master on contract changes. +- Ask SDET for targeted tests when coverage is weak. + +Definition of Done +- Approve only when all gates pass, patterns followed, and no contract regressions. diff --git a/.github/agents/sdet.agent.md b/.github/agents/sdet.agent.md new file mode 100644 index 0000000..e04f984 --- /dev/null +++ b/.github/agents/sdet.agent.md @@ -0,0 +1,36 @@ +--- +name: SDET +description: Ensures automated test coverage, determinism, and fast feedback across the codebase. +--- + +SDET (Software Development Engineer in Test) + +Mission +- Ensure automated coverage, determinism, and fast feedback for EventGate Lambda. + +Inputs +- Specs/acceptance criteria, code changes, handler/writer implementations. + +Outputs +- Tests in `tests/` (unit, integration), coverage reports ≥80%. + +Responsibilities +- Maintain pytest tests for handlers (`HandlerApi`, `HandlerToken`, `HandlerTopic`, `HandlerHealth`). +- Mock external services via `conftest.py`: Kafka (confluent_kafka), EventBridge (boto3), PostgreSQL (psycopg2), S3. +- Test writers independently (`tests/writers/`) inheriting from `Writer` base class: mock `write()` and `check_health()`. +- Validate config files (`tests/test_conf_validation.py`): `config.json`, `access.json`, `topic_schemas/*.json`. +- Ensure deterministic fixtures; no real API/DB calls in tests. +- Enforce: Black, Pylint ≥9.5, mypy clean, pytest-cov ≥80%. + +EventGate Test Structure +- `tests/handlers/` - Handler class tests +- `tests/writers/` - Writer module tests (EventBridge, Kafka, Postgres) +- `tests/utils/` - Utility function tests +- `conftest.py` - Shared fixtures, mocked boto3/kafka/psycopg2 + +Collaboration +- Work with Senior Developer on test-first for new handlers/writers. +- Confirm specs with Specification Master; surface gaps early. + +Definition of Done +- Tests pass locally and in CI; coverage ≥80%; zero flakiness; no skipped tests. diff --git a/.github/agents/senior-developer.agent.md b/.github/agents/senior-developer.agent.md new file mode 100644 index 0000000..b0dd204 --- /dev/null +++ b/.github/agents/senior-developer.agent.md @@ -0,0 +1,36 @@ +--- +name: Senior Developer +description: Implements features and fixes with high quality, meeting specs and tests. +--- + +Senior Developer + +Mission +- Deliver maintainable features and fixes for EventGate Lambda, aligned to specs and tests. + +Inputs +- Task descriptions, specs from Specification Master, test plans from SDET, PR feedback. + +Outputs +- Focused code changes (PRs), unit tests for new logic, README updates when needed. + +Responsibilities +- Implement handlers following existing patterns (`HandlerApi`, `HandlerToken`, `HandlerTopic`, `HandlerHealth`). +- Implement writers inheriting from `Writer` base class: `__init__(config)`, `write(topic, message)`, `check_health()`. +- Use route dispatch pattern in `event_gate_lambda.py` (`ROUTE_HANDLERS` dict). +- Keep `__init__` methods exception-free; use lazy initialization in `write()` or `check_health()`. +- Meet quality gates: Black, Pylint ≥9.5, mypy clean, pytest-cov ≥80%. +- Use Python 3.13, type hints, `logging.getLogger(__name__)`. + +EventGate Patterns +- Handlers: class with `__init__` (no exceptions), `load_*()` methods returning `self` for chaining. +- Writers: inherit from `Writer(ABC)`, implement `write(topic, message) -> (bool, str|None)` and `check_health() -> (bool, str)`. +- Config: loaded from `conf/config.json`, `conf/access.json`, `conf/topic_schemas/*.json`. +- API responses: `{"statusCode": int, "headers": {...}, "body": json.dumps(...)}`. + +Collaboration +- Clarify acceptance criteria with Specification Master before coding. +- Pair with SDET on test-first for complex logic; respond quickly to Reviewer feedback. + +Definition of Done +- All checks green (Black, Pylint, mypy, pytest ≥80%); acceptance criteria met; no regressions. diff --git a/.github/agents/specification-master.agent.md b/.github/agents/specification-master.agent.md new file mode 100644 index 0000000..f08a7e1 --- /dev/null +++ b/.github/agents/specification-master.agent.md @@ -0,0 +1,43 @@ +--- +name: Specification Master +description: Produces precise, testable specs and maintains SPEC.md as the contract source of truth. +--- + +Specification Master + +Mission +- Produce precise, testable specifications for EventGate Lambda endpoints and handlers. + +Inputs +- Product goals, API requirements, prior failures, reviewer feedback. + +Outputs +- Task descriptions, acceptance criteria, edge cases. +- `SPEC.md` as single source of truth for EventGate contracts. + +Responsibilities +- Define API Gateway request/response contracts for each route. +- Specify handler behavior: inputs, outputs, error conditions. +- Document writer contracts: inherit from `Writer(ABC)`, implement `write(topic, message) -> (bool, str|None)` and `check_health() -> (bool, str)`. +- Define config schema requirements (`config.json`, `access.json`, topic schemas). +- Keep error response format stable: `{"success": false, "statusCode": int, "errors": [...]}`. + +EventGate Contracts to Maintain +- Routes: `/api`, `/token`, `/health`, `/topics`, `/topics/{topic_name}`. +- Handlers: `HandlerApi`, `HandlerToken`, `HandlerTopic`, `HandlerHealth`. +- Writers: EventBridge, Kafka, PostgreSQL - inherit from `Writer(ABC)`, use lazy initialization. +- Config: `conf/config.json` (required keys), `conf/access.json` (topic->users), `conf/topic_schemas/*.json`. + +`SPEC.md` Structure +- API Endpoints (routes, methods, request/response formats) +- Handler Contracts (inputs, outputs, error conditions) +- Writer Contracts (topic handling, success/failure semantics) +- Configuration Schema (required fields, types) +- Error Response Format + +Collaboration +- Align feasibility with Senior Developer. +- Review test plans with SDET; ensure specs are testable. + +Definition of Done +- Unambiguous acceptance criteria; contract changes documented with test update plan. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..e0ba44f --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,63 @@ +Copilot instructions for EventGate + +Purpose +EventGate is an AWS Lambda-based event gateway that receives messages via API Gateway and dispatches them to Kafka, EventBridge, and PostgreSQL. + +Context +- AWS Lambda entry point: `src/event_gate_lambda.py` +- Handlers: `HandlerApi`, `HandlerToken`, `HandlerTopic`, `HandlerHealth` in `src/handlers/` +- Writers: `WriterEventBridge`, `WriterKafka`, `WriterPostgres` in `src/writers/` (inherit from `Writer` base class) +- Config: `conf/config.json`, `conf/access.json`, `conf/topic_schemas/*.json` +- Routes defined in `ROUTE_HANDLERS` dict in `event_gate_lambda.py` + +Coding guidelines +- Keep changes small and focused +- Prefer clear, explicit code over clever tricks +- Do not change existing error messages or API response formats without approval +- Keep API Gateway response structure stable: `{"statusCode": int, "headers": {...}, "body": "..."}` + +Python and style +- Target Python 3.13 +- Add type hints for all public functions and classes +- Use `logging.getLogger(__name__)`, not print +- Use lazy % formatting in logging: `logger.info("msg %s", var)` +- All imports at top of file + +Patterns +- Handlers: class with `__init__` (no exceptions) +- Writers: inherit from `Writer(ABC)`, implement `write(topic, message) -> (bool, str|None)` and `check_health() -> (bool, str)` +- Use lazy initialization in writers; keep `__init__` exception-free +- Route dispatch via `ROUTE_HANDLERS` dict mapping routes to handler functions + +Testing +- Use pytest with tests in `tests/` +- Mock external services via `conftest.py`: Kafka, EventBridge, PostgreSQL, S3 +- No real API/DB calls in unit tests +- Test structure: `tests/handlers/`, `tests/writers/`, `tests/utils/` + +Quality gates +- Black formatting +- Pylint >= 9.5 +- mypy clean +- pytest coverage >= 80% + +Run `./ci_local.sh` before every commit to verify all quality gates pass. + +File overview +- `src/event_gate_lambda.py`: Lambda entry point, route dispatch +- `src/handlers/`: Handler classes (api, token, topic, health) +- `src/writers/`: Writer classes inheriting from `Writer` base +- `src/utils/`: Utilities (logging, config path, error responses) +- `conf/`: Configuration files and topic schemas +- `tests/`: pytest tests mirroring src structure + +Architecture notes +- Lambda receives API Gateway events, dispatches via `ROUTE_HANDLERS` +- Handlers manage request/response logic +- Writers handle message delivery to external systems +- All external dependencies mocked in tests + +Learned rules +- Keep error response format stable: `{"success": false, "statusCode": int, "errors": [...]}` +- Handler `__init__` must be exception-free +- Writers use lazy initialization diff --git a/ci_local.sh b/ci_local.sh new file mode 100755 index 0000000..e8c49cd --- /dev/null +++ b/ci_local.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# EventGate Local CI Quality Gates + +set -eo pipefail + +echo "=== EventGate Local CI ===" +echo "" + +echo "[1/4] Black (formatting)..." +black --check $(git ls-files '*.py') || { echo "Run: black \$(git ls-files '*.py')"; exit 1; } +echo "✓ Black passed" +echo "" + +echo "[2/4] Pylint (linting)..." +pylint --fail-under=9.5 $(git ls-files '*.py') || exit 1 +echo "✓ Pylint passed (≥9.5)" +echo "" + +echo "[3/4] mypy (type checking)..." +mypy src/ || exit 1 +echo "✓ mypy passed" +echo "" + +echo "[4/4] pytest (tests + coverage)..." +pytest --cov=src --cov-fail-under=80 -q tests/ || exit 1 +echo "✓ pytest passed (≥80% coverage)" +echo "" + +echo "=== All quality gates passed ✓ ===" From 4b5662b0a9d0bda53d7ec9438df25cc5fd8b9f1d Mon Sep 17 00:00:00 2001 From: "Tobias.Mikula" Date: Thu, 22 Jan 2026 16:53:16 +0100 Subject: [PATCH 2/4] Test --- .github/workflows/aquasec_repo_scan.yml | 328 +----------------------- test_aquasec_scan.sarif | 20 ++ 2 files changed, 24 insertions(+), 324 deletions(-) create mode 100644 test_aquasec_scan.sarif diff --git a/.github/workflows/aquasec_repo_scan.yml b/.github/workflows/aquasec_repo_scan.yml index f363a4e..686f390 100644 --- a/.github/workflows/aquasec_repo_scan.yml +++ b/.github/workflows/aquasec_repo_scan.yml @@ -17,7 +17,6 @@ concurrency: jobs: aquasec-scanning: - name: AquaSec Full Repository Scan runs-on: ubuntu-latest steps: - name: Checkout repository @@ -26,334 +25,15 @@ jobs: persist-credentials: false fetch-depth: 0 - - name: Retrieve AquaSec Scan Results - env: - AQUA_KEY: ${{ secrets.AQUA_KEY }} - AQUA_SECRET: ${{ secrets.AQUA_SECRET }} - REPOSITORY_ID: ${{ secrets.AQUA_REPOSITORY_ID }} + - name: Verify SARIF file exists run: | - set -euo pipefail - - echo "=== Validating secret variables ===" - - if ! [[ "$REPOSITORY_ID" =~ ^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$ ]]; then - echo "Error: AQUA_REPOSITORY_ID is not a valid UUID format" + if [ ! -f "test_aquasec_scan.sarif" ]; then + echo "test_aquasec_scan.sarif not found in repository root" exit 1 fi - - echo "=== Authenticating with AquaSec ===" - - METHOD="POST" - AUTH_ENDPOINT="https://eu-1.api.cloudsploit.com/v2/tokens" - TIMESTAMP=$(date -u +%s) - POST_BODY='{"group_id":1228,"allowed_endpoints":["GET"],"validity":240}' - STRING_TO_SIGN="${TIMESTAMP}${METHOD}/v2/tokens${POST_BODY}" - SIGNATURE=$(echo -n "$STRING_TO_SIGN" | openssl dgst -sha256 -hmac "${AQUA_SECRET}" -hex | sed 's/.*= //g') - - AUTH_RESPONSE=$(curl -s --max-time 30 -X $METHOD "$AUTH_ENDPOINT" \ - -H "Content-Type: application/json" \ - -H "X-API-Key: $AQUA_KEY" \ - -H "X-Timestamp: $TIMESTAMP" \ - -H "X-Signature: $SIGNATURE" \ - -d "$POST_BODY") - - RESPONSE_STATUS=$(echo "$AUTH_RESPONSE" | jq -r '.status') - - if [ "$RESPONSE_STATUS" = "200" ]; then - echo "Login successful." - BEARER_TOKEN=$(echo "$AUTH_RESPONSE" | jq -r '.data') - else - echo "Login failed with error message: $(echo "$AUTH_RESPONSE" | jq -r '.errors')" - exit 1 - fi - - echo "=== Receiving AquaSec Scan Results ===" - - SCAN_RESULTS_ENDPOINT="https://eu-1.codesec.aquasec.com/api/v1/scans/results" - FINDINGS_JSON="[]" - PAGE_NUM=1 - PAGE_SIZE=100 - TOTAL_EXPECTED=0 - - while true; do - echo "Fetching page $PAGE_NUM..." - - REQUEST_URL="${SCAN_RESULTS_ENDPOINT}?repositoryIds=${REPOSITORY_ID}&size=${PAGE_SIZE}&page=${PAGE_NUM}" - - PAGE_RESPONSE=$(curl -s --max-time 30 -X GET "$REQUEST_URL" \ - -H "Authorization: Bearer $BEARER_TOKEN" \ - -H "Accept: application/json") - - if [ -z "$PAGE_RESPONSE" ]; then - echo "Failed to retrieve scan results on page $PAGE_NUM" - exit 1 - fi - - if [ $PAGE_NUM -eq 1 ]; then - TOTAL_EXPECTED=$(echo "$PAGE_RESPONSE" | jq -r '.total // 0') - echo "Total findings expected: $TOTAL_EXPECTED" - fi - - PAGE_DATA=$(echo "$PAGE_RESPONSE" | jq -c '.data // []') - PAGE_COUNT=$(echo "$PAGE_DATA" | jq 'length') - echo "Retrieved $PAGE_COUNT findings on page $PAGE_NUM" - - FINDINGS_JSON=$(echo "$FINDINGS_JSON" "$PAGE_DATA" | jq -s 'add') - - FINDINGS_COUNT=$(echo "$FINDINGS_JSON" | jq 'length') - - if [ "$FINDINGS_COUNT" -ge "$TOTAL_EXPECTED" ] || [ "$PAGE_COUNT" -eq 0 ]; then - break - fi - - PAGE_NUM=$((PAGE_NUM + 1)) - sleep 2 - done - - FINDINGS_COUNT=$(echo "$FINDINGS_JSON" | jq 'length') - echo "Total findings retrieved: $FINDINGS_COUNT" - - jq -n --argjson total "$FINDINGS_COUNT" --argjson data "$FINDINGS_JSON" \ - '{"total": $total, "size": $total, "page": 1, "data": $data}' > aquasec_scan_results.json - - echo "Full repository scan retrieved successfully" - - - name: Convert to SARIF 2.1.0 - shell: python - run: | - import json - - print("=== Converting Scan Result to SARIF Format ===") - - # Severity mapping: SARIF level, security-severity, severity tag - SEVERITY_MAP = { - 1: ("note", "2.0", "LOW"), - 2: ("warning", "5.5", "MEDIUM"), - 3: ("error", "8.0", "HIGH"), - 4: ("error", "9.5", "CRITICAL"), - } - - # Truncate text to follow with GitHub SARIF field limits - def truncate(text, max_len=1024): - if not text: - return "Security issue detected" - return text[:max_len] if len(text) > max_len else text - - with open("aquasec_scan_results.json", "r") as f: - data = json.load(f) - - aquasec_findings = data.get("data", []) - rule_index_lookup = {} - sarif_unique_rules = [] - sarif_findings = [] - - for finding in aquasec_findings: - target_file = finding.get("target_file", "") - avd_id = finding.get("avd_id", "") - severity = finding.get("severity", 1) - level, sec_severity, sev_tag = SEVERITY_MAP.get(severity, SEVERITY_MAP[1]) - title = finding.get("title", "") - message = finding.get("message", "") - extra = finding.get("extraData", {}) - category = finding.get("category", "") - - if avd_id not in rule_index_lookup: - tags = [category, "security", sev_tag] - - refs = extra.get("references", []) - remediation = extra.get("remediation", "") - - rule = { - "id": avd_id, - "name": category, - "shortDescription": {"text": truncate(title)}, - "fullDescription": {"text": truncate(message)}, - "defaultConfiguration": {"level": level}, - "help": { - "text": truncate(remediation), - "markdown": f"**{category} {avd_id}**\n| Severity | Check | Message |\n| --- | --- | --- |\n|{sev_tag}|{truncate(title, 100)}|{truncate(message, 200)}|" - }, - "properties": { - "precision": "very-high", - "security-severity": sec_severity, - "tags": tags - } - } - - if refs: - rule["helpUri"] = refs[0] - - rule_index_lookup[avd_id] = len(sarif_unique_rules) - sarif_unique_rules.append(rule) - - # Sanitize security finding line numbers to please SARIF schema - start_line = finding.get("target_start_line") - if not start_line or start_line < 1: - start_line = 1 - end_line = finding.get("target_end_line") - if not end_line or end_line < start_line: - end_line = start_line - - sarif_finding = { - "ruleId": avd_id, - "ruleIndex": rule_index_lookup[avd_id], - "level": level, - "message": {"text": truncate(message)}, - "locations": [{ - "physicalLocation": { - "artifactLocation": {"uri": target_file}, - "region": {"startLine": start_line, "endLine": end_line} - } - }] - } - - sarif_findings.append(sarif_finding) - - sarif_output = { - "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json", - "version": "2.1.0", - "runs": [{ - "tool": { - "driver": { - "fullName": "AquaSec Security Scanner", - "informationUri": "https://www.aquasec.com/", - "name": "AquaSec", - "version": "1.0.0", - "rules": sarif_unique_rules - } - }, - "results": sarif_findings - }] - } - - with open("aquasec_scan.sarif", "w") as f: - json.dump(sarif_output, f, indent=2) - - print(f"Converted {len(sarif_findings)} findings to SARIF 2.1.0 format") - name: Upload Scan Results to GitHub Security uses: github/codeql-action/upload-sarif@4e94bd11f71e507f7f87df81788dff88d1dacbfb with: - sarif_file: aquasec_scan.sarif + sarif_file: test_aquasec_scan.sarif category: aquasec - - - name: Create Scan Summary Table - id: scan_summary_table - shell: python - run: | - import os - import json - import sys - from collections import Counter - - SARIF_PATH = "aquasec_scan.sarif" - SEVERITIES = ["CRITICAL", "HIGH", "MEDIUM", "LOW"] - CATEGORIES = ["sast", "vulnerabilities", "iacMisconfigurations", "secrets", "pipelineMisconfigurations", "license"] - - print("=== Generating Scan Summary Table ===") - - try: - with open(SARIF_PATH, "r", encoding="utf-8") as f: - sarif = json.load(f) - - if "runs" not in sarif or not sarif["runs"]: - raise ValueError("SARIF file contains no runs") - - run = sarif["runs"][0] - rules = run.get("tool", {}).get("driver", {}).get("rules", []) - results = run.get("results", []) - - except (IOError, json.JSONDecodeError, ValueError) as e: - print(f"Error processing SARIF file: {e}", file=sys.stderr) - sys.exit(1) - - # Initialize counters for each category - category_severity_counts = {cat: Counter() for cat in CATEGORIES} - - # Count results by category and severity - for result in results: - rule_idx = result.get("ruleIndex") - if rule_idx is None or rule_idx >= len(rules): - continue - - rule = rules[rule_idx] - category = rule.get("name", "") - tags = rule.get("properties", {}).get("tags", []) - severity = next((s for s in SEVERITIES if s in tags), None) - - if category in CATEGORIES and severity: - category_severity_counts[category][severity] += 1 - - # Build Markdown summary table - headers = ["AQUASEC"] + SEVERITIES + ["TOTAL"] - summary_table = "| " + " | ".join(headers) + " |\n" - summary_table += "|---|---|---|---|---|---|\n" - - total_severity = Counter() - total_all = 0 - for category in CATEGORIES: - row = [category] - category_total = 0 - for severity in SEVERITIES: - count = category_severity_counts[category][severity] - row.append(str(count)) - total_severity[severity] += count - category_total += count - row.append(f"**{category_total}**") - total_all += category_total - summary_table += "| " + " | ".join(row) + " |\n" - - total_row = ["**➡️ Total**"] + [f"**{total_severity[sev]}**" for sev in SEVERITIES] + [f"**{total_all}**"] - summary_table += "| " + " | ".join(total_row) + " |" - - try: - if "GITHUB_OUTPUT" in os.environ: - with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as f: - f.write("table< c.body.includes(marker)); - - // Create a new comment or update existing one - if (existingComment) { - await github.rest.issues.updateComment({ - owner: context.repo.owner, - repo: context.repo.repo, - comment_id: existingComment.id, - body - }); - } else { - await github.rest.issues.createComment({ - issue_number: context.issue.number, - owner: context.repo.owner, - repo: context.repo.repo, - body - }); - } diff --git a/test_aquasec_scan.sarif b/test_aquasec_scan.sarif new file mode 100644 index 0000000..6855eff --- /dev/null +++ b/test_aquasec_scan.sarif @@ -0,0 +1,20 @@ +{ + "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json", + "version": "2.1.0", + "runs": [ + { + "tool": { + "driver": { + "fullName": "AquaSec Security Scanner", + "informationUri": "https://www.aquasec.com/", + "name": "AquaSec", + "version": "1.0.0", + "rules": [ + ] + } + }, + "results": [ + ] + } + ] +} \ No newline at end of file From c30b293811a8b662fb9c5ad1b86f2cf68a59489c Mon Sep 17 00:00:00 2001 From: "Tobias.Mikula" Date: Thu, 22 Jan 2026 21:35:17 +0100 Subject: [PATCH 3/4] Revert "Test" This reverts commit 4b5662b0a9d0bda53d7ec9438df25cc5fd8b9f1d. --- .github/workflows/aquasec_repo_scan.yml | 328 +++++++++++++++++++++++- test_aquasec_scan.sarif | 20 -- 2 files changed, 324 insertions(+), 24 deletions(-) delete mode 100644 test_aquasec_scan.sarif diff --git a/.github/workflows/aquasec_repo_scan.yml b/.github/workflows/aquasec_repo_scan.yml index 686f390..f363a4e 100644 --- a/.github/workflows/aquasec_repo_scan.yml +++ b/.github/workflows/aquasec_repo_scan.yml @@ -17,6 +17,7 @@ concurrency: jobs: aquasec-scanning: + name: AquaSec Full Repository Scan runs-on: ubuntu-latest steps: - name: Checkout repository @@ -25,15 +26,334 @@ jobs: persist-credentials: false fetch-depth: 0 - - name: Verify SARIF file exists + - name: Retrieve AquaSec Scan Results + env: + AQUA_KEY: ${{ secrets.AQUA_KEY }} + AQUA_SECRET: ${{ secrets.AQUA_SECRET }} + REPOSITORY_ID: ${{ secrets.AQUA_REPOSITORY_ID }} run: | - if [ ! -f "test_aquasec_scan.sarif" ]; then - echo "test_aquasec_scan.sarif not found in repository root" + set -euo pipefail + + echo "=== Validating secret variables ===" + + if ! [[ "$REPOSITORY_ID" =~ ^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$ ]]; then + echo "Error: AQUA_REPOSITORY_ID is not a valid UUID format" exit 1 fi + + echo "=== Authenticating with AquaSec ===" + + METHOD="POST" + AUTH_ENDPOINT="https://eu-1.api.cloudsploit.com/v2/tokens" + TIMESTAMP=$(date -u +%s) + POST_BODY='{"group_id":1228,"allowed_endpoints":["GET"],"validity":240}' + STRING_TO_SIGN="${TIMESTAMP}${METHOD}/v2/tokens${POST_BODY}" + SIGNATURE=$(echo -n "$STRING_TO_SIGN" | openssl dgst -sha256 -hmac "${AQUA_SECRET}" -hex | sed 's/.*= //g') + + AUTH_RESPONSE=$(curl -s --max-time 30 -X $METHOD "$AUTH_ENDPOINT" \ + -H "Content-Type: application/json" \ + -H "X-API-Key: $AQUA_KEY" \ + -H "X-Timestamp: $TIMESTAMP" \ + -H "X-Signature: $SIGNATURE" \ + -d "$POST_BODY") + + RESPONSE_STATUS=$(echo "$AUTH_RESPONSE" | jq -r '.status') + + if [ "$RESPONSE_STATUS" = "200" ]; then + echo "Login successful." + BEARER_TOKEN=$(echo "$AUTH_RESPONSE" | jq -r '.data') + else + echo "Login failed with error message: $(echo "$AUTH_RESPONSE" | jq -r '.errors')" + exit 1 + fi + + echo "=== Receiving AquaSec Scan Results ===" + + SCAN_RESULTS_ENDPOINT="https://eu-1.codesec.aquasec.com/api/v1/scans/results" + FINDINGS_JSON="[]" + PAGE_NUM=1 + PAGE_SIZE=100 + TOTAL_EXPECTED=0 + + while true; do + echo "Fetching page $PAGE_NUM..." + + REQUEST_URL="${SCAN_RESULTS_ENDPOINT}?repositoryIds=${REPOSITORY_ID}&size=${PAGE_SIZE}&page=${PAGE_NUM}" + + PAGE_RESPONSE=$(curl -s --max-time 30 -X GET "$REQUEST_URL" \ + -H "Authorization: Bearer $BEARER_TOKEN" \ + -H "Accept: application/json") + + if [ -z "$PAGE_RESPONSE" ]; then + echo "Failed to retrieve scan results on page $PAGE_NUM" + exit 1 + fi + + if [ $PAGE_NUM -eq 1 ]; then + TOTAL_EXPECTED=$(echo "$PAGE_RESPONSE" | jq -r '.total // 0') + echo "Total findings expected: $TOTAL_EXPECTED" + fi + + PAGE_DATA=$(echo "$PAGE_RESPONSE" | jq -c '.data // []') + PAGE_COUNT=$(echo "$PAGE_DATA" | jq 'length') + echo "Retrieved $PAGE_COUNT findings on page $PAGE_NUM" + + FINDINGS_JSON=$(echo "$FINDINGS_JSON" "$PAGE_DATA" | jq -s 'add') + + FINDINGS_COUNT=$(echo "$FINDINGS_JSON" | jq 'length') + + if [ "$FINDINGS_COUNT" -ge "$TOTAL_EXPECTED" ] || [ "$PAGE_COUNT" -eq 0 ]; then + break + fi + + PAGE_NUM=$((PAGE_NUM + 1)) + sleep 2 + done + + FINDINGS_COUNT=$(echo "$FINDINGS_JSON" | jq 'length') + echo "Total findings retrieved: $FINDINGS_COUNT" + + jq -n --argjson total "$FINDINGS_COUNT" --argjson data "$FINDINGS_JSON" \ + '{"total": $total, "size": $total, "page": 1, "data": $data}' > aquasec_scan_results.json + + echo "Full repository scan retrieved successfully" + + - name: Convert to SARIF 2.1.0 + shell: python + run: | + import json + + print("=== Converting Scan Result to SARIF Format ===") + + # Severity mapping: SARIF level, security-severity, severity tag + SEVERITY_MAP = { + 1: ("note", "2.0", "LOW"), + 2: ("warning", "5.5", "MEDIUM"), + 3: ("error", "8.0", "HIGH"), + 4: ("error", "9.5", "CRITICAL"), + } + + # Truncate text to follow with GitHub SARIF field limits + def truncate(text, max_len=1024): + if not text: + return "Security issue detected" + return text[:max_len] if len(text) > max_len else text + + with open("aquasec_scan_results.json", "r") as f: + data = json.load(f) + + aquasec_findings = data.get("data", []) + rule_index_lookup = {} + sarif_unique_rules = [] + sarif_findings = [] + + for finding in aquasec_findings: + target_file = finding.get("target_file", "") + avd_id = finding.get("avd_id", "") + severity = finding.get("severity", 1) + level, sec_severity, sev_tag = SEVERITY_MAP.get(severity, SEVERITY_MAP[1]) + title = finding.get("title", "") + message = finding.get("message", "") + extra = finding.get("extraData", {}) + category = finding.get("category", "") + + if avd_id not in rule_index_lookup: + tags = [category, "security", sev_tag] + + refs = extra.get("references", []) + remediation = extra.get("remediation", "") + + rule = { + "id": avd_id, + "name": category, + "shortDescription": {"text": truncate(title)}, + "fullDescription": {"text": truncate(message)}, + "defaultConfiguration": {"level": level}, + "help": { + "text": truncate(remediation), + "markdown": f"**{category} {avd_id}**\n| Severity | Check | Message |\n| --- | --- | --- |\n|{sev_tag}|{truncate(title, 100)}|{truncate(message, 200)}|" + }, + "properties": { + "precision": "very-high", + "security-severity": sec_severity, + "tags": tags + } + } + + if refs: + rule["helpUri"] = refs[0] + + rule_index_lookup[avd_id] = len(sarif_unique_rules) + sarif_unique_rules.append(rule) + + # Sanitize security finding line numbers to please SARIF schema + start_line = finding.get("target_start_line") + if not start_line or start_line < 1: + start_line = 1 + end_line = finding.get("target_end_line") + if not end_line or end_line < start_line: + end_line = start_line + + sarif_finding = { + "ruleId": avd_id, + "ruleIndex": rule_index_lookup[avd_id], + "level": level, + "message": {"text": truncate(message)}, + "locations": [{ + "physicalLocation": { + "artifactLocation": {"uri": target_file}, + "region": {"startLine": start_line, "endLine": end_line} + } + }] + } + + sarif_findings.append(sarif_finding) + + sarif_output = { + "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json", + "version": "2.1.0", + "runs": [{ + "tool": { + "driver": { + "fullName": "AquaSec Security Scanner", + "informationUri": "https://www.aquasec.com/", + "name": "AquaSec", + "version": "1.0.0", + "rules": sarif_unique_rules + } + }, + "results": sarif_findings + }] + } + + with open("aquasec_scan.sarif", "w") as f: + json.dump(sarif_output, f, indent=2) + + print(f"Converted {len(sarif_findings)} findings to SARIF 2.1.0 format") - name: Upload Scan Results to GitHub Security uses: github/codeql-action/upload-sarif@4e94bd11f71e507f7f87df81788dff88d1dacbfb with: - sarif_file: test_aquasec_scan.sarif + sarif_file: aquasec_scan.sarif category: aquasec + + - name: Create Scan Summary Table + id: scan_summary_table + shell: python + run: | + import os + import json + import sys + from collections import Counter + + SARIF_PATH = "aquasec_scan.sarif" + SEVERITIES = ["CRITICAL", "HIGH", "MEDIUM", "LOW"] + CATEGORIES = ["sast", "vulnerabilities", "iacMisconfigurations", "secrets", "pipelineMisconfigurations", "license"] + + print("=== Generating Scan Summary Table ===") + + try: + with open(SARIF_PATH, "r", encoding="utf-8") as f: + sarif = json.load(f) + + if "runs" not in sarif or not sarif["runs"]: + raise ValueError("SARIF file contains no runs") + + run = sarif["runs"][0] + rules = run.get("tool", {}).get("driver", {}).get("rules", []) + results = run.get("results", []) + + except (IOError, json.JSONDecodeError, ValueError) as e: + print(f"Error processing SARIF file: {e}", file=sys.stderr) + sys.exit(1) + + # Initialize counters for each category + category_severity_counts = {cat: Counter() for cat in CATEGORIES} + + # Count results by category and severity + for result in results: + rule_idx = result.get("ruleIndex") + if rule_idx is None or rule_idx >= len(rules): + continue + + rule = rules[rule_idx] + category = rule.get("name", "") + tags = rule.get("properties", {}).get("tags", []) + severity = next((s for s in SEVERITIES if s in tags), None) + + if category in CATEGORIES and severity: + category_severity_counts[category][severity] += 1 + + # Build Markdown summary table + headers = ["AQUASEC"] + SEVERITIES + ["TOTAL"] + summary_table = "| " + " | ".join(headers) + " |\n" + summary_table += "|---|---|---|---|---|---|\n" + + total_severity = Counter() + total_all = 0 + for category in CATEGORIES: + row = [category] + category_total = 0 + for severity in SEVERITIES: + count = category_severity_counts[category][severity] + row.append(str(count)) + total_severity[severity] += count + category_total += count + row.append(f"**{category_total}**") + total_all += category_total + summary_table += "| " + " | ".join(row) + " |\n" + + total_row = ["**➡️ Total**"] + [f"**{total_severity[sev]}**" for sev in SEVERITIES] + [f"**{total_all}**"] + summary_table += "| " + " | ".join(total_row) + " |" + + try: + if "GITHUB_OUTPUT" in os.environ: + with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as f: + f.write("table< c.body.includes(marker)); + + // Create a new comment or update existing one + if (existingComment) { + await github.rest.issues.updateComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: existingComment.id, + body + }); + } else { + await github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body + }); + } diff --git a/test_aquasec_scan.sarif b/test_aquasec_scan.sarif deleted file mode 100644 index 6855eff..0000000 --- a/test_aquasec_scan.sarif +++ /dev/null @@ -1,20 +0,0 @@ -{ - "$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json", - "version": "2.1.0", - "runs": [ - { - "tool": { - "driver": { - "fullName": "AquaSec Security Scanner", - "informationUri": "https://www.aquasec.com/", - "name": "AquaSec", - "version": "1.0.0", - "rules": [ - ] - } - }, - "results": [ - ] - } - ] -} \ No newline at end of file From 674cd11b35718987370edcd8208975aec1fcf908 Mon Sep 17 00:00:00 2001 From: "Tobias.Mikula" Date: Tue, 27 Jan 2026 15:36:51 +0100 Subject: [PATCH 4/4] copilot-instructions.md file update. --- .github/copilot-instructions.md | 76 ++++++++++++--------------------- ci_local.sh | 29 ------------- 2 files changed, 27 insertions(+), 78 deletions(-) delete mode 100755 ci_local.sh diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index e0ba44f..dd55154 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,63 +1,41 @@ Copilot instructions for EventGate Purpose -EventGate is an AWS Lambda-based event gateway that receives messages via API Gateway and dispatches them to Kafka, EventBridge, and PostgreSQL. +AWS Lambda event gateway that receives messages via API Gateway and dispatches them to Kafka, EventBridge, and PostgreSQL. -Context -- AWS Lambda entry point: `src/event_gate_lambda.py` -- Handlers: `HandlerApi`, `HandlerToken`, `HandlerTopic`, `HandlerHealth` in `src/handlers/` -- Writers: `WriterEventBridge`, `WriterKafka`, `WriterPostgres` in `src/writers/` (inherit from `Writer` base class) +Structure +- Entry point: `src/event_gate_lambda.py` +- Handlers: `src/handlers/` (HandlerApi, HandlerToken, HandlerTopic, HandlerHealth) +- Writers: `src/writers/` (inherit from `Writer` base class) - Config: `conf/config.json`, `conf/access.json`, `conf/topic_schemas/*.json` -- Routes defined in `ROUTE_HANDLERS` dict in `event_gate_lambda.py` +- Terraform scripts are not part of this repository -Coding guidelines -- Keep changes small and focused -- Prefer clear, explicit code over clever tricks -- Do not change existing error messages or API response formats without approval -- Keep API Gateway response structure stable: `{"statusCode": int, "headers": {...}, "body": "..."}` - -Python and style -- Target Python 3.13 -- Add type hints for all public functions and classes +Python style +- Python 3.13 +- Type hints for public functions and classes - Use `logging.getLogger(__name__)`, not print -- Use lazy % formatting in logging: `logger.info("msg %s", var)` -- All imports at top of file +- Lazy % formatting in logging: `logger.info("msg %s", var)` +- F-strings in exceptions: `raise ValueError(f"Error {var}")` +- All imports at top of file (never inside functions) Patterns -- Handlers: class with `__init__` (no exceptions) +- Classes with `__init__` cannot throw exceptions - Writers: inherit from `Writer(ABC)`, implement `write(topic, message) -> (bool, str|None)` and `check_health() -> (bool, str)` -- Use lazy initialization in writers; keep `__init__` exception-free -- Route dispatch via `ROUTE_HANDLERS` dict mapping routes to handler functions +- Writers use lazy initialization +- Route dispatch via `ROUTES` dict mapping routes to handler functions +- Preserve existing formatting and conventions +- Keep API Gateway response structure stable: `{"statusCode": int, "headers": {...}, "body": "..."}` +- Keep error response format stable: `{"success": false, "statusCode": int, "errors": [...]}` Testing -- Use pytest with tests in `tests/` +- Mirror src structure: `src/handlers/` -> `tests/handlers/` - Mock external services via `conftest.py`: Kafka, EventBridge, PostgreSQL, S3 - No real API/DB calls in unit tests -- Test structure: `tests/handlers/`, `tests/writers/`, `tests/utils/` - -Quality gates -- Black formatting -- Pylint >= 9.5 -- mypy clean -- pytest coverage >= 80% - -Run `./ci_local.sh` before every commit to verify all quality gates pass. - -File overview -- `src/event_gate_lambda.py`: Lambda entry point, route dispatch -- `src/handlers/`: Handler classes (api, token, topic, health) -- `src/writers/`: Writer classes inheriting from `Writer` base -- `src/utils/`: Utilities (logging, config path, error responses) -- `conf/`: Configuration files and topic schemas -- `tests/`: pytest tests mirroring src structure - -Architecture notes -- Lambda receives API Gateway events, dispatches via `ROUTE_HANDLERS` -- Handlers manage request/response logic -- Writers handle message delivery to external systems -- All external dependencies mocked in tests - -Learned rules -- Keep error response format stable: `{"success": false, "statusCode": int, "errors": [...]}` -- Handler `__init__` must be exception-free -- Writers use lazy initialization +- Use `mocker.patch("module.dependency")` or `mocker.patch.object(Class, "method")` +- Assert pattern: `assert expected == actual` + +Quality gates (run after changes, fix only if below threshold) +- black . +- mypy . +- pylint $(git ls-files '*.py') >= 9.5 +- pytest tests/ >= 80% coverage diff --git a/ci_local.sh b/ci_local.sh deleted file mode 100755 index e8c49cd..0000000 --- a/ci_local.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash -# EventGate Local CI Quality Gates - -set -eo pipefail - -echo "=== EventGate Local CI ===" -echo "" - -echo "[1/4] Black (formatting)..." -black --check $(git ls-files '*.py') || { echo "Run: black \$(git ls-files '*.py')"; exit 1; } -echo "✓ Black passed" -echo "" - -echo "[2/4] Pylint (linting)..." -pylint --fail-under=9.5 $(git ls-files '*.py') || exit 1 -echo "✓ Pylint passed (≥9.5)" -echo "" - -echo "[3/4] mypy (type checking)..." -mypy src/ || exit 1 -echo "✓ mypy passed" -echo "" - -echo "[4/4] pytest (tests + coverage)..." -pytest --cov=src --cov-fail-under=80 -q tests/ || exit 1 -echo "✓ pytest passed (≥80% coverage)" -echo "" - -echo "=== All quality gates passed ✓ ==="