Skip to content

🧪 QA: Add SOC2 compliance monitor tests #327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
daggerstuff wants to merge 8 commits into
base: staging
Choose a base branch
from
+66 −0
Open

🧪 QA: Add SOC2 compliance monitor tests #327

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
3bfe5c9
🧪 QA: Add tests for SOC2 compliance monitor checks
google-labs-jules[bot] Mar 31, 2026
c7a1aa2
🧪 QA: Add tests for SOC2 compliance monitor checks
google-labs-jules[bot] Mar 31, 2026
e3f8cc5
🧪 QA: Add tests for SOC2 compliance monitor checks
google-labs-jules[bot] Mar 31, 2026
a78e452
🧪 QA: Add tests for SOC2 compliance monitor checks
google-labs-jules[bot] Mar 31, 2026
8880f7f
🧪 QA: Add tests for SOC2 compliance monitor checks
google-labs-jules[bot] Mar 31, 2026
282a0a7
🧪 QA: Add tests for SOC2 compliance monitor checks
google-labs-jules[bot] Apr 1, 2026
52a4763
🧪 QA: Add tests for SOC2 compliance monitor checks
google-labs-jules[bot] Apr 1, 2026
caebe6d
🧪 QA: Add tests for SOC2 compliance monitor checks
google-labs-jules[bot] Apr 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions tests/test_compliance_monitor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
import pytest
import sys
import os
import importlib.util
Comment on lines +1 to +4
Copy link
Copy Markdown

@charliecreates charliecreates bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These imports are unused in this test module, which adds noise and can trip Python linting if it’s enabled in your CI/tooling.

Suggestion

Remove unused imports (sys, os) and keep only what’s needed.

import importlib.util

import pytest

Reply with "@CharlieHelps yes please" if you’d like me to add a commit with this cleanup.


spec = importlib.util.spec_from_file_location("compliance_monitor", "security/compliance-monitor.py")
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot Mar 31, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The module import uses a cwd-relative path; running pytest from a non-root working directory will fail to locate security/compliance-monitor.py. Resolve the path relative to this test file (and guard the spec/loader) to make the test robust.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At tests/test_compliance_monitor.py, line 6:

<comment>The module import uses a cwd-relative path; running pytest from a non-root working directory will fail to locate `security/compliance-monitor.py`. Resolve the path relative to this test file (and guard the spec/loader) to make the test robust.</comment>

<file context>
@@ -0,0 +1,66 @@
+import os
+import importlib.util
+
+spec = importlib.util.spec_from_file_location("compliance_monitor", "security/compliance-monitor.py")
+compliance_monitor = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(compliance_monitor)
</file context>
Fix with Cubic

compliance_monitor = importlib.util.module_from_spec(spec)
spec.loader.exec_module(compliance_monitor)
Comment on lines +6 to +8
Copy link
Copy Markdown

@charliecreates charliecreates bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The module path is resolved relative to the current working directory ("security/compliance-monitor.py"). This is brittle in CI and when running tests from different directories (e.g., pytest tests/ vs repo root) and can cause test collection/import to fail.

Suggestion

Resolve the module path relative to the test file and validate spec/loader before executing.

from pathlib import Path
import importlib.util

MODULE_PATH = Path(__file__).resolve().parents[1] / "security" / "compliance-monitor.py"

spec = importlib.util.spec_from_file_location("compliance_monitor", MODULE_PATH)
assert spec and spec.loader
compliance_monitor = importlib.util.module_from_spec(spec)
spec.loader.exec_module(compliance_monitor)

Reply with "@CharlieHelps yes please" if you’d like me to add a commit with this change.


def test_monitor_soc2_compliance_all_pass():
monitor = compliance_monitor.ComplianceMonitor()
result = monitor.monitor_soc2_compliance()

assert result["framework"] == "SOC2"
assert result["compliance_score"] == 100.0
assert result["status"] == "compliant"

checks = result["checks"]
assert checks["access_controls"] is True
assert checks["system_monitoring"] is True
assert checks["data_encryption"] is True
assert checks["backup_procedures"] is True
assert checks["incident_response"] is True

def test_monitor_soc2_compliance_partial_fail():
monitor = compliance_monitor.ComplianceMonitor()

monitor.check_access_controls = lambda: False
monitor.check_data_encryption = lambda: False

result = monitor.monitor_soc2_compliance()

assert result["framework"] == "SOC2"
assert result["compliance_score"] == 60.0
assert result["status"] == "non_compliant"

checks = result["checks"]
assert checks["access_controls"] is False
assert checks["system_monitoring"] is True
assert checks["data_encryption"] is False
assert checks["backup_procedures"] is True
assert checks["incident_response"] is True

Comment on lines +10 to +43
Copy link
Copy Markdown

@charliecreates charliecreates bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests partially rely on the real implementations of other check_* methods being stable and returning True (e.g., test_monitor_soc2_compliance_all_pass and the non-overridden checks in partial_fail). That can make the suite flaky if those checks start depending on environment state, I/O, or change behavior independently of monitor_soc2_compliance()’s scoring logic.

Suggestion

Make the tests deterministic by monkeypatching all check_* methods to known values per scenario (using pytest’s monkeypatch), so you’re only testing the aggregation/scoring logic.

def test_monitor_soc2_compliance_all_pass(monkeypatch):
    monitor = compliance_monitor.ComplianceMonitor()
    monkeypatch.setattr(monitor, "check_access_controls", lambda: True)
    monkeypatch.setattr(monitor, "check_system_monitoring", lambda: True)
    monkeypatch.setattr(monitor, "check_data_encryption", lambda: True)
    monkeypatch.setattr(monitor, "check_backup_procedures", lambda: True)
    monkeypatch.setattr(monitor, "check_incident_response", lambda: True)
    ...

Reply with "@CharlieHelps yes please" if you’d like me to add a commit applying this pattern to all scenarios.

def test_monitor_soc2_compliance_edge_cases():
monitor = compliance_monitor.ComplianceMonitor()

monitor.check_access_controls = lambda: False
monitor.check_system_monitoring = lambda: False
monitor.check_data_encryption = lambda: False
monitor.check_backup_procedures = lambda: False
monitor.check_incident_response = lambda: False

result = monitor.monitor_soc2_compliance()

assert result["compliance_score"] == 0.0
assert result["status"] == "non_compliant"

def test_monitor_soc2_compliance_just_below_threshold():
monitor = compliance_monitor.ComplianceMonitor()

monitor.check_access_controls = lambda: False

result = monitor.monitor_soc2_compliance()

assert result["compliance_score"] == 80.0
assert result["status"] == "non_compliant"
Comment on lines +15 to +66
Copy link
Copy Markdown

@charliecreates charliecreates bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assertions compare floating-point values with exact equality. If the score is computed via division (even with “nice” decimals), minor representation differences can cause brittle failures across Python versions/implementations.

Suggestion

Use pytest.approx(...) for compliance_score checks.

assert result["compliance_score"] == pytest.approx(60.0)

Reply with "@CharlieHelps yes please" if you’d like me to add a commit with this change.

Loading