Audit phase 5h#7
Open
Patrick010 wants to merge 574 commits into
Open
Conversation
This commit includes the newly compiled snitch.exe executable. The underlying Go source code was fixed to use http.GET instead of http.POST, but a build issue was preventing the change from being reflected in the binary. This build was performed after cleaning the Go build and module caches, and it has been internally verified to start correctly. This commit is to allow the user to pull the new binary and perform their own verification before the issue is marked as fully resolved.
This commit refactors the `snitch` Go module to resolve a fundamental structural conflict that was causing the build to fail. The previous implementation had a `package main` file (`snitch.go`) in the module root, which conflicted with the intended entry point at `cmd/snitch/main.go`. This caused the build to use old, incorrect code. This change replaces the old `snitch.go` with a new version that acts as a proper library (`package snitch`). This new file contains the necessary application logic but correctly uses the new `GET`-based login handler from the `internal/listener` package. The conflicting `main` function has been removed. This should allow the application to be built correctly from the `cmd/snitch/main.go` entry point.
This commit radically simplifies the `snitch` application to resolve a persistent and un-debuggable build issue. The previous multi-file structure with `cmd/` and `internal/` directories was causing build conflicts. This has been replaced with a single, self-contained `snitch.go` file that holds the entire application. This new file is a `package main`, uses the correct `http.Get` logic for the API callback, and removes all the structural complexity that was leading to the build failures.
This commit adds validation to the `snitch` application to ensure the `SNITCH_API_CALLBACK_URL` environment variable is a full URL. The application will now check if the URL starts with `http://` or `https://` on startup and will exit with a clear error message if it does not. This prevents runtime errors caused by incorrect configuration.
This commit fixes a `TypeError` in the `/api/auth/spotify/callback` endpoint. The code was attempting to `await resp.json()`, but the `resp` object's `json()` method was not awaitable in the runtime environment, causing an exception. This commit removes the `await` keyword to correctly parse the JSON response.
This commit updates all relevant project documentation to reflect the resolution of the `snitch` build issue and the subsequent API `TypeError`. - `ACTIVITY.md`, `CURRENT_STATE.md`, and `AUDIT-PHASE-4.md` have been updated with the full history and current status of the fixes. - `snitch/README.md` and `snitch/docs/USER_MANUAL.md` have been updated to reflect the new, simplified single-file structure of the `snitch` application.
This commit brings all project documentation up to date. - `ACTIVITY.md`, `CURRENT_STATE.md`, and `AUDIT-PHASE-4.md` have been updated with the full history and current status of the `snitch` and API fixes. - The documentation within the `snitch` module (`README.md`, `docs/ARCHITECTURE.md`, `docs/INSTALLATION.md`, `docs/MODULES.md`) has been completely overhauled to reflect the new, simplified single-file architecture and build process.
This commit fixes a bug where the API server would fail to start if not run from the repository root. The paths to `logging_framework.yml` were hardcoded as `api/logging_framework.yml`, which caused a `FileNotFoundError` when the server was started from within the `api/` directory (as the `start.sh` script does). This change corrects the paths to be relative (`logging_framework.yml`), allowing the server to start correctly.
This commit fixes a `FileNotFoundError` that occurred when running the API server locally. 1. `api/logging_framework.yml`: The hardcoded absolute path (`/app/api/logs/debug.log`), which was only valid in a specific Docker environment, has been changed to a relative path (`logs/debug.log`). 2. `scripts/start.sh`: The startup script has been updated to create the `api/logs` directory, ensuring the relative path for the log file is valid on startup.
This commit refactors the logging framework initialization to make the log file path configurable via an environment variable, removing hardcoded paths. - `api/logging_framework.yml`: The `path` key has been removed from the file sink definition. - `api/src/zotify_api/main.py`: The application now reads the `LOG_FILE_PATH` environment variable (defaulting to `logs/debug.log`) and dynamically injects it into the logging configuration at runtime. This makes the logging system more flexible and robust to different environments, per user feedback.
This commit refactors the logging framework initialization to more robustly handle configurable log paths, following user feedback on design principles. - `api/logging_framework.yml`: The `path` key has been removed from the default file sink definition. - `api/src/zotify_api/main.py`: The application now injects a default log path (from the `LOG_FILE_PATH` environment variable) *only* if a file sink is defined without an explicit `path`. This preserves the flexibility to define multiple file sinks with their own hardcoded paths if desired.
This commit implements a more robust and flexible way of handling security-related log messages, based on user feedback. - `api/logging_framework.yml`: The trigger mechanism has been changed from a specific event name to a generic `security` tag. This allows any log event with this tag to be routed to the `security_log`. - `api/src/zotify_api/routes/auth.py`: The error handling for Spotify token exchange failures has been updated to use `log_event` and apply the `security` tag. This decouples the logging of the event from its routing.
This commit delivers the final, correct implementation of the Flexible Logging Framework, incorporating user feedback on security, flexibility, and design principles.
Key features and fixes:
- **Tag-Based Routing:** The framework now supports routing logs to specific sinks based on tags (e.g., a 'security' tag). The service and schemas have been updated to support this more flexible design.
- **Configurable File Paths:** Log file paths are now fully configurable via environment variables with sensible defaults (e.g., `${DEBUG_LOG_PATH:-logs/debug.log}`). The YAML loader now expands these variables at runtime.
- **Security Redaction:** A `SensitiveDataFilter` is now conditionally applied in production environments (`APP_ENV=production`) to redact sensitive tokens and codes from all log messages, preventing data leakage. Both the Python API and the `snitch` helper app have been updated.
- **Structured Security Events:** The Spotify authentication callback now uses `log_event` to generate a structured, tagged security event on failure.
This commit reverts the logging framework's file path configuration to be purely YAML-based, per user feedback. - The logic for expanding environment variables in `main.py` has been removed. - The paths for file sinks in `logging_framework.yml` are now defined as hardcoded relative paths. This simplifies the configuration and removes the need for environment variables to define log locations.
This is a major documentation overhaul to bring all project and module-level documentation into alignment with the final, implemented state of the `snitch` application and the Flexible Logging Framework. This commit updates the following documents to reflect new features like tag-based routing, security redaction, and simplified application architecture: - High-Level Design (HLD) - Low-Level Design (LLD) - Project Initiation Document (PID) - Traceability Matrix - All `snitch` module documentation - The `LOGGING_GUIDE.md`
This commit fixes a server startup crash caused by an incorrect Pydantic validator. The `@root_validator` is deprecated in Pydantic v2 and was being used incorrectly. This has been replaced with the new, recommended `@model_validator(mode='before')` to ensure compatibility and correct behavior.
This commit adds a `log_event` call to the Spotify callback handler to log successful authentication events. This ensures that the `security.log` contains a complete audit trail of both successful and failed login attempts, per user feedback.
This commit adds a `log_event` call to the Spotify callback handler to log successful authentication events. This ensures that the `security.log` contains a complete audit trail of both successful and failed login attempts, per user feedback.
This commit fixes a bug where successful authentication events were being logged twice to the security log. The original `log_event` call for successful logins did not specify primary destinations, causing it to be broadcast to all sinks by default, in addition to being routed by the tag-based trigger. This has been fixed by specifying the default destinations in the `log_event` call, preventing the broadcast and removing the duplicate log entry.
This commit adds a new section to the `LOGGING_GUIDE.md` to explicitly explain the concept of custom tag-based routing. It provides a clear, step-by-step example of how a developer can invent a new tag and how an administrator can configure a new sink and trigger in the YAML to create a custom log stream. This addresses user feedback that the flexibility of the tagging system was not clear.
This commit secures the current state of the project by updating the main log files. - `ACTIVITY.md` has been updated with a comprehensive entry (`ACT-034`) detailing the resolution of the `snitch` regression and the subsequent hardening of the logging framework. - `AUDIT-PHASE-4.md` has been updated with a final report summarizing the work of the entire session.
This commit secures the current state of the project by updating the main log files. - `ACTIVITY.md` has been updated with a comprehensive entry (`ACT-034`) detailing the resolution of the `snitch` regression and the subsequent hardening of the logging framework. - `AUDIT-PHASE-4.md` has been updated with a final report summarizing the work of the entire session.
This commit secures the current state of the project by updating the main log files. - `ACTIVITY.md` has been updated with a comprehensive entry (`ACT-034`) detailing the resolution of the `snitch` regression and the subsequent hardening of the logging framework. - `AUDIT-PHASE-4.md` has been updated with a final report summarizing the work of the entire session.
This commit introduces a new feature proposal and integrates it into the project's documentation. - A new document, `project/DYNAMIC_PLUGIN_PROPOSAL.md`, has been created to formally propose a dynamic plugin system for the Flexible Logging Framework. - The `PROJECT_REGISTRY`, `FUTURE_ENHANCEMENTS`, `TRACEABILITY_MATRIX`, `PID`, `HLD`, and `LLD` have all been updated to reference and trace this new proposal.
This commit updates the `DYNAMIC_PLUGIN_PROPOSAL.md` to include a critical 'Security Considerations' section, based on user feedback. The new section outlines the risks of a dynamic plugin system and proposes a multi-layered mitigation strategy, including administrator warnings, safe loading practices, and a future plan for cryptographic signing of plugins.
This commit introduces a new feature proposal for a dynamic plugin architecture and integrates it deeply into the project's high-level documentation. - **New Proposal:** Adds `project/DYNAMIC_PLUGIN_PROPOSAL.md` to outline a system for extensible logging sinks using `entry_points`. - **Security:** The proposal includes a 'Security Considerations' section to address the risks of such a system. - **Architectural Vision:** The proposal and the HLD/LLD have been updated to position this plugin system as the long-term successor to the current Provider Abstraction Layer. - **Traceability:** The `PROJECT_REGISTRY`, `FUTURE_ENHANCEMENTS`, `TRACEABILITY_MATRIX`, and `PID` have all been updated to trace this new strategic feature.
This commit represents a major documentation effort to capture the new strategic vision for a plugin-based architecture and other future enhancements. - **New Proposal:** Adds `project/DYNAMIC_PLUGIN_PROPOSAL.md` to formally propose a dynamic plugin system for the API, including security considerations. - **Architectural Vision:** The proposal, `HLD.md`, and `LLD.md` have been updated to position the plugin system as the long-term successor to the current Provider Abstraction Layer. - **Future Enhancements:** The `FUTURE_ENHANCEMENTS.md` document has been updated to include the plugin system and a new vision for Low-Code/No-Code platform integration (e.g., Node-RED). - **Traceability:** The `PROJECT_REGISTRY`, `TRACEABILITY_MATRIX`, and `PID` have all been updated to trace this new strategic work.
This commit adds two new formal proposals and integrates them, along with the dynamic plugin proposal, into all high-level project documentation. - **New Proposals:** Adds `LOW_CODE_PROPOSAL.md` and `HOME_AUTOMATION_PROPOSAL.md` to outline the strategic vision for platform integration. - **Architectural Vision:** The plugin proposal was updated to be the designated successor to the Provider Abstraction Layer. - **Traceability:** All proposals have been fully integrated and traced in the `PROJECT_REGISTRY`, `FUTURE_ENHANCEMENTS`, `TRACEABILITY_MATRIX`, `PID`, `HLD`, and `LLD`. - **Final Logs:** `ACTIVITY.md` and `AUDIT-PHASE-4.md` have been updated to provide a final summary of the session's work.
This large commit represents a full documentation overhaul session. It includes two major, user-directed tasks: a complete cleanup of the project archive and the creation of a new, execution-oriented project plan.
Key changes include:
1. **Archive Cleanup & Consolidation:**
- Deleted over 20 obsolete or inaccurate documents from the `project/archive/` directory.
- Migrated valuable historical information from archived files into the current `CHANGELOG.md`, `SYSTEM_INTEGRATION_GUIDE.md`, and `SECURITY.md`.
- Addressed a feature gap for GDPR privacy endpoints by designing them in the HLD/LLD and adding them to the backlog and traceability matrix.
2. **New Project Plan:**
- Created `project/PROJECT_PLAN.md` as a central execution reference.
- The plan synthesizes information from the roadmap, PID, and backlog to provide a clear overview of milestones, modules, and tasks.
- Updated `PROJECT_REGISTRY.md` to include the new plan.
3. **Process Improvements:**
- Updated `AGENTS.md` to clarify the manual execution policy for the `log-work.py` script.
- Retroactively updated all Trinity logs to ensure full and accurate reporting for the session's work.
This large commit represents a full documentation overhaul session. It includes two major, user-directed tasks: a complete cleanup of the project archive and the creation of a new, execution-oriented project plan.
Key changes include:
1. **Archive Cleanup & Consolidation:**
- Deleted over 20 obsolete or inaccurate documents from the `project/archive/` directory.
- Migrated valuable historical information from archived files into the current `CHANGELOG.md`, `SYSTEM_INTEGRATION_GUIDE.md`, and `SECURITY.md`.
- Addressed a feature gap for GDPR privacy endpoints by designing them in the HLD/LLD and adding them to the backlog and traceability matrix.
2. **New Project Plan:**
- Created `project/PROJECT_PLAN.md` as a central execution reference.
- The plan synthesizes information from the roadmap, PID, and backlog to provide a clear overview of milestones, modules, and tasks.
- Updated `PROJECT_REGISTRY.md` to include the new plan.
3. **Process Improvements:**
- Updated `AGENTS.md` to clarify the manual execution policy for the `log-work.py` script.
- Retroactively updated all Trinity logs to ensure full and accurate reporting for the session's work.
This large commit represents a full documentation overhaul and planning session.
Key changes include:
1. **Archive Cleanup & Consolidation:**
- Deleted over 20 obsolete documents from the `project/archive/` directory.
- Migrated valuable historical information into the current `CHANGELOG.md`, `SYSTEM_INTEGRATION_GUIDE.md`, and `SECURITY.md`.
- Addressed a feature gap for GDPR privacy endpoints by designing them in the HLD/LLD and adding them to the backlog.
2. **Snitch Project Plan:**
- Created `snitch/docs/PROJECT_PLAN.md` as a central execution reference for the module.
- Updated `project/PID.md` and `project/PROJECT_REGISTRY.md` to integrate the new plan.
3. **Process Improvements:**
- Updated `AGENTS.md` to clarify the manual execution policy for `log-work.py`.
- Retroactively updated all Trinity logs to ensure full and accurate reporting for the session's work.
Create the initial `project/LOOSE_ENDS_BACKLOG.md` and the `project/process/GAP_ANALYSIS_TEMPLATE.md` as specified in the handover brief. In addition, this commit refines the project's documentation process to prevent future errors. It clarifies the distinction between project-level and API-level documentation and updates the `lint-docs.py` script to enforce this new, clearer standard. - Create `project/LOOSE_ENDS_BACKLOG.md` - Create `project/process/GAP_ANALYSIS_TEMPLATE.md` - Update `project/PROJECT_REGISTRY.md` with the new files. - Update `AGENTS.md` with clearer instructions for file registration. - Update `scripts/lint-docs.py` to check for project file registration in the Project Registry.
Mark the "Gap Analysis Framework (Template + Skeleton)" item as Done, as the required template was created in a previous task.
Mark the "Gap Analysis Framework (Template + Skeleton)" item as Done, as the required template was created in a previous task.
This commit resolves all items in the temporary `LOOSE_ENDS_BACKLOG.md` and then deletes the file. - Updates the Snitch project plan with new security milestones. - Rewrites the main project Roadmap for clarity and accuracy. - Fleshes out the Privacy Compliance documentation and adds cross-links. - Deletes the temporary backlog file now that all items are closed. Additionally, this commit refines the project's documentation process to prevent future errors. It clarifies the distinction between project-level and API-level documentation in AGENTS.md and updates the `lint-docs.py` script to enforce this new standard.
Creates a new traceability matrix in TRACEABILITY_MATRIX.md to map the strategic roadmap to the operational execution plan. This new matrix makes any drift between the two documents immediately obvious by explicitly mapping roadmap themes to execution plan phases and flagging any gaps or unmapped items. This resolves the drift identified between the two documents.
Aligns the strategic ROADMAP.md with the operational EXECUTION_PLAN.md, using the TRACEABILITY_MATRIX.md as the authoritative bridge. - Adds missing execution phases (Admin, Release Readiness) as themes to the roadmap. - Adds missing roadmap deliverables (Extensibility, Plugins) as tasks to the execution plan. - Updates the traceability matrix to reflect the new 1:1 alignment, resolving all gaps. - Updates the audit log to mark the reconciliation as complete.
This commit performs two corrective actions: 1. Updates ROADMAP.md to mark Phase 5 as complete, following the successful closure of the LOOSE_ENDS_BACKLOG.md task. 2. Corrects the project logs by adding a missing entry for the traceability matrix alignment task and removing a duplicate entry.
Adds the missing log file updates for the documentation alignment task. This corrects the previous incomplete commit.
This commit makes final corrections to project documentation and tooling. - Updates ROADMAP.md to mark Phase 5 as complete. - Fixes a bug in the `scripts/log-work.py` that prevented it from updating all three trinity log files. - Corrects the log files by removing a duplicate entry and running the fixed script to generate the correct state.
This commit addresses several process and tooling issues. - Fixes a bug in `scripts/log-work.py` that prevented it from updating `CURRENT_STATE.md` and `SESSION_LOG.md`. - Adds a unit test to verify the logging framework's ability to create log directories. - Corrects the project logs by removing duplicate entries and generating the missing ones. - Updates `ROADMAP.md` to mark Phase 5 as complete.
Moves the `endpoints.yaml` file from `api/docs/` to `project/api/` to better reflect its project-wide scope. All references to the file in the documentation have been updated accordingly.
Creates a new living `ALIGNMENT_MATRIX.md` file to serve as the authoritative mapping between design, code, and documentation. Adds a new rule to the `doc-lint-rules.yml` to enforce that this matrix is updated whenever relevant source code or scripts are changed. This locks the new matrix into the project's "living documentation" workflow.
Fleshes out the new `ALIGNMENT_MATRIX.md` with accurate, up-to-date data, mapping HLD, LLD, code, and documentation for all major components. Enforces maintenance of the matrix by adding a new rule to `doc-lint-rules.yml`. This ensures the matrix remains a living document.
Key changes include: - Added a new 'Audit Ref' column to the matrix for traceability. - Populated the matrix with granular details, linking audit requirements to specific sections in the HLD and LLD. - Inserted markdown anchors into HIGH_LEVEL_DESIGN.md and LOW_LEVEL_DESIGN.md to enable direct linking. - Corrected an outdated path in the 'doc-lint-rules.yml' to ensure the linter correctly references the alignment matrix. This provides a robust, traceable link between audit requirements and the design documentation.
Following a detailed user review, the previous matrix was found to be incomplete. This new version is the result of a systematic, ground-up analysis of the entire project. Key activities performed: - A full file-system audit of the `api`, `snitch`, `gonk-testUI`, and `scripts` modules was conducted to list all code components. - The HLD, LLD, and Audit Traceability Matrix were reviewed to extract all documented features and architectural layers. - A new, comprehensive matrix was constructed that maps every identified component to its corresponding design, code, and documentation artifacts. - A new, sequential `AR-XXX` audit reference was created for each item to ensure clear traceability. The resulting `ALIGNMENT_MATRIX.md` provides a complete, gap-free, and verifiable view of the entire project, fulfilling the user's requirement for an exhaustive traceability artifact.
… streamline the pre-submission verification process. Key changes: - A new script, `scripts/linter.sh`, has been created to act as a single entrypoint for all verification checks. - This script conditionally runs `pytest` only when source code files (`.py`, `.go`) have been changed. - It conditionally runs `mkdocs build` only when documentation files under `api/docs/` have been changed. - The documentation cross-reference linter (`lint-docs.py`) is always run to ensure code-doc alignment. - The `AGENTS.md` file has been updated to instruct agents to use this new unified script. This change optimizes the verification workflow by avoiding unnecessary checks, saving time and compute resources.
…single, unified, and intelligent Python script. Key changes: - A new script, `scripts/linter.py`, has been created to act as a single entrypoint for all verification checks. - The logic from the old `lint-docs.py` has been moved into this new script. - The new script conditionally runs `pytest` for code changes and `mkdocs build` for documentation changes, optimizing the verification process. - The old `lint-docs.py` and `run_lint.sh` scripts have been deleted. - The `AGENTS.md` file has been updated to instruct agents to use this new unified script. This change fully unifies the linting and verification logic into a single Python script, as per the user's request, making the process more efficient and easier to maintain.
…-linter` job was attempting to run `lint-docs.py`, a script that was deleted in a previous refactoring. The CI configuration in `.github/workflows/ci.yml` has been updated to call the new unified linter script, `scripts/linter.py`. This aligns the CI process with the current state of the project's tooling.
The `doc-linter` CI job was failing with a `ModuleNotFoundError` because the new unified linter script (`scripts/linter.py`) has a dependency on the `PyYAML` package, which was not installed in the CI environment. This commit adds a step to the `doc-linter` job in the `ci.yml` workflow to explicitly install `PyYAML` before the linter is run. This also corrects the project log files, which were corrupted by a previous, failed attempt to log this change.
This commit addresses two issues: 1. **Fix CI `doc-linter` job:** The CI was failing with a `ModuleNotFoundError` because the new unified `scripts/linter.py` script requires the `PyYAML` package. This change adds a step to the `doc-linter` job in `.github/workflows/ci.yml` to install this dependency. 2. **Restore Corrupted Log Files:** During a previous attempt to fix a logging issue, the `project/logs/SESSION_LOG.md` file was accidentally truncated, losing most of its history. This change includes the restored version of this file, preserving the project's history. The `project/logs/ACTIVITY.md` file was also corrected to remove a duplicate entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.