diff --git a/.aider/conventions.md b/.aider/conventions.md index 9dff479..df5694e 100644 --- a/.aider/conventions.md +++ b/.aider/conventions.md @@ -4,6 +4,24 @@ These are the active engineering decisions for this repository. Aider should follow all of these conventions when making changes. +## Performance + +### Replace awk with sed in src/payment.ts bash scripts + +**Convention:** Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. + +**Why:** The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues. + +**Relevant files:** `src/payment.ts` + +## Mobile + +### Standardization on iPhones for mobile communication + +**Convention:** The team will use iPhones to perform mobile calls. + +**Why:** The team aligned on a single mobile device platform for communication consistency. + ## Validation ### Use RFC 307895 for JSON theme validation @@ -122,9 +140,9 @@ Aider should follow all of these conventions when making changes. ### Establish ownership and modification constraints for credits and billing system -**Convention:** Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. +**Convention:** Replace all usage of the double type for money representations with the string type in src/billing.ts. -**Why:** To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized. +**Why:** Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations. > ⚠️ This is a **HIGH** priority rule — do not violate it. @@ -132,9 +150,9 @@ Aider should follow all of these conventions when making changes. ### Ownership of Billing Module -**Convention:** The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. +**Convention:** Replace all usages of double with string to represent money transactions in src/billing.ts. -**Why:** This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance. +**Why:** Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations. **Relevant files:** `packages/api/src/billing/` @@ -277,13 +295,3 @@ Aider should follow all of these conventions when making changes. **Why:** Automating the verification of architectural decisions during the review process helps maintain consistency and ensures that developers adhere to established guidelines. > ⚠️ This is a **HIGH** priority rule — do not violate it. - -## Database - -### Standardize on PostgreSQL and Redis; Prohibit MongoDB - -**Convention:** MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. - -**Why:** ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements. - -> ⚠️ This is a **CRITICAL** priority rule — do not violate it. diff --git a/.clinerules b/.clinerules index 9ed355e..d3301ba 100644 --- a/.clinerules +++ b/.clinerules @@ -16,96 +16,100 @@ Before writing or modifying code, review these rules and do not violate them. 3. 🔴 **[CRITICAL]** MongoDB is strictly prohibited from being integrated into the current technology stack, including for analytics events. - **Why:** There is an active and non-negotiable architectural constraint against MongoDB in the stack due to the critical requirement for ACID compliance across all billing and user data. MongoDB does not satisfy this fundamental requirement. -4. 🔴 **[CRITICAL]** MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. - - **Why:** ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements. - -5. 🟠 **[HIGH]** Migrate the core decision storage from PostgreSQL to MongoDB, utilizing MongoDB Atlas for vector search capabilities. +4. 🟠 **[HIGH]** Migrate the core decision storage from PostgreSQL to MongoDB, utilizing MongoDB Atlas for vector search capabilities. - **Why:** MongoDB was chosen to accommodate high-write volume scenarios and to provide a flexible schema structure that the previous relational model failed to support efficiently. - **Files:** `context-store/`, `package.json` -6. 🟠 **[HIGH]** Migrate all email services to Zoho and update the SMTP server infrastructure, including the implementation of new routing rules to block any traffic to the legacy SMTP server. +5. 🟠 **[HIGH]** Migrate all email services to Zoho and update the SMTP server infrastructure, including the implementation of new routing rules to block any traffic to the legacy SMTP server. - **Why:** The team decided to move to Zoho to consolidate mailing services and address the limitations or overhead associated with the existing legacy SMTP infrastructure. - **Files:** `infrastructure/mail`, `services/smtp`, `config/email_routing` -7. 🟠 **[HIGH]** Switch from the third-party Shipsy provider to an in-house developed mapping event system. +6. 🟠 **[HIGH]** Switch from the third-party Shipsy provider to an in-house developed mapping event system. - **Why:** The team identified that the Shipsy service was negatively impacting the platform's scalability, and moving to an internal solution reduces external dependencies. - **Files:** `services/shipping-integration`, `infrastructure/event-bus` -8. 🟠 **[HIGH]** We have decided to officially discontinue the use of RFC 7807 (Problem Details for HTTP APIs) for all API error responses moving forward. +7. 🟠 **[HIGH]** We have decided to officially discontinue the use of RFC 7807 (Problem Details for HTTP APIs) for all API error responses moving forward. - **Why:** The team determined that the RFC 7807 specification is outdated and no longer aligns with the current requirements and standards of the API architecture. - **Files:** `api/responses`, `api/error-handling` -9. 🟠 **[HIGH]** Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. - - **Why:** To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized. +8. 🟠 **[HIGH]** Replace all usage of the double type for money representations with the string type in src/billing.ts. + - **Why:** Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations. - **Files:** `packages/api/src/routes/credits.ts`, `packages/decision-store/src/repositories/credit-repository.ts`, `packages/common/src/types/credits.ts` -10. 🟠 **[HIGH]** Establish explicit provider fallback orderings: For extraction, use Anthropic → DeepSeek → OpenAI. For detection, use Google → OpenAI → DeepSeek. +9. 🟠 **[HIGH]** Establish explicit provider fallback orderings: For extraction, use Anthropic → DeepSeek → OpenAI. For detection, use Google → OpenAI → DeepSeek. - **Why:** To maintain system reliability and avoid task failure when individual LLM providers hit rate limits, a hierarchical fallback mechanism ensures work is diverted to alternative models before resorting to the Dead Letter Queue (DLQ) after retries. - **Files:** `src/llm/client_factory.py`, `src/llm/fallback_logic.py` -11. 🟠 **[HIGH]** The team will migrate from AWS ECS to AWS EKS for container orchestration. +10. 🟠 **[HIGH]** The team will migrate from AWS ECS to AWS EKS for container orchestration. - **Why:** EKS provides superior orchestration flexibility, including native Horizontal Pod Autoscaler and improved multi-AZ/multi-region failover capabilities, which are necessary for the current scale, outweighing the operational overhead of Kubernetes. - **Files:** `infrastructure/terraform`, `infrastructure/k8s` -12. 🟠 **[HIGH]** Use PostgreSQL with pgvector and HNSW indexes as the standard solution for primary datastore and vector search operations. +11. 🟠 **[HIGH]** Use PostgreSQL with pgvector and HNSW indexes as the standard solution for primary datastore and vector search operations. - **Why:** PostgreSQL with pgvector provides the ability to manage both SQL-based relational data and vector search capabilities within a single system, simplifying the architecture compared to managing separate databases. -13. 🟠 **[HIGH]** Use MongoDB Atlas specifically for the analytics event ingestion pipeline, while keeping all other core application data in PostgreSQL. +12. 🟠 **[HIGH]** Use MongoDB Atlas specifically for the analytics event ingestion pipeline, while keeping all other core application data in PostgreSQL. - **Why:** MongoDB Atlas provides the necessary horizontal sharding and schemaless structure to handle the required 50k write operations per second, whereas PostgreSQL performance degrades under this load. - **Files:** `services/analytics-webhook-handler`, `infrastructure/database-clusters` -14. 🟠 **[HIGH]** We will use MongoDB for the analytics events pipeline, provisioning a MongoDB Atlas cluster to handle the data. +13. 🟠 **[HIGH]** We will use MongoDB for the analytics events pipeline, provisioning a MongoDB Atlas cluster to handle the data. - **Why:** MongoDB offers 10x the write throughput compared to PostgreSQL for high-cardinality event data, which is essential to meet the current scale requirements. The previous constraint was established before these new scale demands emerged. - **Files:** `packages/api/src/analytics/` -15. 🟠 **[HIGH]** Each LLM pipeline step (detection, extraction, formatting) has its own provider configuration managed via environment variables. An 'effort mode' concept allows overriding these configurations per company at request time, defining specific LLM models for different quality/cost tiers: Saver uses gemini-flash, Balanced mixes gemini-flash, claude-haiku, and gpt-4o-mini, Pro uses claude-sonnet for extraction, and Super uses claude-opus. +14. 🟠 **[HIGH]** Each LLM pipeline step (detection, extraction, formatting) has its own provider configuration managed via environment variables. An 'effort mode' concept allows overriding these configurations per company at request time, defining specific LLM models for different quality/cost tiers: Saver uses gemini-flash, Balanced mixes gemini-flash, claude-haiku, and gpt-4o-mini, Pro uses claude-sonnet for extraction, and Super uses claude-opus. - **Why:** The strategy is designed to provide flexibility and optimization across different pipeline steps and 'effort modes'. By configuring providers per step and allowing overrides based on company effort modes, the system can balance cost, performance, and model quality according to specific requirements, from 'Saver' (likely cost-optimized) to 'Super' (likely highest quality/cost). The multi-provider abstraction facilitates this dynamic selection. - **Files:** `packages/analyzer/` -16. 🟠 **[HIGH]** The trigger metric for initiating the AWS migration has been adjusted from 20 paying customers to 30 paying customers. The Q3 2026 timeline for the migration still holds. +15. 🟠 **[HIGH]** The trigger metric for initiating the AWS migration has been adjusted from 20 paying customers to 30 paying customers. The Q3 2026 timeline for the migration still holds. - **Why:** This adjustment is due to Railway costs being more predictable than initially expected. Additionally, the VPC isolation requirement, which was a significant factor, only applies to enterprise customers, a segment we are targeting at a later stage. -17. 🟠 **[HIGH]** We will integrate decision-guardian into our PR pipeline to enforce and track architectural decisions. +16. 🟠 **[HIGH]** We will integrate decision-guardian into our PR pipeline to enforce and track architectural decisions. - **Why:** Automating the verification of architectural decisions during the review process helps maintain consistency and ensures that developers adhere to established guidelines. -18. 🟠 **[HIGH]** The specific LLM model combinations for the multi-provider effort modes were finalized: Saver mode uses `gemini-flash` for detection, extraction, and format. Balanced mode uses `gemini-flash` for detection, `claude-haiku` for extraction, and `gpt-4o-mini` for format. Pro mode uses `gemini-flash` for detection, `claude-sonnet` for extraction, and `gpt-4o-mini` for format. Super mode uses `gemini-flash` for detection, `claude-opus` for extraction, and `claude-sonnet` for format. +17. 🟠 **[HIGH]** The specific LLM model combinations for the multi-provider effort modes were finalized: Saver mode uses `gemini-flash` for detection, extraction, and format. Balanced mode uses `gemini-flash` for detection, `claude-haiku` for extraction, and `gpt-4o-mini` for format. Pro mode uses `gemini-flash` for detection, `claude-sonnet` for extraction, and `gpt-4o-mini` for format. Super mode uses `gemini-flash` for detection, `claude-opus` for extraction, and `claude-sonnet` for format. - **Why:** The chosen LLM model combinations for each effort mode (Saver, Balanced, Pro, Super) were selected to provide different performance and cost profiles, aligning with the multi-provider strategy. Cost analysis confirmed that the proposed combinations, ranging from ~$0.08/1M tokens for Saver to ~$4.50/1M tokens for Super, ensure fine margins at current credit pricing. -19. 🟠 **[HIGH]** We will implement a multi-provider abstraction where each pipeline step (detection, extraction, enrichment, formatting) has its own LLM provider configuration via environment variables. At request time, an 'effort mode' can override the provider selection on a per-company basis. +18. 🟠 **[HIGH]** We will implement a multi-provider abstraction where each pipeline step (detection, extraction, enrichment, formatting) has its own LLM provider configuration via environment variables. At request time, an 'effort mode' can override the provider selection on a per-company basis. - **Why:** This approach allows companies with high context volume (Tier 3+) to pay extra for Claude-Sonnet's accuracy where needed, while companies with tighter budgets can use more cost-effective options like Gemini-Flash for all steps. It also decouples our infrastructure from individual LLM vendor stability and enables independent contract negotiations with different providers (Anthropic, OpenAI, Google). -20. 🟡 **[MEDIUM]** Use RFC 307895 as the standard for validating user input when adding custom themes through the form at /addCustomTheme. +19. 🟡 **[MEDIUM]** Use RFC 307895 as the standard for validating user input when adding custom themes through the form at /addCustomTheme. - **Why:** Adopting an existing RFC provides a standardized, well-documented approach to input validation, ensuring consistency and security for custom theme data handling. - **Files:** `assets/theme` -21. 🟡 **[MEDIUM]** All new vector indexes must be created using the HNSW algorithm. Existing IVFFlat indexes (specifically in the llm_cache table) are to be migrated to HNSW in Sprint 16. +20. 🟡 **[MEDIUM]** All new vector indexes must be created using the HNSW algorithm. Existing IVFFlat indexes (specifically in the llm_cache table) are to be migrated to HNSW in Sprint 16. - **Why:** HNSW is the current architectural standard for vector indexing. The previous rejection of the migration to HNSW was due to operational risks in production, not a lack of performance or technical suitability of HNSW. - **Files:** `db/schema/vector_indexes`, `db/migrations/sprint_16/migrate_llm_cache_to_hnsw` -22. 🟡 **[MEDIUM]** Adopt the HIGH severity specification as the authoritative version for the RFC 7807 error format, which includes fields: type, title, status, detail, and instance. +21. 🟡 **[MEDIUM]** Adopt the HIGH severity specification as the authoritative version for the RFC 7807 error format, which includes fields: type, title, status, detail, and instance. - **Why:** The team identified that two existing conventions were redundant. Designating the HIGH severity entry as canonical while allowing the fusion engine to merge duplicate references ensures consistency across documentation and API implementations. - **Files:** `packages/api/src/plugins/error-handler.ts` -23. 🟡 **[MEDIUM]** We have standardized on cosine distance (using the <=> operator in pgvector) for all similarity search operations. +22. 🟡 **[MEDIUM]** We have standardized on cosine distance (using the <=> operator in pgvector) for all similarity search operations. - **Why:** Cosine distance provides significantly better recall (12% improvement) on normalized text embeddings compared to L2 distance. Furthermore, L2 distance is overly sensitive to embedding magnitude, making it less reliable for our specific use case. -24. 🟡 **[MEDIUM]** MongoDB is strictly prohibited for use in core pipeline services (including the core decision pipeline, authentication, and the context store). These services must exclusively use PostgreSQL 16 and Redis. Any deviation requires a formal ADR. +23. 🟡 **[MEDIUM]** MongoDB is strictly prohibited for use in core pipeline services (including the core decision pipeline, authentication, and the context store). These services must exclusively use PostgreSQL 16 and Redis. Any deviation requires a formal ADR. - **Why:** To maintain architectural integrity and prevent fragmentation in the core tech stack. Previous attempts to introduce MongoDB for event queues nearly caused instability, highlighting the need for a hard, enforceable constraint. - **Files:** `analytics/storage`, `infrastructure/database-policy` -25. 🟡 **[MEDIUM]** The team decided to discontinue the use of EventStoreDB and removed event sourcing as an architectural pattern following the migration back to a monorepo. +24. 🟡 **[MEDIUM]** The team decided to discontinue the use of EventStoreDB and removed event sourcing as an architectural pattern following the migration back to a monorepo. - **Why:** The complexity of maintaining three separate runbooks for EventStoreDB operations outweighed the benefits of its auditability features for the current team size and system scale. -26. 🟡 **[MEDIUM]** All internal API routes must adhere to the RFC 7807 error format, consistent with public-facing API routes. +25. 🟡 **[MEDIUM]** All internal API routes must adhere to the RFC 7807 error format, consistent with public-facing API routes. - **Why:** Inconsistent error formats, specifically plain strings from internal routes, prevent AI tools from reliably parsing and analyzing errors, leading to broken analysis workflows. - **Files:** `packages/api/src/routes/internal/` -27. 🟡 **[MEDIUM]** We decided to use cosine distance for semantic similarity search of text embeddings with pgvector HNSW for deduplication. +26. 🟡 **[MEDIUM]** We decided to use cosine distance for semantic similarity search of text embeddings with pgvector HNSW for deduplication. - **Why:** Cosine distance is invariant to vector magnitude, meaning it only considers the direction of vectors. This property is precisely what is desired for semantic similarity of text embeddings, as it allows for accurate comparison of semantic meaning regardless of variations in embedding vector norms. L2 (Euclidean) distance, on the other hand, would incorrectly penalize vectors with different magnitudes, even if they share the same semantic direction. -28. 🟡 **[MEDIUM]** Implemented Redis semantic caching for LLM embedding calls. The cache key is a hash of the input text, model, and provider. The cache entries have a Time-To-Live (TTL) of 1 hour. +27. 🟡 **[MEDIUM]** Implemented Redis semantic caching for LLM embedding calls. The cache key is a hash of the input text, model, and provider. The cache entries have a Time-To-Live (TTL) of 1 hour. - **Why:** Redis was a natural extension since it is already in use for BullMQ and session caching. This implementation reduced redundant embedding calls by approximately 40% in tests. -29. 🟡 **[MEDIUM]** The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. - - **Why:** This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance. +28. 🟡 **[MEDIUM]** Replace all usages of double with string to represent money transactions in src/billing.ts. + - **Why:** Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations. - **Files:** `packages/api/src/billing/` + +29. 🟢 **[LOW]** Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. + - **Why:** The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues. + - **Files:** `src/payment.ts` + +30. 🟢 **[LOW]** The team will use iPhones to perform mobile calls. + - **Why:** The team aligned on a single mobile device platform for communication consistency. diff --git a/.cursorrules b/.cursorrules index 1b06f9e..636daff 100644 --- a/.cursorrules +++ b/.cursorrules @@ -1,6 +1,18 @@ # Project Decisions & Conventions # Auto-generated by Decispher — Do not edit manually +## Performance + +- Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. + Rationale: The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues. + Files: src/payment.ts + +## Mobile + +- The team will use iPhones to perform mobile calls. + Rationale: The team aligned on a single mobile device platform for communication consistency. + Files: * + ## Validation - Use RFC 307895 as the standard for validating user input when adding custom themes through the form at /addCustomTheme. @@ -61,12 +73,12 @@ ## Billing -- Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. - Rationale: To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized. +- Replace all usage of the double type for money representations with the string type in src/billing.ts. + Rationale: Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations. Files: packages/api/src/routes/credits.ts, packages/decision-store/src/repositories/credit-repository.ts, packages/common/src/types/credits.ts -- The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. - Rationale: This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance. +- Replace all usages of double with string to represent money transactions in src/billing.ts. + Rationale: Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations. Files: packages/api/src/billing/ ## Llm @@ -148,9 +160,3 @@ - We will integrate decision-guardian into our PR pipeline to enforce and track architectural decisions. Rationale: Automating the verification of architectural decisions during the review process helps maintain consistency and ensures that developers adhere to established guidelines. Files: * - -## Database - -- MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. - Rationale: ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements. - Files: * diff --git a/.decispher/context-rules.json b/.decispher/context-rules.json index bdc8dd1..4ad173d 100644 --- a/.decispher/context-rules.json +++ b/.decispher/context-rules.json @@ -1,28 +1,124 @@ { "$schema": "https://decispher.dev/schemas/context-rules/v1.json", "specVersion": "1.0.0", - "generatedAt": "2026-05-05T10:03:57.578Z", + "generatedAt": "2026-05-05T12:33:53.121Z", "companyId": "1489dcdc-ef7f-4cc5-b0cb-b453efa059f4", "meta": { - "totalRules": 28, + "totalRules": 29, "rulesByType": { - "decision": 19, + "decision": 21, "convention": 2, - "constraint": 3, + "constraint": 2, "rationale": 0, "ownership": 2, "history": 1, "plan": 1 }, "rulesBySeverity": { - "CRITICAL": 3, + "CRITICAL": 2, "HIGH": 15, "MEDIUM": 10, - "LOW": 0 + "LOW": 2 }, "generator": "decispher-ai-blocker@1.0.0" }, "rules": [ + { + "id": "0170331f-f276-46d6-a601-54093718039f", + "type": "decision", + "title": "Replace awk with sed in src/payment.ts bash scripts", + "problem": "Current awk commands used within the bash script in src/payment.ts are causing performance delays.", + "decision": "Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands.", + "rationale": "The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues.", + "severity": "LOW", + "status": "active", + "confidence": 0.48, + "affectedFiles": [ + "src/payment.ts" + ], + "tags": [ + "performance", + "bash", + "refactoring", + "scripting" + ], + "alternatives": [], + "sources": [ + { + "type": "slack", + "confidence": 0.48, + "ref": { + "priority": "high", + "threadTs": "1777983539.585479", + "channelId": "C0ALKBAGZQS", + "sourceUrl": "https://newworkspace-zdx9462.slack.com/archives/C0ALKBAGZQS/p1777983539585479?thread_ts=1777983539.585479&cid=C0ALKBAGZQS", + "channelName": "bot-test-1", + "slackTeamId": null, + "triggeredBy": "U05F9P78LTG", + "detectionMethod": "explicit", + "triggeredByName": "Ali Abbas" + }, + "snippet": "Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands." + } + ], + "supersededBy": null, + "enforcement": { + "blockOnViolation": false, + "requiresExplicitOverride": false, + "filePatterns": [ + "src/payment.ts" + ], + "exemptPatterns": [] + }, + "createdAt": "2026-05-05T12:19:32.050Z", + "updatedAt": "2026-05-05T12:20:51.371Z", + "version": 1, + "createdBy": "U05F9P78LTG", + "reviewedBy": "U05F9P78LTG" + }, + { + "id": "606e27a4-4b72-485e-94dd-7a2008e80874", + "type": "decision", + "title": "Standardization on iPhones for mobile communication", + "problem": null, + "decision": "The team will use iPhones to perform mobile calls.", + "rationale": "The team aligned on a single mobile device platform for communication consistency.", + "severity": "LOW", + "status": "active", + "confidence": 0.45, + "affectedFiles": [], + "tags": [ + "mobile", + "iphone", + "communication" + ], + "alternatives": [], + "sources": [ + { + "type": "slack", + "confidence": 0.45, + "ref": { + "priority": "high", + "threadTs": "1777983344.187829", + "channelId": "C0ALKBAGZQS", + "sourceUrl": "https://newworkspace-zdx9462.slack.com/archives/C0ALKBAGZQS/p1777983344187829?thread_ts=1777983344.187829&cid=C0ALKBAGZQS", + "channelName": "bot-test-1", + "slackTeamId": null, + "triggeredBy": "U05F9P78LTG", + "detectionMethod": "explicit", + "triggeredByName": "Ali Abbas" + }, + "snippet": "The team will use iPhones to perform mobile calls." + } + ], + "supersededBy": null, + "enforcement": null, + "createdAt": "2026-05-05T12:16:07.372Z", + "updatedAt": "2026-05-05T12:17:28.718Z", + "version": 1, + "createdBy": "U05F9P78LTG", + "reviewedBy": "U05F9P78LTG" + }, { "id": "1fbd69ae-f638-48bd-869d-cd54a993f0b0", "type": "decision", @@ -443,8 +539,8 @@ "type": "ownership", "title": "Establish ownership and modification constraints for credits and billing system", "problem": "Uncertainty regarding ownership of the billing module and the requirements for implementing new effort modes.", - "decision": "Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements.", - "rationale": "To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized.", + "decision": "Replace all usage of the double type for money representations with the string type in src/billing.ts.", + "rationale": "Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations.", "severity": "HIGH", "status": "active", "confidence": 0.43, @@ -479,6 +575,22 @@ "triggeredByName": null }, "snippet": "Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must st…" + }, + { + "type": "slack", + "confidence": 0.48, + "ref": { + "priority": "high", + "threadTs": "1777984088.950089", + "channelId": "C0ALKBAGZQS", + "sourceUrl": "https://newworkspace-zdx9462.slack.com/archives/C0ALKBAGZQS/p1777984088950089?thread_ts=1777984088.950089&cid=C0ALKBAGZQS", + "channelName": "bot-test-1", + "slackTeamId": null, + "triggeredBy": "U05F9P78LTG", + "detectionMethod": "explicit", + "triggeredByName": "Ali Abbas" + }, + "snippet": "Replace all usage of the double type for money representations with the string type in src/billing.ts." } ], "supersededBy": null, @@ -493,8 +605,8 @@ "exemptPatterns": [] }, "createdAt": "2026-04-22T06:14:04.546Z", - "updatedAt": "2026-04-22T11:43:40.979Z", - "version": 1, + "updatedAt": "2026-05-05T12:33:12.799Z", + "version": 2, "createdBy": "U_SARA", "reviewedBy": "df51fb32-47a2-48b2-ac32-3887a582a966" }, @@ -1408,8 +1520,8 @@ "type": "ownership", "title": "Ownership of Billing Module", "problem": null, - "decision": "The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review.", - "rationale": "This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance.", + "decision": "Replace all usages of double with string to represent money transactions in src/billing.ts.", + "rationale": "Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations.", "severity": "MEDIUM", "status": "active", "confidence": 0.38, @@ -1434,74 +1546,38 @@ "triggeredBy": "U05F9P78LTG", "detectionMethod": "explicit" } - } - ], - "supersededBy": null, - "enforcement": { - "blockOnViolation": false, - "requiresExplicitOverride": false, - "filePatterns": [ - "packages/api/src/billing/" - ], - "exemptPatterns": [] - }, - "createdAt": "2026-04-18T13:24:50.857Z", - "updatedAt": "2026-04-18T13:25:43.024Z", - "version": 1, - "createdBy": "U05F9P78LTG", - "reviewedBy": "U05F9P78LTG" - }, - { - "id": "3998d8e6-48a3-4161-860b-be094e4feb87", - "type": "constraint", - "title": "Standardize on PostgreSQL and Redis; Prohibit MongoDB", - "problem": "Ensure ACID compliance for critical billing and user data, and standardize data storage technologies to maintain data integrity and consistency.", - "decision": "MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes.", - "rationale": "ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements.", - "severity": "CRITICAL", - "status": "active", - "confidence": 0.74, - "affectedFiles": [], - "tags": [ - "database", - "postgresql", - "redis", - "mongodb", - "data-storage", - "acid" - ], - "alternatives": [ - { - "option": "MongoDB", - "reasonRejected": "MongoDB was rejected because it does not provide the necessary ACID compliance required for critical billing and user data, which is a non-negotiable architectural requirement for data integrity." - } - ], - "sources": [ + }, { "type": "slack", - "confidence": 0.5, + "confidence": 0.48, "ref": { "priority": "high", - "threadTs": "1776518248.102319", - "channelId": "C0ANM3QAQMN", - "channelName": "decispher-live-test", - "triggeredBy": "U05F9P78LTG", - "detectionMethod": "explicit" - } + "threadTs": "1777984088.950089", + "channelId": "C0ALKBAGZQS", + "sourceUrl": null, + "channelName": null, + "slackTeamId": null, + "triggeredBy": null, + "detectionMethod": "explicit", + "triggeredByName": null + }, + "snippet": "Replace all usages of double with string to represent money transactions in src/billing.ts." } ], "supersededBy": null, "enforcement": { "blockOnViolation": false, "requiresExplicitOverride": false, - "filePatterns": [], + "filePatterns": [ + "packages/api/src/billing/" + ], "exemptPatterns": [] }, - "createdAt": "2026-04-18T13:17:48.935Z", - "updatedAt": "2026-04-18T23:32:02.232Z", - "version": 1, + "createdAt": "2026-04-18T13:24:50.857Z", + "updatedAt": "2026-05-05T12:33:52.890Z", + "version": 2, "createdBy": "U05F9P78LTG", - "reviewedBy": "U05F9P78LTG" + "reviewedBy": "df51fb32-47a2-48b2-ac32-3887a582a966" }, { "id": "b6869b8c-7d43-47d4-9cd6-dddb0f9f92b9", diff --git a/.decispher/decisions.md b/.decispher/decisions.md index 3e3b20e..6033240 100644 --- a/.decispher/decisions.md +++ b/.decispher/decisions.md @@ -1,3 +1,62 @@ + +## Decision: Replace awk with sed in src/payment.ts bash scripts + +**Status**: Active +**Date**: 2026-05-05 +**Severity**: Info + +**Files**: +- `src/payment.ts` + +**Rules**: +```json +{ + "conditions": [ + { + "type": "file", + "pattern": "src/payment.ts", + "content_rules": [ + { + "mode": "string", + "patterns": [ + "awk" + ] + } + ] + } + ], + "match_mode": "all" +} +``` + +### Context + +**Problem:** Current awk commands used within the bash script in src/payment.ts are causing performance delays. + +**Decision:** Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. + +**Rationale:** The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues. + +--- + + +## Decision: Standardization on iPhones for mobile communication + +**Status**: Active +**Date**: 2026-05-05 +**Severity**: Info + +**Files**: +- `**/*` + +### Context + +**Decision:** The team will use iPhones to perform mobile calls. + +**Rationale:** The team aligned on a single mobile device platform for communication consistency. + +--- + ## Decision: Use RFC 307895 for JSON theme validation @@ -295,10 +354,11 @@ "content_rules": [ { "mode": "regex", - "start": 1, - "pattern": "(credit_ledger|DrizzleCreditRepository|EFFORT_MODE_CONFIGS)" + "start": 0, + "pattern": "number|double" } - ] + ], + "content_match_mode": "any" } ], "match_mode": "all" @@ -309,9 +369,9 @@ **Problem:** Uncertainty regarding ownership of the billing module and the requirements for implementing new effort modes. -**Decision:** Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. +**Decision:** Replace all usage of the double type for money representations with the string type in src/billing.ts. -**Rationale:** To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized. +**Rationale:** Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations. --- @@ -1013,82 +1073,25 @@ "conditions": [ { "type": "file", - "pattern": "packages/api/src/billing/**", + "pattern": "packages/api/src/billing/**/*.ts", "content_rules": [ { - "mode": "string", - "patterns": [ - "stripe", - "credit", - "ledger", - "invoice", - "payment", - "billing" - ] - } - ], - "content_match_mode": "any" - } - ], - "match_mode": "any" -} -``` - -### Context - -**Decision:** The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. - -**Rationale:** This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance. - ---- - - -## Decision: Standardize on PostgreSQL and Redis; Prohibit MongoDB - -**Status**: Active -**Date**: 2026-04-18 -**Severity**: Critical - -**Rules**: -```json -{ - "conditions": [ - { - "type": "file", - "exclude": [ - "**/node_modules/**", - "**/vendor/**", - "**/*.lock", - "**/*.md" - ], - "pattern": "**/*", - "content_rules": [ - { - "mode": "string", - "patterns": [ - "mongodb", - "mongo", - "mongoose" - ] + "mode": "regex", + "start": 0, + "pattern": "number\\s*[:=]\\s*double|double\\s+\\w+" } - ], - "content_match_mode": "any" + ] } ], - "match_mode": "any" + "match_mode": "all" } ``` ### Context -**Problem:** Ensure ACID compliance for critical billing and user data, and standardize data storage technologies to maintain data integrity and consistency. +**Decision:** Replace all usages of double with string to represent money transactions in src/billing.ts. -**Decision:** MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. - -**Rationale:** ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements. - -**Alternatives Considered:** -- **MongoDB**: MongoDB was rejected because it does not provide the necessary ACID compliance required for critical billing and user data, which is a non-negotiable architectural requirement for data integrity. +**Rationale:** Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations. --- diff --git a/.devin/rules.md b/.devin/rules.md index 1554e5d..2115f19 100644 --- a/.devin/rules.md +++ b/.devin/rules.md @@ -36,16 +36,6 @@ Devin MUST follow all rules below. Do not deviate without explicit instruction. **Alternatives considered (rejected):** - ~~MongoDB for analytics events~~ — It violates an active architectural constraint due to its lack of native ACID compliance, which is non-negotiable for billing and user data within our stack. -### Standardize on PostgreSQL and Redis; Prohibit MongoDB - -- **Severity:** CRITICAL -- **Rule:** MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. -- **Rationale:** ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements. -- **Tags:** database, postgresql, redis, mongodb, data-storage, acid - -**Alternatives considered (rejected):** -- ~~MongoDB~~ — MongoDB was rejected because it does not provide the necessary ACID compliance required for critical billing and user data, which is a non-negotiable architectural requirement for data integrity. - ### Migrate core decision store from PostgreSQL to MongoDB - **Severity:** HIGH @@ -87,8 +77,8 @@ Devin MUST follow all rules below. Do not deviate without explicit instruction. ### Establish ownership and modification constraints for credits and billing system - **Severity:** HIGH -- **Rule:** Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. -- **Rationale:** To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized. +- **Rule:** Replace all usage of the double type for money representations with the string type in src/billing.ts. +- **Rationale:** Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations. - **Scope:** packages/api/src/routes/credits.ts, packages/decision-store/src/repositories/credit-repository.ts, packages/common/src/types/credits.ts - **Tags:** billing, ownership, credits, compliance @@ -277,7 +267,22 @@ Devin MUST follow all rules below. Do not deviate without explicit instruction. ### Ownership of Billing Module - **Severity:** MEDIUM -- **Rule:** The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. -- **Rationale:** This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance. +- **Rule:** Replace all usages of double with string to represent money transactions in src/billing.ts. +- **Rationale:** Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations. - **Scope:** packages/api/src/billing/ - **Tags:** billing, ownership, team + +### Replace awk with sed in src/payment.ts bash scripts + +- **Severity:** LOW +- **Rule:** Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. +- **Rationale:** The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues. +- **Scope:** src/payment.ts +- **Tags:** performance, bash, refactoring, scripting + +### Standardization on iPhones for mobile communication + +- **Severity:** LOW +- **Rule:** The team will use iPhones to perform mobile calls. +- **Rationale:** The team aligned on a single mobile device platform for communication consistency. +- **Tags:** mobile, iphone, communication diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 6cd654d..6d8afa1 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -3,32 +3,33 @@ When generating code for this project, follow these rules: -1. Use RFC 307895 as the standard for validating user input when adding custom themes through the form at /addCustomTheme. -2. Migrate the core decision storage from PostgreSQL to MongoDB, utilizing MongoDB Atlas for vector search capabilities. -3. Migrate all email services to Zoho and update the SMTP server infrastructure, including the implementation of new routing rules to block any traffic to the legacy SMTP server. -4. Switch from the third-party Shipsy provider to an in-house developed mapping event system. -5. We have decided to officially discontinue the use of RFC 7807 (Problem Details for HTTP APIs) for all API error responses moving forward. -6. All new vector indexes must be created using the HNSW algorithm. Existing IVFFlat indexes (specifically in the llm_cache table) are to be migrated to HNSW in Sprint 16. -7. Adopt the HIGH severity specification as the authoritative version for the RFC 7807 error format, which includes fields: type, title, status, detail, and instance. -8. Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. -9. Establish explicit provider fallback orderings: For extraction, use Anthropic → DeepSeek → OpenAI. For detection, use Google → OpenAI → DeepSeek. -10. The team will migrate from AWS ECS to AWS EKS for container orchestration. -11. We have standardized on cosine distance (using the <=> operator in pgvector) for all similarity search operations. -12. MongoDB is strictly prohibited for use in core pipeline services (including the core decision pipeline, authentication, and the context store). These services must exclusively use PostgreSQL 16 and Redis. Any deviation requires a formal ADR. -13. The core pipeline must exclusively use PostgreSQL 16 with pgvector and Redis; the use of MongoDB is strictly prohibited. -14. Use PostgreSQL with pgvector and HNSW indexes as the standard solution for primary datastore and vector search operations. -15. Use MongoDB Atlas specifically for the analytics event ingestion pipeline, while keeping all other core application data in PostgreSQL. -16. The team decided to discontinue the use of EventStoreDB and removed event sourcing as an architectural pattern following the migration back to a monorepo. -17. All internal API routes must adhere to the RFC 7807 error format, consistent with public-facing API routes. -18. We will use MongoDB for the analytics events pipeline, provisioning a MongoDB Atlas cluster to handle the data. -19. Each LLM pipeline step (detection, extraction, formatting) has its own provider configuration managed via environment variables. An 'effort mode' concept allows overriding these configurations per company at request time, defining specific LLM models for different quality/cost tiers: Saver uses gemini-flash, Balanced mixes gemini-flash, claude-haiku, and gpt-4o-mini, Pro uses claude-sonnet for extraction, and Super uses claude-opus. -20. After evaluating MongoDB, DynamoDB, and PostgreSQL, we chose PostgreSQL 16 with pgvector HNSW indexes. Reason: vector similarity search, ACID guarantees, and single DB for both structured data and embeddings. -21. We decided to use cosine distance for semantic similarity search of text embeddings with pgvector HNSW for deduplication. -22. Implemented Redis semantic caching for LLM embedding calls. The cache key is a hash of the input text, model, and provider. The cache entries have a Time-To-Live (TTL) of 1 hour. -23. MongoDB is strictly prohibited from being integrated into the current technology stack, including for analytics events. -24. The trigger metric for initiating the AWS migration has been adjusted from 20 paying customers to 30 paying customers. The Q3 2026 timeline for the migration still holds. -25. We will integrate decision-guardian into our PR pipeline to enforce and track architectural decisions. -26. The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. -27. MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. -28. The specific LLM model combinations for the multi-provider effort modes were finalized: Saver mode uses `gemini-flash` for detection, extraction, and format. Balanced mode uses `gemini-flash` for detection, `claude-haiku` for extraction, and `gpt-4o-mini` for format. Pro mode uses `gemini-flash` for detection, `claude-sonnet` for extraction, and `gpt-4o-mini` for format. Super mode uses `gemini-flash` for detection, `claude-opus` for extraction, and `claude-sonnet` for format. -29. We will implement a multi-provider abstraction where each pipeline step (detection, extraction, enrichment, formatting) has its own LLM provider configuration via environment variables. At request time, an 'effort mode' can override the provider selection on a per-company basis. +1. Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. +2. The team will use iPhones to perform mobile calls. +3. Use RFC 307895 as the standard for validating user input when adding custom themes through the form at /addCustomTheme. +4. Migrate the core decision storage from PostgreSQL to MongoDB, utilizing MongoDB Atlas for vector search capabilities. +5. Migrate all email services to Zoho and update the SMTP server infrastructure, including the implementation of new routing rules to block any traffic to the legacy SMTP server. +6. Switch from the third-party Shipsy provider to an in-house developed mapping event system. +7. We have decided to officially discontinue the use of RFC 7807 (Problem Details for HTTP APIs) for all API error responses moving forward. +8. All new vector indexes must be created using the HNSW algorithm. Existing IVFFlat indexes (specifically in the llm_cache table) are to be migrated to HNSW in Sprint 16. +9. Adopt the HIGH severity specification as the authoritative version for the RFC 7807 error format, which includes fields: type, title, status, detail, and instance. +10. Replace all usage of the double type for money representations with the string type in src/billing.ts. +11. Establish explicit provider fallback orderings: For extraction, use Anthropic → DeepSeek → OpenAI. For detection, use Google → OpenAI → DeepSeek. +12. The team will migrate from AWS ECS to AWS EKS for container orchestration. +13. We have standardized on cosine distance (using the <=> operator in pgvector) for all similarity search operations. +14. MongoDB is strictly prohibited for use in core pipeline services (including the core decision pipeline, authentication, and the context store). These services must exclusively use PostgreSQL 16 and Redis. Any deviation requires a formal ADR. +15. The core pipeline must exclusively use PostgreSQL 16 with pgvector and Redis; the use of MongoDB is strictly prohibited. +16. Use PostgreSQL with pgvector and HNSW indexes as the standard solution for primary datastore and vector search operations. +17. Use MongoDB Atlas specifically for the analytics event ingestion pipeline, while keeping all other core application data in PostgreSQL. +18. The team decided to discontinue the use of EventStoreDB and removed event sourcing as an architectural pattern following the migration back to a monorepo. +19. All internal API routes must adhere to the RFC 7807 error format, consistent with public-facing API routes. +20. We will use MongoDB for the analytics events pipeline, provisioning a MongoDB Atlas cluster to handle the data. +21. Each LLM pipeline step (detection, extraction, formatting) has its own provider configuration managed via environment variables. An 'effort mode' concept allows overriding these configurations per company at request time, defining specific LLM models for different quality/cost tiers: Saver uses gemini-flash, Balanced mixes gemini-flash, claude-haiku, and gpt-4o-mini, Pro uses claude-sonnet for extraction, and Super uses claude-opus. +22. After evaluating MongoDB, DynamoDB, and PostgreSQL, we chose PostgreSQL 16 with pgvector HNSW indexes. Reason: vector similarity search, ACID guarantees, and single DB for both structured data and embeddings. +23. We decided to use cosine distance for semantic similarity search of text embeddings with pgvector HNSW for deduplication. +24. Implemented Redis semantic caching for LLM embedding calls. The cache key is a hash of the input text, model, and provider. The cache entries have a Time-To-Live (TTL) of 1 hour. +25. MongoDB is strictly prohibited from being integrated into the current technology stack, including for analytics events. +26. The trigger metric for initiating the AWS migration has been adjusted from 20 paying customers to 30 paying customers. The Q3 2026 timeline for the migration still holds. +27. We will integrate decision-guardian into our PR pipeline to enforce and track architectural decisions. +28. Replace all usages of double with string to represent money transactions in src/billing.ts. +29. The specific LLM model combinations for the multi-provider effort modes were finalized: Saver mode uses `gemini-flash` for detection, extraction, and format. Balanced mode uses `gemini-flash` for detection, `claude-haiku` for extraction, and `gpt-4o-mini` for format. Pro mode uses `gemini-flash` for detection, `claude-sonnet` for extraction, and `gpt-4o-mini` for format. Super mode uses `gemini-flash` for detection, `claude-opus` for extraction, and `claude-sonnet` for format. +30. We will implement a multi-provider abstraction where each pipeline step (detection, extraction, enrichment, formatting) has its own LLM provider configuration via environment variables. At request time, an 'effort mode' can override the provider selection on a per-company basis. diff --git a/.roo/rules.md b/.roo/rules.md index 6d596c2..3b44832 100644 --- a/.roo/rules.md +++ b/.roo/rules.md @@ -19,8 +19,8 @@ *(The team determined that the RFC 7807 specification is outdated and no longer aligns with the current requirements and standards of the API architecture.)* Files: api/responses, api/error-handling -- **Establish ownership and modification constraints for credits and billing system:** Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. - *(To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized.)* +- **Establish ownership and modification constraints for credits and billing system:** Replace all usage of the double type for money representations with the string type in src/billing.ts. + *(Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations.)* Files: packages/api/src/routes/credits.ts, packages/decision-store/src/repositories/credit-repository.ts, packages/common/src/types/credits.ts - **Define Model Fallback Ordering Strategy for API Rate Limits:** Establish explicit provider fallback orderings: For extraction, use Anthropic → DeepSeek → OpenAI. For detection, use Google → OpenAI → DeepSeek. @@ -62,9 +62,6 @@ - **Defer Microservices Adoption, Maintain Monorepo Architecture:** We will integrate decision-guardian into our PR pipeline to enforce and track architectural decisions. *(Automating the verification of architectural decisions during the review process helps maintain consistency and ensures that developers adhere to established guidelines.)* -- **Standardize on PostgreSQL and Redis; Prohibit MongoDB:** MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. - *(ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements.)* - - **Define LLM Model Combinations for Saver, Balanced, Pro, and Super Effort Modes:** The specific LLM model combinations for the multi-provider effort modes were finalized: Saver mode uses `gemini-flash` for detection, extraction, and format. Balanced mode uses `gemini-flash` for detection, `claude-haiku` for extraction, and `gpt-4o-mini` for format. Pro mode uses `gemini-flash` for detection, `claude-sonnet` for extraction, and `gpt-4o-mini` for format. Super mode uses `gemini-flash` for detection, `claude-opus` for extraction, and `claude-sonnet` for format. *(The chosen LLM model combinations for each effort mode (Saver, Balanced, Pro, Super) were selected to provide different performance and cost profiles, aligning with the multi-provider strategy. Cost analysis confirmed that the proposed combinations, ranging from ~$0.08/1M tokens for Saver to ~$4.50/1M tokens for Super, ensure fine margins at current credit pricing.)* @@ -73,6 +70,12 @@ ## General Conventions +- **Replace awk with sed in src/payment.ts bash scripts:** Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. + *(The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues.)* + +- **Standardization on iPhones for mobile communication:** The team will use iPhones to perform mobile calls. + *(The team aligned on a single mobile device platform for communication consistency.)* + - **Use RFC 307895 for JSON theme validation:** Use RFC 307895 as the standard for validating user input when adding custom themes through the form at /addCustomTheme. *(Adopting an existing RFC provides a standardized, well-documented approach to input validation, ensuring consistency and security for custom theme data handling.)* @@ -100,5 +103,5 @@ - **Implement Redis Semantic Caching for LLM Embedding Calls:** Implemented Redis semantic caching for LLM embedding calls. The cache key is a hash of the input text, model, and provider. The cache entries have a Time-To-Live (TTL) of 1 hour. *(Redis was a natural extension since it is already in use for BullMQ and session caching. This implementation reduced redundant embedding calls by approximately 40% in tests.)* -- **Ownership of Billing Module:** The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. - *(This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance.)* +- **Ownership of Billing Module:** Replace all usages of double with string to represent money transactions in src/billing.ts. + *(Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations.)* diff --git a/.windsurfrules b/.windsurfrules index b009c79..4c1a2c9 100644 --- a/.windsurfrules +++ b/.windsurfrules @@ -17,9 +17,6 @@ Cascade MUST respect these rules when generating or modifying code. - MongoDB is strictly prohibited from being integrated into the current technology stack, including for analytics events. Reason: There is an active and non-negotiable architectural constraint against MongoDB in the stack due to the critical requirement for ACID compliance across all billing and user data. MongoDB does not satisfy this fundamental requirement. -- MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. - Reason: ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements. - ## [HIGH] - Migrate the core decision storage from PostgreSQL to MongoDB, utilizing MongoDB Atlas for vector search capabilities. @@ -38,8 +35,8 @@ Cascade MUST respect these rules when generating or modifying code. Reason: The team determined that the RFC 7807 specification is outdated and no longer aligns with the current requirements and standards of the API architecture. Files: api/responses, api/error-handling -- Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. - Reason: To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized. +- Replace all usage of the double type for money representations with the string type in src/billing.ts. + Reason: Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations. Files: packages/api/src/routes/credits.ts, packages/decision-store/src/repositories/credit-repository.ts, packages/common/src/types/credits.ts - Establish explicit provider fallback orderings: For extraction, use Anthropic → DeepSeek → OpenAI. For detection, use Google → OpenAI → DeepSeek. @@ -111,6 +108,15 @@ Cascade MUST respect these rules when generating or modifying code. - Implemented Redis semantic caching for LLM embedding calls. The cache key is a hash of the input text, model, and provider. The cache entries have a Time-To-Live (TTL) of 1 hour. Reason: Redis was a natural extension since it is already in use for BullMQ and session caching. This implementation reduced redundant embedding calls by approximately 40% in tests. -- The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. - Reason: This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance. +- Replace all usages of double with string to represent money transactions in src/billing.ts. + Reason: Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations. Files: packages/api/src/billing/ + +## [LOW] + +- Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. + Reason: The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues. + Files: src/payment.ts + +- The team will use iPhones to perform mobile calls. + Reason: The team aligned on a single mobile device platform for communication consistency. diff --git a/AGENTS.md b/AGENTS.md index a6de045..b2d4c8f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,6 @@ Any code change that violates these decisions MUST be flagged. | The core pipeline must exclusively use PostgreSQL 16 with pgvector and Redis; the use of MongoDB is strictly prohibited. | Enforcing a specific database stack ensures architectural consistency, simplifies maintenance, and leverages existing infrastructure and expertise with PostgreSQL and pgvector. | infrastructure/database, src/db/config.ts | | After evaluating MongoDB, DynamoDB, and PostgreSQL, we chose PostgreSQL 16 with pgvector HNSW indexes. Reason: vector similarity search, ACID guarantees, and single DB for both structured data and embeddings. | After evaluating MongoDB, DynamoDB, and PostgreSQL, we chose PostgreSQL 16 with pgvector HNSW indexes. Reason: vector similarity search, ACID guarantees, and single DB for both structured data and embeddings. | * | | MongoDB is strictly prohibited from being integrated into the current technology stack, including for analytics events. | There is an active and non-negotiable architectural constraint against MongoDB in the stack due to the critical requirement for ACID compliance across all billing and user data. MongoDB does not satisfy this fundamental requirement. | * | -| MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. | ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements. | * | ### HIGH @@ -23,7 +22,7 @@ Any code change that violates these decisions MUST be flagged. | Migrate all email services to Zoho and update the SMTP server infrastructure, including the implementation of new routing rules to block any traffic to the legacy SMTP server. | The team decided to move to Zoho to consolidate mailing services and address the limitations or overhead associated with the existing legacy SMTP infrastructure. | infrastructure/mail, services/smtp, config/email_routing | | Switch from the third-party Shipsy provider to an in-house developed mapping event system. | The team identified that the Shipsy service was negatively impacting the platform's scalability, and moving to an internal solution reduces external dependencies. | services/shipping-integration, infrastructure/event-bus | | We have decided to officially discontinue the use of RFC 7807 (Problem Details for HTTP APIs) for all API error responses moving forward. | The team determined that the RFC 7807 specification is outdated and no longer aligns with the current requirements and standards of the API architecture. | api/responses, api/error-handling | -| Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. | To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized. | packages/api/src/routes/credits.ts, packages/decision-store/src/repositories/credit-repository.ts, packages/common/src/types/credits.ts | +| Replace all usage of the double type for money representations with the string type in src/billing.ts. | Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations. | packages/api/src/routes/credits.ts, packages/decision-store/src/repositories/credit-repository.ts, packages/common/src/types/credits.ts | | Establish explicit provider fallback orderings: For extraction, use Anthropic → DeepSeek → OpenAI. For detection, use Google → OpenAI → DeepSeek. | To maintain system reliability and avoid task failure when individual LLM providers hit rate limits, a hierarchical fallback mechanism ensures work is diverted to alternative models before resorting to the Dead Letter Queue (DLQ) after retries. | src/llm/client_factory.py, src/llm/fallback_logic.py | | The team will migrate from AWS ECS to AWS EKS for container orchestration. | EKS provides superior orchestration flexibility, including native Horizontal Pod Autoscaler and improved multi-AZ/multi-region failover capabilities, which are necessary for the current scale, outweighing the operational overhead of Kubernetes. | infrastructure/terraform, infrastructure/k8s | | Use PostgreSQL with pgvector and HNSW indexes as the standard solution for primary datastore and vector search operations. | PostgreSQL with pgvector provides the ability to manage both SQL-based relational data and vector search capabilities within a single system, simplifying the architecture compared to managing separate databases. | * | @@ -48,4 +47,11 @@ Any code change that violates these decisions MUST be flagged. | All internal API routes must adhere to the RFC 7807 error format, consistent with public-facing API routes. | Inconsistent error formats, specifically plain strings from internal routes, prevent AI tools from reliably parsing and analyzing errors, leading to broken analysis workflows. | packages/api/src/routes/internal/ | | We decided to use cosine distance for semantic similarity search of text embeddings with pgvector HNSW for deduplication. | Cosine distance is invariant to vector magnitude, meaning it only considers the direction of vectors. This property is precisely what is desired for semantic similarity of text embeddings, as it allows for accurate comparison of semantic meaning regardless of variations in embedding vector norms. L2 (Euclidean) distance, on the other hand, would incorrectly penalize vectors with different magnitudes, even if they share the same semantic direction. | * | | Implemented Redis semantic caching for LLM embedding calls. The cache key is a hash of the input text, model, and provider. The cache entries have a Time-To-Live (TTL) of 1 hour. | Redis was a natural extension since it is already in use for BullMQ and session caching. This implementation reduced redundant embedding calls by approximately 40% in tests. | * | -| The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. | This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance. | packages/api/src/billing/ | +| Replace all usages of double with string to represent money transactions in src/billing.ts. | Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations. | packages/api/src/billing/ | + +### LOW + +| Decision | Rationale | Files | +|----------|-----------|-------| +| Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. | The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues. | src/payment.ts | +| The team will use iPhones to perform mobile calls. | The team aligned on a single mobile device platform for communication consistency. | * | diff --git a/CLAUDE.md b/CLAUDE.md index bf29da2..d9821aa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,6 +8,15 @@ Violating these decisions requires explicit approval. ## Active Decisions +### Replace awk with sed in src/payment.ts bash scripts (LOW) +- **Decision:** Remove all awk commands from the bash script in src/payment.ts and replace them with functionally equivalent sed commands. +- **Rationale:** The team identified that the existing awk implementations are causing unnecessary performance bottlenecks; transitioning to sed is expected to resolve these latency issues. +- **Affected files:** `src/payment.ts` + +### Standardization on iPhones for mobile communication (LOW) +- **Decision:** The team will use iPhones to perform mobile calls. +- **Rationale:** The team aligned on a single mobile device platform for communication consistency. + ### Use RFC 307895 for JSON theme validation (MEDIUM) - **Decision:** Use RFC 307895 as the standard for validating user input when adding custom themes through the form at /addCustomTheme. - **Rationale:** Adopting an existing RFC provides a standardized, well-documented approach to input validation, ensuring consistency and security for custom theme data handling. @@ -48,8 +57,8 @@ Violating these decisions requires explicit approval. - **Do NOT:** MEDIUM severity specification (The HIGH severity version was explicitly selected as the authoritative and canonical standard.) ### Establish ownership and modification constraints for credits and billing system (HIGH) -- **Decision:** Sara is the primary owner of the billing module; all changes to the credit_ledger schema, DrizzleCreditRepository, and the EFFORT_MODE_CONFIGS require specific approvals from Sara and Ali. Furthermore, the system must strictly adhere to the append-only ledger constraint per ADR-019 and maintain SERIALIZABLE transaction requirements. -- **Rationale:** To ensure accountability and maintain architectural integrity of the financial ledger and billing configuration, specific code ownership and structural constraints have been formalized. +- **Decision:** Replace all usage of the double type for money representations with the string type in src/billing.ts. +- **Rationale:** Using floating-point numbers (doubles) for currency leads to rounding errors and precision issues due to IEEE 754 binary representation. Using strings ensures that exact decimal precision is maintained during financial calculations. - **Affected files:** `packages/api/src/routes/credits.ts`, `packages/decision-store/src/repositories/credit-repository.ts`, `packages/common/src/types/credits.ts` ### Define Model Fallback Ordering Strategy for API Rate Limits (HIGH) @@ -142,15 +151,10 @@ Violating these decisions requires explicit approval. - **Do NOT:** Adopt a microservices architecture by splitting recorder and analyzer into separate gRPC services. (The previous attempt in Phase 1 led to brutal deployment complexity for a 3-person team, consuming 40% of their time debugging inter-service authentication and network failures.) ### Ownership of Billing Module (MEDIUM) -- **Decision:** The billing module, including Stripe integration, credit ledger, credit deduction logic, and Stripe webhook handlers, is owned by U05F9P78LTG. All changes to billing flows require their review. -- **Rationale:** This statement clarifies responsibility for the billing module and its components to ensure proper review and maintenance. +- **Decision:** Replace all usages of double with string to represent money transactions in src/billing.ts. +- **Rationale:** Using string types for monetary values prevents floating-point arithmetic errors inherent in the double type, ensuring accuracy for financial calculations. - **Affected files:** `packages/api/src/billing/` -### Standardize on PostgreSQL and Redis; Prohibit MongoDB (CRITICAL) -- **Decision:** MongoDB is strictly prohibited in this stack due to its lack of ACID compliance. PostgreSQL will be used as the primary datastore for all persistent data, especially critical billing and user data. Redis will be used exclusively for caching purposes. -- **Rationale:** ACID compliance is a non-negotiable requirement for billing and user data to guarantee data integrity and consistency. PostgreSQL provides robust ACID transaction support. Adopting a standardized approach with PostgreSQL and Redis simplifies the technology stack and enforces critical data integrity requirements. -- **Do NOT:** MongoDB (MongoDB was rejected because it does not provide the necessary ACID compliance required for critical billing and user data, which is a non-negotiable architectural requirement for data integrity.) - ### Define LLM Model Combinations for Saver, Balanced, Pro, and Super Effort Modes (HIGH) - **Decision:** The specific LLM model combinations for the multi-provider effort modes were finalized: Saver mode uses `gemini-flash` for detection, extraction, and format. Balanced mode uses `gemini-flash` for detection, `claude-haiku` for extraction, and `gpt-4o-mini` for format. Pro mode uses `gemini-flash` for detection, `claude-sonnet` for extraction, and `gpt-4o-mini` for format. Super mode uses `gemini-flash` for detection, `claude-opus` for extraction, and `claude-sonnet` for format. - **Rationale:** The chosen LLM model combinations for each effort mode (Saver, Balanced, Pro, Super) were selected to provide different performance and cost profiles, aligning with the multi-provider strategy. Cost analysis confirmed that the proposed combinations, ranging from ~$0.08/1M tokens for Saver to ~$4.50/1M tokens for Super, ensure fine margins at current credit pricing.