letsconfuse · letsconfuse · Jul 21, 2026 · Jul 21, 2026
diff --git a/Makefile b/Makefile
@@ -1,4 +1,4 @@
-SHELL := /usr/bin/env bash
+SHELL := bash
 CLUSTER_NAME ?= astronomy-shop
 NAMESPACE ?= astronomy-shop
 RELEASE_NAME ?= astronomy-shop

diff --git a/docs/adr/0005-observability-strategy.md b/docs/adr/0005-observability-strategy.md
@@ -0,0 +1,46 @@
+# 5. Observability and Alerting Strategy
+
+Date: 2026-07-21
+
+## Status
+
+Accepted
+
+## Context
+
+With over 15+ independent microservices operating in our local Kubernetes cluster, diagnosing service failures, networking latencies, and transaction bottlenecks cannot be handled using localized container logging alone. We require:
+1.  **Unified Telemetry Collection:** A single, standardized pipeline to ingest and route metrics and distributed traces.
+2.  **Telemetry Storage & Querying:** High-performance, scalable backends to store metrics and trace files.
+3.  **Visualization:** A single-pane-of-glass dashboard to visualize service graphs, latency spikes, and system health.
+4.  **Proactive Alerting:** A declarative alert definition and routing engine to notify administrators of failures or degradations.
+
+## Decision
+
+We will implement a complete local observability stack integrated directly with the OpenTelemetry Astronomy Shop.
+
+```mermaid
+flowchart TD
+  Services[Microservices 15+] -->|Metrics & Traces| OTelCol[OpenTelemetry Collector]
+  OTelCol -->|Metrics Exporter| Prom[Prometheus Server]
+  OTelCol -->|Traces Exporter| Jaeger[Jaeger Query/Collector]
+  Prom -->|Triggers| AM[Alertmanager]
+  Prom -->|Data Source| Grafana[Grafana Dashboard]
+  Jaeger -->|Data Source| Grafana
+```
+
+1.  **OpenTelemetry Collector as Telemetry Gateway:** The OTel Collector will receive all metrics and traces from the application services, process them, and export them to backend datastores. This prevents application services from coupling directly to specific vendor backends.
+2.  **Prometheus for Metrics & Alerting:** We will enable the Prometheus sub-chart to collect and query metrics.
+3.  **Alertmanager Integration:** We will enable the Alertmanager sub-chart within Prometheus to manage alerting lifecycle. We will configure declarative alerting rules at the Helm level:
+    -   `ServiceTargetDown`: Availability alert using Prometheus standard scrape targets metrics.
+    -   `OtelCollectorDroppedSpans`: Pipeline health alert looking at collector processor drop rate.
+    -   `FrontendHighHttpErrorRate`: Application SLA/performance alert monitoring Frontend proxy HTTP 5xx responses.
+4.  **Jaeger for Distributed Tracing:** Tracing spans collected by the OTel Collector will be exported to Jaeger for end-to-end distributed transaction tracking.
+5.  **Grafana for Dashboards:** Grafana will query metrics from Prometheus and traces from Jaeger to display real-time analytics.
+
+## Consequences
+
+*   **Positive:** Standardized observability stack built on open standards (OpenTelemetry), preventing vendor lock-in.
+*   **Positive:** Proactive system monitoring through automated alerting rules.
+*   **Positive:** Correlation between metric spikes (Prometheus) and transaction trace graphs (Jaeger) within Grafana.
+*   **Negative:** Significant cluster resource overhead. Running the full observability suite (Collector, Prometheus, Alertmanager, Jaeger, Grafana) increases Kind cluster memory usage by approximately 500MiB–800MiB.
+*   **Neutral:** In a production cloud setting, local Prometheus/Jaeger storage would be replaced by cloud-managed services (e.g. AWS Managed Prometheus, AWS X-Ray, Datadog), but the OTel Collector pipeline configurations and alerting rules would remain largely identical.
diff --git a/docs/journey.md b/docs/journey.md
@@ -112,7 +112,24 @@ Using Argo CD's Multiple Sources feature is a powerful way to reference upstream
 
 ---
 
+## 2026-07-21
+
+### Observability and Alerting Stack
+
+Completed:
+
+* Enabled Alertmanager sub-chart in Prometheus values configuration in `platform/helm/values.yaml`.
+* Defined declarative Prometheus alerting rules for availability (`ServiceTargetDown`), OTel pipeline integrity (`OtelCollectorDroppedSpans`), and application SLA (`FrontendHighHttpErrorRate`).
+* Authored ADR-0005 documenting the Observability and Alerting architecture.
+* Fixed cross-platform SHELL compatibility in the `Makefile` for Termux and standard Linux.
+
+Lesson learned:
+
+Defining alerts declaratively at the infrastructure configuration layer (Helm values) ensures that monitoring policies are versioned alongside the deployment resources, maintaining GitOps alignment for our alerting configurations.
+
+---
+
 ## Next Objective
 
-Push the `feature/phase4-gitops` branch, run validation and security checks, and merge the Pull Request.
+Push the `feature/phase5-observability` branch, verify Helm values rendering, and merge the Pull Request.
 
diff --git a/docs/roadmap.md b/docs/roadmap.md
@@ -100,33 +100,32 @@ Completed
 
 ## Phase 6 — GitOps
 
-Status: 🚧 In Progress
+Status: ✅ Completed
 
 Completed
 
 * GitOps engine comparative analysis (Argo CD vs Flux CD - ADR-0004)
 * Declarative Argo CD bootstrap automation (`install-argocd.sh`)
 * Native Multiple Sources configuration to separate upstream chart and local values config (`application.yaml`)
 * Makefile integration (`argocd-bootstrap`, `argocd-status`, `argocd-port-forward`, `argocd-down`)
-
-Pending
-
-* Progressive delivery
-* Rollbacks and automated self-healing validation
+* Validation and merging of Phase 4 into main
 
 ---
 
 ## Phase 7 — Observability
 
-Planned
+Status: 🚧 In Progress
+
+Completed
+
+* Integrated and enabled Alertmanager sub-chart in Prometheus configuration
+* Defined declarative Prometheus alerting rules for container target availability, pipeline throughput, and frontend SLAs
+* Authored ADR-0005 documenting the Observability and Alerting architecture
+
+Pending
 
-* Prometheus
-* Grafana
-* Loki
-* Tempo
-* Jaeger
-* OpenTelemetry
-* Alertmanager
+* Verification of alert state transitions and metrics collection
+* Advanced log aggregation (Loki) and distributed tracing (Tempo) analysis
 
 ---
 
@@ -221,11 +220,11 @@ By the end of this repository the following skills should be demonstrated.
 
 Current Branch
 
-feature/phase4-gitops
+feature/phase5-observability
 
 Current Objective
 
-Verify declarative Argo CD bootstrapping, test multiple source Application sync, configure port-forwarding targets, and submit a Pull Request.
+Verify Prometheus alerting rule configuration, test Alertmanager container rollout, and submit a Pull Request.
 
 ---
 

diff --git a/platform/helm/values.yaml b/platform/helm/values.yaml
@@ -21,3 +21,43 @@ opentelemetry-collector:
     limits:
       cpu: 500m
       memory: 512Mi
+
+# Enable Alertmanager and configure alerting rules for our microservices.
+prometheus:
+  alertmanager:
+    enabled: true
+  serverFiles:
+    alerting_rules.yml:
+      groups:
+        - name: platform-alerts
+          rules:
+            # Alert when a Prometheus target is unreachable
+            - alert: ServiceTargetDown
+              expr: up == 0
+              for: 1m
+              labels:
+                severity: critical
+              annotations:
+                summary: "Service target is down"
+                description: "Prometheus target {{ $labels.instance }} has been unreachable for more than 1 minute."
+
+            # Alert when OpenTelemetry Collector begins dropping spans
+            - alert: OtelCollectorDroppedSpans
+              expr: rate(otelcol_processor_dropped_spans[2m]) > 0
+              for: 1m
+              labels:
+                severity: warning
+              annotations:
+                summary: "OTel Collector is dropping telemetry spans"
+                description: "The OpenTelemetry Collector processor {{ $labels.processor }} in node {{ $labels.node }} is dropping spans."
+
+            # Alert when HTTP 5xx responses from the Frontend service spike
+            - alert: FrontendHighHttpErrorRate
+              expr: sum(rate(http_server_duration_count{http_status=~"5.."}[5m])) / sum(rate(http_server_duration_count[5m])) * 100 > 5
+              for: 2m
+              labels:
+                severity: critical
+              annotations:
+                summary: "Frontend high HTTP 5xx error rate"
+                description: "Frontend service HTTP 5xx error rate is {{ printf \"%.2f\" $value }}% (threshold: 5%)."
+