Running-microservices metric should drop to zero after a CID's microservices are shut down

[allister-beamable]

Summary

The daily running-microservices snapshot (microservice_state collection in the wizard_master database) never records a zero for a CID whose microservices have all been shut down. Once a CID drops to zero running instances, its most recent snapshot document persists unchanged indefinitely, so every downstream consumer (Portal's microservice usage charts, internal usage/revenue reporting) keeps reporting the last nonzero count.

The snapshot job runs daily (05:00 UTC).

Expected

When all of a CID's microservices are scaled to zero, its running-microservices count should read zero from that point forward. Usage charts and any period averages that include the post-shutdown days should reflect the drop.

Actual

The CID's snapshot stops updating at its last nonzero value and stays frozen there. Charts and reports show a phantom count equal to the pre-shutdown value for as long as the CID remains off. Monthly/period averages that span the shutdown stay elevated instead of trending toward zero.

How it manifests

• A customer offboarded over a month ago (all microservices scaled to 0, verified 0 running tasks in ECS) still appears on the usage chart at its pre-shutdown service count.
• Cross-referencing the chart against ECS -- the ground truth for running tasks -- shows 0 running while the chart shows the stale nonzero number.
• From the collection alone there is no way to distinguish "still running at N" from "shut down, last seen at N."

Impact

• Usage and revenue reporting overstates active microservice counts for offboarded or idled CIDs.
• Offboarding verification cannot rely on the chart; it silently disagrees with reality, and the disagreement is indefinite rather than lagging by a day.
• Period averages are biased upward for any CID that dropped to zero mid-period.

[allister-beamable]

Diagnosis and possible remediation (kept out of the issue body, which is intentionally problem-only).

Mechanism. microservice_state is populated by a daily scheduled Lambda -- terraform-modules/scheduled-functions/microservice-state-snapshot/files/lambda_function.py in BeamablePrivateCloud, cron(0 5 * * ? *). It aggregates wizard_master.service_topology (which holds one row per live microservice instance), rolls up per CID, and ends with {"$merge": "microservice_state"}, inserting one new snapshot document per active CID per run (it is a time series; consumers read the latest doc per CID).

Root cause. service_topology contains no rows for a CID with zero running instances, so that CID falls out of the aggregation and the pipeline emits nothing for it. $merge only inserts/updates the documents the pipeline produced -- it never removes or supersedes prior docs for CIDs that dropped out. Result: on every day a CID is off, no new doc is written, and the latest-doc-per-CID read that the charts rely on stays pinned to the last day the CID was on.

Candidate directions (not prescribing one):

• Emit an explicit zero-count snapshot for CIDs that were present in a recent prior run but are absent from the current one, so "off" is recorded as a real data point rather than an absence.
• Alternatively, fix it on the read side: have consumers treat "no snapshot for a given run/day" as zero by joining against a CID reference set and the known snapshot cadence, rather than carrying the last value forward.

The read side lives in BeamableBackend/tools/history/.../MicroserviceStateStorage.scala (getMicroserviceState), feeding Portal's MicroservicesUsageBarChart.js / MicroservicesUsageLineChart.js.