Cost accounting: replace unsafe ledgers and make attribution, pricing, and coverage authoritative

[manishiitg]

Problem

Cost accounting currently spans a global JSONL ledger, workflow phase/run JSON files, paid-tool writers, Pulse summaries, and agent-authored org-dashboard cards. The paths do not share one transactional event model and several production semantics are incorrect.

Confirmed findings

1. The global ledger is unsafe and already malformed

   • costledger.Ledger.Append reads and rewrites the entire _system/costs.jsonl for every event.
   • The live ledger is roughly 4.65 MB / 15k lines and its first line is malformed.
   • Mutexes protect only individual Ledger instances, so tool writers, agent writers, or multiple processes can race and lose data.

2. The dashboard's “LLM calls” count is not an LLM-call count

   • Aggregation increments CallCount once per ledger entry.
   • Production emits cumulative conversation/turn events; the actual llm_call_count in generation metadata is discarded.

3. Effective model metadata is dropped

   • CLI adapters can expose the served/effective model.
   • The final cumulative token event builds a new metadata map and does not retain cost_model_id / effective-model fields.
   • Aggregation groups by requested ModelID, even when an effective model exists.

4. Billing basis is mislabeled

   • Token-rate calculations can flow through TokenUsageEvent.TotalCost and are then labeled provider.
   • Provider actuals, token estimates, and subscription shadow estimates are mixed into one total.

5. Reused agents can double-count

   • Agent token APIs return cumulative lifetime totals.
   • Builder/Pulse persistence can merge those cumulative snapshots as new deltas across sequential turns.

6. Workflow cost writers can overwrite each other

   • LLM and paid-tool paths independently read-modify-write the same daily JSON files without one shared transaction.

7. Missing/corrupt evidence is hidden

   • The workflow-cost API converts read failures into empty collections and still returns success: true.
   • Pulse can mistake accounting failure for zero cost.

8. Historical Pulse lookup uses the retired path

   • get_cost_summary checks runs/<run>/token_usage.json for non-current runs instead of the current costs/execution store.

9. Scope attribution is inferred unreliably

   • Builder, Pulse, Goal Advisor, evaluation, and background-review costs are inferred from phase/step state.
   • Only planning is currently treated as phase-only, so other maintenance/background work can be misbucketed or omitted.

10. Tool-cost attribution depends on output paths

    • Paid operations without output files are not recorded.
    • Workflow attribution is inferred from the first output path rather than explicit execution context.

11. Frontend cache tokens can be double-counted

    • The fallback to aggregate cache_tokens is used when cache_read_tokens is zero, then cache-write tokens are added separately.

12. The workflow-cost popup does not scale

    • It loads all cost files and then requests execution logs serially per run only to recover titles.
    • The workspace already contains more than a thousand cost files.

Proposed architecture

Use a single SQLite-backed accounting repository with immutable event rows.

Each event should include:

• unique event_id and idempotency key
• timestamp and user/workflow/session/run/execution IDs
• explicit scope: builder, pulse, workflow_execution, evaluation, chief_of_staff, background_review, tool
• requested and effective provider/model
• turn count and actual LLM-call count
• prompt/output/reasoning/cache-read/cache-write tokens
• cost and currency
• billing basis: provider_actual, token_estimate, subscription_shadow, unpriced
• pricing source/version
• tool name and operation metadata where applicable

Dashboards, Pulse, and org reporting should derive rollups from these immutable events. JSON/HTML outputs may remain presentation artifacts, not accounting sources of truth.

Migration plan

1. Add SQLite event schema and repository with atomic insert/upsert.
2. Emit per-turn or per-call deltas, never cumulative snapshots as deltas.
3. Route LLM and paid-tool events through the same repository.
4. Add explicit accounting scope to execution context.
5. Expose paginated summary APIs with coverage and parse/persistence errors.
6. Migrate valid legacy JSONL/JSON records with deterministic event IDs; quarantine malformed records.
7. Switch Cost Dashboard, workflow popup, Pulse get_cost_summary, and org cost cards to the canonical API.
8. Remove legacy read-modify-write ledgers after a verification window.

Acceptance criteria

• Concurrent LLM and tool events cannot overwrite or duplicate each other.
• Retried events are idempotent.
• Actual LLM-call count is distinct from turns/accounting events.
• Requested and effective models are both retained.
• Actual, estimated, subscription-shadow, and unpriced amounts are separately visible.
• Pulse reports missing/unpriced accounting evidence explicitly.
• Historical and UTC-midnight-spanning runs resolve correctly.
• Builder/Pulse/evaluation/workflow/tool costs are separately attributable.
• Org dashboard cost status includes freshness and accounting coverage.
• Tests exercise real production EndAgentSession events, reused multi-turn agents, concurrent writers, retries, malformed migration input, and midnight-spanning runs.

[manishiitg]

Foundation implementation is ready in draft PR #151.

Completed in this PR:

• authoritative SQLite cost events with WAL and idempotent concurrent inserts
• legacy JSONL migration plus malformed-row quarantine
• per-LLM-call accounting from LLMGenerationEnd (cumulative totals are fallback only)
• requested/effective model retention and explicit billing basis
• actual LLM call count separated from accounting/tool event count
• paid-tool costs routed through the same database, including no-output operations
• workflow/session/run/execution attribution fields and concurrency/retry tests

Remaining for a follow-up PR: switch workflow/Pulse/org readers from compatibility JSON files to scoped/paginated SQLite queries, expose persistence/coverage failures in those surfaces, then retire legacy writers after verification.