Add SQLite database page attribution

AGENT_GUIDE.md

@@ -137,6 +137,7 @@ CI watching must be bounded. Do not loop indefinitely.
 
 - `status --json` and related JSON/MCP payloads currently expose the trust fields documented in `README.md` and `DEVELOPER_GUIDE.md`, including `fold_ready`, `fold_ready_reason`, `graph_table_available`, `graph_data_current`, `index_complete`, `index_incomplete_reasons`, `issues_table_available`, `file_issues_data_current`, `migration_in_progress`, `sql_graph_contract_ready`, `sql_graph_contract_degraded_reason`, `hotspot_family_ready`, `hotspot_family_degraded_reason`, `language_readiness`, `csharp_symbol_name_ready`, `csharp_metadata_target_ready`, `csharp_metadata_target_degraded_reason`, `indexed_head_commit`, `worktree_head_changed`, `indexed_head_sha`, `indexed_head_branch`, `indexed_head_timestamp`, `commits_ahead_of_indexed_head`, `head_freshness`, `index_writer_version`, `index_newer_than_reader`, `index_newer_than_reader_reason`, `unknown_extension_file_count`, `unknown_extension_files`, `unknown_extension_files_truncated`, `unknown_extension_file_path_limit`, `unknown_extension_extension_counts`, `unknown_extension_category_counts`, `unknown_extension_groups`, `extractors`, `git_executable`, `path_case_sensitive`, `data_dir`, `data_dir_source`, `data_dir_mode`, `db_file_mode`, `database_permission_policy`, `database_permission_diagnostics`, `mac_profile`, `mac_profile_diagnostics`, `db_size_bytes`, `wal_size_bytes`, `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`), `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`), `maintenance_guidance`, WAL checkpoint diagnostics (`read_only_fallback`, `wal_checkpoint_attempted`, `wal_checkpoint_succeeded`, `wal_checkpoint_skipped_reason`, `wal_checkpoint_failure_reason`, `wal_checkpoint_busy`, `wal_checkpoint_log_page_count`, `wal_checkpoint_checkpointed_page_count`, `wal_checkpoint_remaining_page_count`, `read_only_immutable_fallback`, `wal_stale_snapshot_risk`, `wal_stale_snapshot_reason`), `symbol_kinds`, `symbols_by_language`, status kind cap metadata (`symbol_kind_limit`, `symbol_kind_name_limit`, `symbol_kind_total_count`, `symbol_kind_omitted_count`, `symbol_kind_names_truncated`, `symbols_by_language_kind_total_counts`, `symbols_by_language_kind_omitted_counts`, `symbols_by_language_kind_names_truncated`), `process`, `last_index_run`, `last_failed_or_partial_index_run`, `last_failed_or_partial_index_run.progress_persisted`, `last_failed_or_partial_index_run.recovery_hint`, `last_failed_or_partial_index_run.file_errors`, `last_workspace_freshened_at`, `hooks`, `hook_diagnostics`, `trust_overrides`, MCP-only `mcp_session`, `mcp.rate_limit.bucket_limit`, `mcp.rate_limit.bucket_limit_rejection_count`, and the `status --check`-only `stale_after_seconds` / `index_age_seconds` threshold audit fields and `repair_commands`.
 - A valid CLI `status --stale-after <duration>` implies the workspace check. Check-mode JSON includes `query_context.check_mode` (`explicit` or `implied_by_stale_after`) and `query_context.stale_after_seconds`; ordinary status JSON omits `query_context`.
+- `database_size_attribution` is part of the synchronized status contract. Preserve its read-only main/WAL/SHM separation; exact logical reconciliation across object, freelist, and unexplained-residual bytes; table/index and page-type subtotals; 20-object/128-character sanitized bounds; and explicit `available=false` / stable `unavailable_reason` behavior without zero-valued unavailable object metrics.
 - Explicit WAL truncate-checkpoint diagnostics must preserve SQLite's `(busy, log, checkpointed)` result, treat non-zero `busy` or positive remaining pages as unsuccessful with bounded machine reasons, accept `(0, -1, -1)` as the successful non-WAL no-op, and never expose raw exception text or paths.
 - When any readiness field is degraded, the CLI adds `degraded_root_cause`, `degraded_reason`, `recommended_action`, `alternative_action`, and `readiness_degradations[]`. `degraded_root_cause` is the primary stable machine code; `readiness_degradations[]` lists every degraded field with `root_cause`, human reason, and remediation strings.
 - `hotspot_family_degraded_reason` currently uses `hotspot_family_support_not_indexed`, `hotspot_family_metadata_stale`, `hotspot_family_disabled_at_index_time`, `partial_family_key_population`, and `hotspot_family_marker_fingerprint_incomplete`; the incomplete marker fingerprint code means marker traversal hit safety caps and should stay synchronized with README / developer-guide recovery notes.

README.md

@@ -247,6 +247,7 @@ cancellation, and WAL snapshot-risk diagnostics.
 | Unknown-extension and runtime diagnostics | `unknown_extension_file_count`, `unknown_extension_files`, `unknown_extension_files_truncated`, `unknown_extension_file_path_limit`, `unknown_extension_extension_counts`, `unknown_extension_category_counts`, `unknown_extension_groups`, `extractors`, `hooks`, `hook_diagnostics`, `trust_overrides`, `git_executable`, `path_case_sensitive`, `data_dir_mode`, `db_file_mode`, `database_permission_policy`, `database_permission_diagnostics`, `mac_profile`, `mac_profile_diagnostics`, `stale_after_seconds`, `index_age_seconds`, `query_context.check_mode`, `query_context.stale_after_seconds`, `process`, `last_index_run`, `last_workspace_freshened_at`, `last_index_run.bytes_read_skipped_file_count`, `last_index_run.bytes_read_incomplete`, `last_index_run.diagnostics`, `last_index_run.diagnostic_count`, `last_index_run.diagnostics_truncated`, `last_index_run.reference_extraction_cap_hits`, `last_failed_or_partial_index_run`, `last_failed_or_partial_index_run.progress_persisted`, `last_failed_or_partial_index_run.recovery_hint`, `last_failed_or_partial_index_run.file_errors`. |
 | Database maintenance | `sqlite_connection_policy` (`active_mode`, `open_mode`, `immutable_uri`, WAL checkpoint/fallback fields), `db_size_bytes`, `wal_size_bytes`, `db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`), `prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`), `maintenance_guidance`. Query-only status reports `pooling=false` and either `read_only` or `immutable_read_only_uri`; checkpointed WAL databases use an immutable private snapshot, while non-empty WAL databases use a stable private main/WAL snapshot so source sidecars remain unchanged. Persistent private-snapshot copy failures report `query_only_snapshot_copy_failed` with temporary-storage capacity and permission guidance instead of being misreported as WAL churn. |
 | WAL checkpoint diagnostics | `read_only_fallback`, `wal_checkpoint_attempted`, `wal_checkpoint_succeeded`, `wal_checkpoint_skipped_reason`, `wal_checkpoint_failure_reason`, `wal_checkpoint_busy`, `wal_checkpoint_log_page_count`, `wal_checkpoint_checkpointed_page_count`, `wal_checkpoint_remaining_page_count`, `read_only_immutable_fallback`, `wal_stale_snapshot_risk`, `wal_stale_snapshot_reason`. |
+| Database size attribution | `database_size_attribution` separates main DB, WAL, and SHM file bytes and reconciles logical pages across tables, indexes, freelist pages, internal/leaf/overflow page types, payload, unused space, structural overhead, and `unexplained_residual_bytes`. It emits at most 20 redacted/truncated object names. `available=false` with `unavailable_reason` means page attribution was not measurable; omitted object-byte fields must not be interpreted as zero. |
 | Remediation fields | `degraded_root_cause`, `degraded_reason`, `recommended_action`, `alternative_action`, `readiness_degradations`, `repair_commands`. |
 | MCP-only session diagnostics | `mcp_session`, `mcp_session.metrics`, `mcp_session.audit_log`, `mcp.rate_limit.bucket_limit`, `mcp.rate_limit.bucket_limit_rejection_count`. |
 
@@ -697,6 +698,7 @@ immutable URI の選択、timeout、cancellation、WAL snapshot risk の diagnos
 | unknown-extension / runtime diagnostics | `unknown_extension_file_count`, `unknown_extension_files`, `unknown_extension_files_truncated`, `unknown_extension_file_path_limit`, `unknown_extension_extension_counts`, `unknown_extension_category_counts`, `unknown_extension_groups`, `extractors`, `hooks`, `hook_diagnostics`, `trust_overrides`, `git_executable`, `path_case_sensitive`, `data_dir_mode`, `db_file_mode`, `database_permission_policy`, `database_permission_diagnostics`, `mac_profile`, `mac_profile_diagnostics`, `stale_after_seconds`, `index_age_seconds`, `query_context.check_mode`, `query_context.stale_after_seconds`, `process`, `last_index_run`, `last_workspace_freshened_at`, `last_index_run.bytes_read_skipped_file_count`, `last_index_run.bytes_read_incomplete`, `last_index_run.diagnostics`, `last_index_run.diagnostic_count`, `last_index_run.diagnostics_truncated`, `last_index_run.reference_extraction_cap_hits`, `last_failed_or_partial_index_run`, `last_failed_or_partial_index_run.progress_persisted`, `last_failed_or_partial_index_run.recovery_hint`, `last_failed_or_partial_index_run.file_errors`。 |
 | database maintenance | `sqlite_connection_policy` (`active_mode`、`open_mode`、`immutable_uri`、WAL checkpoint / fallback field)、`db_size_bytes`、`wal_size_bytes`、`db_pragma_settings` (`journal_mode`, `synchronous`, `wal_autocheckpoint`, `busy_timeout_ms`, `page_count`, `freelist_count`, `page_size`, `auto_vacuum`)、`prepared_command_cache` (`count`, `capacity`, `hit_count`, `miss_count`, `eviction_count`)、`maintenance_guidance`。query-only status は `pooling=false` と、`read_only` または `immutable_read_only_uri` の mode を返します。checkpoint 済み WAL database は immutable な private snapshot、non-empty WAL database は安定した private main/WAL snapshot から読むため source sidecar は変化しません。private snapshot の永続的な copy failure は WAL churn と誤報せず、temporary storage の容量・権限を案内する `query_only_snapshot_copy_failed` を返します。 |
 | WAL checkpoint diagnostics | `read_only_fallback`、`wal_checkpoint_attempted`、`wal_checkpoint_succeeded`、`wal_checkpoint_skipped_reason`、`wal_checkpoint_failure_reason`、`wal_checkpoint_busy`、`wal_checkpoint_log_page_count`、`wal_checkpoint_checkpointed_page_count`、`wal_checkpoint_remaining_page_count`、`read_only_immutable_fallback`、`wal_stale_snapshot_risk`、`wal_stale_snapshot_reason`。 |
+| database size attribution | `database_size_attribution` は main DB / WAL / SHM の file byte を分離し、論理 page を table、index、freelist、internal / leaf / overflow page、payload、unused space、structural overhead、`unexplained_residual_bytes` に再照合します。object 名は伏字・切り詰めを適用して最大20件だけ返します。`available=false` と `unavailable_reason` がある場合は page attribution を計測できなかったことを示し、省略された object-byte field をゼロとして解釈してはいけません。 |
 | remediation fields | `degraded_root_cause`, `degraded_reason`, `recommended_action`, `alternative_action`, `readiness_degradations`, `repair_commands`。 |
 | MCP-only session diagnostics | `mcp_session`, `mcp_session.metrics`, `mcp_session.audit_log`, `mcp.rate_limit.bucket_limit`, `mcp.rate_limit.bucket_limit_rejection_count`。 |
 

TESTING_GUIDE.md

@@ -103,6 +103,7 @@ Use `docs/test-doc-maintenance-plan.md` before moving oversized suites or adding
   C# qualified common-call coverage keeps static BCL, instance, LINQ extension, alias-qualified, current-instance, and unresolved-receiver cases together. Assert that extraction retains every row, default bare-name references/callers/callees and hotspot counts retain resolved evidence while excluding unresolved noise, the completeness option restores that noise deterministically, and dependency queries remain identity-scoped (#4867).
   Crystal, Groovy, Tcl, Prolog, and `ambiguous_pl` graph fixtures keep imports, parenthesized calls, conservative same-file command/predicate calls, caller containers, and keyword false-positive controls independently diagnosable; capability tests must keep their advertised symbol/reference/graph flags aligned with those extractor fixtures (#4746).
   Database status tests must also pin stale and current extractor-version stamps for those languages so pre-graph rows cannot report authoritative graph readiness.
+  Database page-attribution coverage keeps empty/schema-only, WAL-visible overflow, database-truncating WAL, and a connection-pinned read snapshot after a newer WAL commit, cancellation of both `dbstat` aggregation and WAL validation, the bounded large-page-count rejection, reconciliation, corrupt-file rejection, main/WAL/SHM separation, the 20-object and 128-character support-safe output bounds, explicit unavailable/not-requested values, and unchanged `total_changes()` / `PRAGMA query_only` in coordinated `DbReaderTests` fixtures.
   HDL graph coverage keeps Verilog, SystemVerilog, and VHDL in one coordinated extractor fixture for syntax-visible hierarchy/package/architecture edges, known same-file signal/type references, scope attribution, declaration/literal false-positive suppression, and comment masking. The paired `QueryCommandRunnerReferencesTests` fixture must persist and query one representative edge per HDL language so extractor-only assertions cannot hide database or CLI graph drift. `IndexCommandRunnerFullScanTests` must also clear the HDL graph contract and persisted edge, then prove that a normal full scan refreshes an unchanged HDL file before restoring readiness.
   JavaScript/TypeScript discriminant-guard coverage shares one JavaScript-compatible fixture across both languages, while `DatabaseTests` pins `type_tag` as an accepted persisted reference kind so extractor-only assertions cannot hide database kind-catalog drift.
   Dockerfile named-stage reference variants share one multi-stage fixture when ordinary, lowercase, platform-flagged, commented, hyphenated, and dotted forms can be distinguished by exact per-stage call counts; keep external base-image exclusions in that fixture as the negative control.
@@ -1039,6 +1040,7 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests"
   C# の修飾付き一般名 call の coverage は、static BCL、instance、LINQ extension、alias 修飾、current instance、未解決 receiver の各 case を同じ fixture に維持します。extraction が全 row を保持すること、無修飾名による references / callers / callees と hotspot count の既定動作が解決済み evidence を維持しつつ未解決 noise を除外すること、completeness option がその noise を決定的に復元すること、dependency query が identity scope のままであることを検証してください（#4867）。
   Crystal、Groovy、Tcl、Prolog、`ambiguous_pl` の graph fixture では、import、括弧付き call、同一ファイルに限定した保守的な command / predicate call、caller container、keyword の false-positive control を個別に診断可能な状態で維持し、capability test の symbol / reference / graph 広告を extractor fixture と一致させてください（#4746）。
   さらに database status test でこれらの言語の古い extractor-version stamp と現行 stamp を固定し、graph 対応前の row が authoritative な graph readiness を報告できないことを検証してください。
+  database page-attribution coverage では、empty / schema-only、WAL から可視な overflow、database を縮小する WAL、後続 WAL commit 後も connection に固定された read snapshot の各 case、`dbstat` 集約と WAL 検証のキャンセル、large page count の上限付き拒否、再照合、破損 file の拒否、main/WAL/SHM の分離、20 object / 128文字の support-safe 出力上限、明示的な unavailable / not-requested 値、`total_changes()` / `PRAGMA query_only` が不変であることを、連携した `DbReaderTests` fixture で維持してください。
   HDL graph coverage では Verilog、SystemVerilog、VHDL を1つの連携した extractor fixture にまとめ、構文上確認できる hierarchy/package/architecture edge、同一ファイル内の既知 signal/type reference、scope attribution、declaration/literal の false-positive 抑止、comment masking を検証します。対になる `QueryCommandRunnerReferencesTests` fixture では各 HDL 言語の代表 edge を1件ずつ永続化して query し、extractor-only assertion が database または CLI graph のずれを隠せないようにします。さらに `IndexCommandRunnerFullScanTests` では HDL graph contract と永続化 edge を消去し、通常の full scan が未変更 HDL file を更新してから readiness を復元することを検証します。
   JavaScript / TypeScript の discriminant-guard coverage は両言語で JavaScript-compatible な fixture を共有し、`DatabaseTests` で `type_tag` が永続化可能な reference kind であることも固定して、extractor-only assertion が database kind catalog のずれを隠せないようにします。
   Dockerfile の named-stage reference variant は、通常、小文字、platform flag、comment、hyphen、dot 形式を stage ごとの厳密な call 数で区別できる場合、1つの multi-stage fixture を共有します。外部 base image の除外も negative control として同じ fixture に残します。

changelog.d/unreleased/4888.added.md

@@ -0,0 +1,26 @@
+---
+category: added
+issues:
+  - 4888
+affected:
+  - src/CodeIndex/Database/DbReader.DatabaseSizeAttribution.cs
+  - src/CodeIndex/Database/DbReader.cs
+  - src/CodeIndex/Database/SqlitePageAttributionReader.cs
+  - src/CodeIndex/Database/DbReader.Status.cs
+  - src/CodeIndex/Cli/QueryCommandRunner.Status.cs
+  - src/CodeIndex/Mcp/McpToolHandlers.Query.Status.cs
+  - src/CodeIndex/Models/QueryResults.cs
+  - tests/CodeIndex.Tests/DbReaderTests.cs
+  - README.md
+  - DEVELOPER_GUIDE.md
+  - TESTING_GUIDE.md
+  - AGENT_GUIDE.md
+---
+
+## English
+
+- **`status --json` now reconciles SQLite database bytes by object and page usage (#4888)** — The bounded, source-read-only `database_size_attribution` block separates main DB, WAL, and SHM bytes; attributes tables, indexes, freelist, internal/leaf/overflow pages, payload, unused space, structural overhead, and unexplained residual bytes; and emits at most 20 support-sanitized object names. A live WAL fallback pins the connection's active read snapshot before traversal so concurrent commits cannot mix status generations. Unsupported or inconsistent page probes report an explicit unavailable reason without presenting missing measurements as zero, while human/compact and unrelated internal status consumers skip the page scan.
+
+## 日本語
+
+- **`status --json` が SQLite database byte を object / page usage 別に再照合できるようになりました (#4888)** — source を変更しない件数上限付きの `database_size_attribution` block が main DB、WAL、SHM の byte を分離し、table、index、freelist、internal/leaf/overflow page、payload、unused space、structural overhead、未説明の残差 byte を分類し、support-safe に処理した object 名を最大20件返します。live WAL fallback は traversal 前に connection の active read snapshot を固定し、並行 commit による status 世代の混在を防ぎます。page probe が非対応または不整合の場合は欠損値をゼロとして示さず、明示的な unavailable reason を返し、人間向け/compact output と無関係な内部 status 利用では page scan を省略します。

src/CodeIndex/Cli/IndexCommandRunner.Maintenance.cs

@@ -230,7 +230,7 @@ private static int RunOptimizeFtsPreviewForDb(
                         MaintenanceDatabaseFailureKind.NotDatabase));
             }
 
-            var status = new DbReader(db).GetStatus();
+            var status = new DbReader(db).GetStatus(includeDatabaseSizeAttribution: false);
             var objectSizes = ReadOptimizeObjectSizes(
                 db,
                 forceLogicalObjectSizeFallbackForTesting,