MoneyWiz read compatibility for schema variants

[marcomc]

Summary

This PR adds compatibility profiles for three observed MoneyWiz SQLite investment-column layouts: unsuffixed, suffixed, and mixed consumer-specific.
Unknown or ambiguous layouts are rejected rather than mapped speculatively.

MoneyWiz has changed its physical database columns between versions. Investment values may use suffixed names such as ZNUMBEROFSHARES1, unsuffixed names such as ZNUMBEROFSHARES, or a mixture where holdings and transactions use different columns.

Without schema-aware parsing, the API can fail to load a valid database or silently read a financial value belonging to another entity type.

Why the redesign was necessary

Selecting the first available column alias is unsafe because Core Data rows can expose both columns simultaneously:

Consumer	Required share column in a mixed schema
Investment holding	ZNUMBEROFSHARES
Investment buy/sell transaction	ZNUMBEROFSHARES1

Both values can be valid decimals, so selecting the wrong one may not produce an obvious error.

The implementation now detects the complete schema topology once and selects an explicit alias for each consumer.

Schema profiles

Motivation

Individual model rows do not contain enough context to decide what each physical column represents. The decision must be based on the database schema rather than whichever value appears first.

Implementation

src/moneywiz_api/schema_profile.py inspects ZSYNCOBJECT using PRAGMA table_info and identifies:

• Unsuffixed investment columns.
• Suffixed investment columns.
• Mixed holding and transaction columns.
• Unknown or ambiguous schemas.

Ambiguous schemas are rejected instead of risking incorrect financial data.

src/moneywiz_api/database_accessor.py detects and retains this profile when opening the database so every record receives the same interpretation.

Investment parsing

Motivation

Investment holdings and investment transactions are separate schema consumers. They cannot safely share one global “number of shares” alias.

Buy and sell transactions must also follow the same policy; fixing only one would leave inconsistent transaction behavior.

Implementation

• src/moneywiz_api/model/investment_holding.py uses the holding-specific share column.
• src/moneywiz_api/model/transaction.py uses transaction-specific share and price columns for both buy and sell transactions.
• src/moneywiz_api/model/raw_data_handler.py centralizes required, nullable, aliased, and profile-selected decimal parsing.
• The investment managers pass the database profile to their model constructors.

Transfer compatibility

Motivation

Some MoneyWiz transfer rows omit a derived amount even when the canonical transaction amount and exchange metadata remain available.

Reconstructing the value blindly is unsafe because a zero exchange rate is invalid and a rounded exchange rate may not reproduce the exact value stored by MoneyWiz.

Implementation

src/moneywiz_api/model/transaction.py now:

• Reconstructs missing derived transfer amounts when sufficient data exists.
• Preserves the canonical stored amount when exchange-rate rounding would alter it.
• Rejects reconstruction when the exchange rate is zero.
• Normalizes nullable currency values.

Best-effort record loading

Motivation

One malformed or unsupported row should not prevent thousands of otherwise valid records from loading.

At the same time, skipped records must remain observable so callers can diagnose incomplete results.

Implementation

src/moneywiz_api/managers/record_manager.py adds:

• A construction hook for models requiring schema context.
• Isolation of expected parsing and validation failures.
• A load_errors collection containing the record ID, entity type, and exception type.

Unexpected programming and infrastructure errors continue to propagate.

Reduced-schema compatibility

Motivation

Older, reduced, and fixture databases may omit optional MoneyWiz relationship tables or filtered metadata columns.

Those omissions should produce empty optional relationships rather than prevent the main records from loading.

Implementation

• src/moneywiz_api/database_accessor.py verifies optional tables before querying them.
• src/moneywiz_api/model/raw_data_handler.py tolerates absent optional filtered fields.

Regression coverage

• tests/unit/test_schema_profiles.py covers suffixed, unsuffixed, mixed, and ambiguous schemas.
• tests/unit/test_profiled_investment_values.py verifies consumer-specific values for holdings and both buy and sell transactions.
• tests/unit/test_transaction_compatibility.py covers transfer reconstruction, zero-rate rejection, rounding preservation, and continued loading after an invalid record.

Validation

• uv run pytest -q tests/unit — 23 passed
• uv run ruff check src tests/unit — passed
• uv run ruff format --check src tests/unit — passed
• git diff --check — passed

[marcomc]

This helper preserves consistent parsing across both supported access paths.

The investment managers already pass the database-level SchemaProfile to
investment constructors. However, get_record() and get_record_by_gid() also
accept model constructors directly. Calling one of those methods with
InvestmentHolding, InvestmentBuyTransaction, or InvestmentSellTransaction
must therefore pass the same profile.

Without this delegation, direct accessor calls would fall back to row-level
alias selection. That is unsafe for mixed schemas, where holdings and
transactions intentionally use different share columns. The same record could
then be interpreted differently depending on whether it was loaded through a
manager or directly through the accessor.

Non-investment constructors retain the original constructor(row) behavior.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9181c31626

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ileodo]

could you please check if #40 solve the same issue?

[marcomc]

could you please check if #40 solve the same issue?

My reading is that #40 and #37 address related but different schema problems:

• [Refactor] Support different schema version #40 is a general schema-version refactor. It introduces declarative field mappings and dynamically discovers version-dependent tables and columns, such as the tag join table.
• MoneyWiz read compatibility for schema variants #37 addresses an investment-specific ambiguity. In some databases, holdings and buy/sell transactions use different columns, for example:
  • holdings: ZNUMBEROFShares
  • transactions: ZNUMBEROFShares1

The key difference is that #40 resolves aliases row-by-row by selecting the first available column. That cannot safely distinguish those mixed investment layouts when both columns exist.

#37 detects the database schema once, assigns aliases per consumer, and rejects ambiguous layouts. Therefore, #40 is related infrastructure, but it does not replace the consumer-specific investment handling in #37.

I think #37 should be merged for this issue. Useful parts of #40, especially dynamic tag-table discovery, could be adapted separately while preserving #37’s schema-profile logic.