Added additional data type models

[jreakin]

Expanded Unified Data Model for Campaign Finance Transactions

TL;DR

Added support for additional transaction types in the unified data model, including debts, credits, travel expenses, and campaign assets.

What changed?

• Added new transaction types to the TransactionType enum: DEBT, CREDIT, TRAVEL, and ASSET
• Created new SQLModel classes for these transaction types:
  • UnifiedDebt: Tracks outstanding debts owed by campaigns
  • UnifiedCredit: Tracks credits, refunds, and returns
  • UnifiedTravel: Tracks travel expenses with itinerary details
  • UnifiedAsset: Tracks campaign assets and their valuation
• Added corresponding field definitions to the unified field library
• Updated state field mappings for Texas data
• Added database indexes for the new models
• Added comprehensive documentation in DATA_RELATIONSHIPS.md
• Added validation scripts for Texas data files

How to test?

1. Run the validation scripts to verify Texas data files:

   python scripts/validate_texas_simple.py
   

2. Check that the new transaction types are properly detected and processed
3. Verify that the new models are created in the database schema

Why make this change?

This enhancement provides a more complete picture of campaign finance activities by capturing additional transaction types beyond basic contributions and expenditures. The expanded model allows for:

• Tracking outstanding campaign debts and obligations
• Recording refunds and credits received by campaigns
• Detailed tracking of travel expenses with itinerary information
• Monitoring campaign assets and their valuation over time

These additions enable more comprehensive analysis of campaign finances across different states using a unified data model.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

• 🔍 Trigger a full review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[jreakin]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Phase 4: Testing Improvements - Complete ✅ 4.1 Integration Tests for Production Loader Created tests/test_production_loader_integration.py with 27 tests covering: #9
• Phase 2.3: Download Wait Logic ✅ Added exponential backoff to app/states/texas/texas_downloader.py: #8
• Summary I've completed the implementation of Phase 1 (Quick Wins) from the code review plan. Here's what was accomplished: #7
• Added additional data type models #6 👈 (View in Graphite)
• feat(cleanup): cleaned up root, setup agent documentation #5
• jan25-polars

This stack of pull requests is managed by Graphite. Learn more about stacking.

[sentry]

Bug: Fallback logic for DEBT and CREDIT transactions can create invalid self-referential records where an entity is both the debtor and creditor, or payor and recipient.
_{Severity: MEDIUM}

Suggested Fix

Instead of falling back to the same entity, consider not creating the record if a distinct debtor/recipient cannot be found and logging a warning. Alternatively, raise an error to halt processing, or assign the debtor/recipient to a designated 'unknown' entity.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: app/states/unified_sqlmodels.py#L1740

Potential issue: When processing DEBT or CREDIT transactions, if a debtor or recipient
entity cannot be determined, the fallback logic incorrectly assigns the creditor or
payor entity to both roles. Specifically, in `UnifiedDebt`, the `debtor` can be set to
the `creditor_entity`, and in `UnifiedCredit`, the `recipient` can be set to the
`payor_entity`. This creates semantically invalid financial records where an entity
appears to owe itself money or give itself a credit. These corrupted records are
silently saved to the database, which can lead to incorrect financial analysis and
reporting.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[Copilot]

Pull request overview

This pull request expands the unified data model for campaign finance transactions by adding support for four new transaction types: DEBT, CREDIT, TRAVEL, and ASSET. The changes include new SQLModel classes, field definitions, state mappings for Texas data, database indexes, comprehensive documentation, and validation scripts.

Changes:

• Added new transaction types (DEBT, CREDIT, TRAVEL, ASSET) to the TransactionType enum
• Created four new unified SQLModel classes with corresponding relationships: UnifiedDebt, UnifiedCredit, UnifiedTravel, and UnifiedAsset
• Extended field library with definitions for debt, travel, and asset-related fields
• Added Texas state field mappings for the new transaction types
• Added pytest-cov dependency and coverage configuration
• Created comprehensive documentation in DATA_RELATIONSHIPS.md showing data relationships and entity diagrams
• Added two validation scripts for Texas data files

Reviewed changes

Copilot reviewed 7 out of 9 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
uv.lock	Added pytest-cov and coverage dependencies to support test coverage reporting
pyproject.toml	Configured pytest-cov with source paths and coverage exclusion rules
app/states/unified_sqlmodels.py	Added four new model classes (UnifiedDebt, UnifiedCredit, UnifiedTravel, UnifiedAsset) with relationships, indexes, and transaction processing logic
app/states/unified_field_library.py	Added field definitions for debt, travel, and asset fields, plus Texas state field mappings
scripts/validate_texas_simple.py	New validation script using simple Pydantic validators for Texas data files
scripts/validate_texas_files.py	New validation script using full SQLModel validators for Texas data
docs/DATA_RELATIONSHIPS.md	Comprehensive documentation with ER diagrams, data flow charts, and coverage analysis
app/logs/campaign_finance.log.2026-01-17	Runtime log file (should not be committed)
app/.DS_Store	macOS system file (should not be committed)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The validator modifies the input dictionary in place, which can lead to unexpected behavior if the same dictionary is reused. Consider creating a copy of the values dictionary before modifying it, or return a new dictionary instead of modifying the input.

Suggested change

	for k, v in values.items():
	if v in ["", '"', "null", "None"]:
	values[k] = None
	return values
	@model_validator(mode="before")
	@classmethod
	def parse_dates(cls, values):
	for date_field in ['receivedDt', 'contributionDt']:
	if date_field in values and values[date_field]:
	v = values[date_field]
	if isinstance(v, str):
	try:
	values[date_field] = datetime.strptime(v, "%Y%m%d").date()
	except ValueError:
	try:
	values[date_field] = datetime.strptime(v, "%Y-%m-%d").date()
	except ValueError:
	pass
	return values
	# Guard against non-dict inputs that Pydantic might pass through
	if not isinstance(values, dict):
	return values
	# Work on a shallow copy to avoid mutating the original input in place
	cleaned = dict(values)
	for k, v in cleaned.items():
	if v in ["", '"', "null", "None"]:
	cleaned[k] = None
	return cleaned
	@model_validator(mode="before")
	@classmethod
	def parse_dates(cls, values):
	# Guard against non-dict inputs that Pydantic might pass through
	if not isinstance(values, dict):
	return values
	# Work on a shallow copy to avoid mutating the original input in place
	parsed = dict(values)
	for date_field in ['receivedDt', 'contributionDt']:
	if date_field in parsed and parsed[date_field]:
	v = parsed[date_field]
	if isinstance(v, str):
	try:
	parsed[date_field] = datetime.strptime(v, "%Y%m%d").date()
	except ValueError:
	try:
	parsed[date_field] = datetime.strptime(v, "%Y-%m-%d").date()
	except ValueError:
	pass
	return parsed

[Copilot]

The field debt_original_amount is used in the code but is not defined in the unified field library. This could cause issues when trying to map state-specific fields to this unified field. Consider adding field definitions for all debt-related fields including debt_original_amount, debt_due_date, debt_paid_flag, debt_payment_amount, and debt_payment_date.

Suggested change

	),
	),
	"debt_original_amount": FieldDefinition(
	name="debt_original_amount",
	category=FieldCategory.DEBT_INFO,
	field_type=FieldType.CURRENCY,
	description="Original principal amount of the debt or loan",
	examples=["debtOriginalAmount", "originalDebtAmount"],
	validation_rules={"required": False}
	),
	"debt_due_date": FieldDefinition(
	name="debt_due_date",
	category=FieldCategory.DEBT_INFO,
	field_type=FieldType.DATE,
	description="Date by which the debt is due",
	examples=["debtDueDate", "dueDate"],
	validation_rules={"required": False}
	),
	"debt_paid_flag": FieldDefinition(
	name="debt_paid_flag",
	category=FieldCategory.DEBT_INFO,
	field_type=FieldType.BOOLEAN,
	description="Whether the debt has been fully paid",
	examples=["debtPaidFlag", "isDebtPaid"],
	validation_rules={"default": False}
	),
	"debt_payment_amount": FieldDefinition(
	name="debt_payment_amount",
	category=FieldCategory.DEBT_INFO,
	field_type=FieldType.CURRENCY,
	description="Amount paid toward the debt",
	examples=["debtPaymentAmount", "paymentAmount"],
	validation_rules={"required": False}
	),
	"debt_payment_date": FieldDefinition(
	name="debt_payment_date",
	category=FieldCategory.DEBT_INFO,
	field_type=FieldType.DATE,
	description="Date when payment toward the debt was made",
	examples=["debtPaymentDate", "paymentDate"],
	validation_rules={"required": False}
	),

[Copilot]

The validation models for travel, candidate, debt, and final reports only validate the most basic fields (recordType, filerIdent, filerName). This provides minimal validation coverage. Consider adding more field validations to these models to match the detail level of TECContributionSimple and TECExpenseSimple, or document why these minimal validators are sufficient.

[Copilot]

The documentation references a TECCredit model and credit files in the Texas data, but there is no corresponding validator class in app/states/texas/validators/. This is inconsistent with the documentation which claims credit records are "✅ FULL" supported. Either create the missing texas_creditdata.py validator or update the documentation to reflect that credit records are not yet implemented.

[Copilot]

Log files should not be committed to the repository. This file appears to contain runtime logs that should be generated locally and excluded via .gitignore.

[Copilot]

Import of 'Tuple' is not used.

Suggested change

	from typing import Dict, List, Tuple, Type, Optional
	from typing import Dict, List, Type, Optional

[Copilot]

Import of 'Any' is not used.

Suggested change

	from typing import Dict, List, Optional, Any
	from typing import Dict, List, Optional

[Copilot]

Import of 'field_validator' is not used.

Suggested change

	from pydantic import BaseModel, field_validator, model_validator, ValidationError
	from pydantic import BaseModel, model_validator, ValidationError

[Copilot]

'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

	except ValueError:
	except ValueError:
	# If parsing fails for both formats, leave the original value;
	# downstream validation or consumers are expected to handle it.

[Copilot]

'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

	except ValueError:
	except ValueError:
	# If the date string does not match any known format here,
	# leave it unchanged so that Pydantic's built-in validation
	# can raise a clear error for this field.

[sentry]

Codecov Report

❌ Patch coverage is 1.86335% with 158 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
app/states/unified_sqlmodels.py	0.00%	158 Missing ⚠️

📢 Thoughts on this report? Let us know!