tech-debt: Transaction.confidence_score is declared but never written by the pipeline (CURR-03)

[longieirl]

Why this exists

Transaction.confidence_score: float = 1.0 was introduced in v1.1 as part of the Transaction model enrichment (phase 21). The intent was to carry a per-row confidence signal from the extraction pipeline to consumers.

However, no code in the extraction pipeline ever sets this field. Every Transaction produced by the pipeline has confidence_score = 1.0 (the default). The field is serialised and deserialised correctly, but the value is always meaningless.

The design decision logged at the time: "confidence_score weight constants not configurable (CURR-03) — deferred."

---

What exists today

# Transaction model
confidence_score: float = 1.0  # default, never overwritten

The pipeline flow:

• RowBuilder builds raw row dicts — no confidence signal
• PDFTableExtractor.extract() calls dicts_to_transactions(rows) — confidence_score defaults to 1.0
• No code in extraction/, services/, or processor.py sets confidence_score to anything else

The only place confidence_score is used at all is in iban_spatial_filter.py — but that is a completely separate IBANCandidate.confidence_score field, not Transaction.confidence_score.

---

What needs to change

Define what factors should contribute to Transaction.confidence_score and wire them in the extraction pipeline. Candidate signals already present in the pipeline:

Signal	Source	Suggested weight
Row has missing balance	RowPostProcessor	-0.2
Date parse used fallback	DateParserService	-0.1
Row flagged by filter_invalid_dates but kept	TransactionFilterService	-0.3
Row from low-density page	ContentDensityService	-0.1
Multi-line merge applied	RowMergerService	-0.05

The score should be clamped to [0.0, 1.0].

The weight constants should live in a ScoringConfig dataclass (injectable, following the pattern established for ScoringConfig in PR #36) rather than being hardcoded in the extractor.

---

What will change

Area	Change
New: domain/models/scoring_config.py	ScoringConfig with penalty weight constants
extraction/pdf_extractor.py	Compute and set tx.confidence_score based on extraction signals
extraction/row_post_processor.py	Emit a signal when balance is missing
services/transaction_filter.py	Optionally emit signal for borderline date rows
PDFTableExtractor / PDFProcessingOrchestrator	Accept optional ScoringConfig

---

Risk

• Medium. The field is currently inert so there is no regression risk on existing behaviour. However, introducing scoring logic in the extraction layer requires the extraction layer to know about warning events — which ties into CURR-02 (structured ExtractionWarning). Recommend implementing CURR-02 first.
• Weight constants are subjective. They should be documented and tested, not just hardcoded.

---

Acceptance criteria

• At least one extraction signal sets confidence_score to a value other than 1.0 on affected rows
• ScoringConfig is injectable with documented weight constants
• confidence_score is clamped to [0.0, 1.0]
• Tests cover at least: full-confidence row, reduced-confidence row, and minimum-confidence row
• All existing tests pass; coverage >= 91%

[longieirl]

Additional context — source_page is also never set by the pipeline

This issue was filed covering confidence_score only, but source_page has the same problem: declared on Transaction (line 53), fully serialised/deserialised, but never written by the extraction pipeline.

Where source_page should be set

pdf_extractor.py iterates pages with for page_num, page in enumerate(pdf.pages, 1) (line 102). Each page produces page_rows (a list[dict]) via _extract_page(page, page_num). After RowPostProcessor.process_page(), those rows are appended to a flat rows list — but page_num is never stamped onto the rows.

When dicts_to_transactions(rows) is called at line 137, every Transaction gets source_page=None.

Impact

source_page was introduced so consumers could trace which page a transaction came from. Currently that information exists in the extraction loop but is discarded before the dicts are created. The field is always None in every produced output.

Fix

In pdf_extractor.py, stamp page_num onto each row dict before appending:

page_rows = self._extract_page(page, page_num)
if page_rows is None:
    continue
for row in page_processor.process_page(page_rows):
    row["source_page"] = page_num
    rows.append(row)

source_page is already in Transaction.standard_keys (line 234) so from_dict will pick it up correctly with no further changes.

Relationship to confidence_score

Both fields are inert for the same root reason: the extraction pipeline accumulates rows as list[dict] and converts to Transaction in a single batch call at the end — there is no opportunity to set per-row metadata during extraction without stamping the dicts first. The fix pattern is the same for both: stamp the dict during the page loop, before dicts_to_transactions().

Suggested scope change

Rename this issue or create a companion to cover both source_page and confidence_score together, since they share the same cause and the same fix location.