Added deterministic preprocessor functionality

[DarrenRiedlinger]

Proposed solution for issue #241 allowing user-defined, deterministic preprocessing of imported sources.

Rather than having the pre-processing logic hard-coded within the reconcile.py, I settled on a minimal approach where reconcile.py is just very slightly modified with a hook to pass the LoadedReconciler instance to a user-defined preprocessor function.

That user-defined preprocessor function can be any user-supplied function. It has full access to the LoadedReconciler state, so it can directly modify the list of pending_entries, and also has access to the JournalEditor class for writing entries to the appropriate ledger. So it hopefully allows for many use-cases.

For simple use cases, I also created a separate BasePreprocessor class and a TransactionRuleMixin. User-defined preprocessors that subclass from these can specify pre-processing rules in a dictionary-like fashion

E.g.

from beancount_import.preprocessor import BasePreprocessor, TransactionRuleMixin, PreprocessingRule

rules = [
    PreprocessingRule(
        source_name="venmo",
        transaction_narration="Tutoring",
        posting_meta={"venmo_description": "Tutoring"},
        target_account="Expenses:Tutoring",
        action="record",
    ),
]


class RuleBasedPreprocessor(TransactionRuleMixin, BasePreprocessor):
    def preprocess(self) -> None:
        self.apply_rules(rules)

And this can be passed to the webserver like so:

beancount_import.webserver.main(
    ...
    preprocessor=RuleBasedPreprocessor,
    )

Let me know your thoughts on this approach.
If I had React experience + time it would be nice to also allow creating PreprocessingRules based on the selected pending entry directly within the web-ui, but that's unfortunately beyond what i can handle at the moment.

[Zburatorul]

Finally got to try this. Worked pretty well! The example you gave in the comment above was very helpful. It would have taken a lot longer to figure out how to set it up.
Could you add an example folder in here or some text in that README?

I haven't reviewed the code itself yet. Tests are failing on list[] vs typing.List[]

[DarrenRiedlinger]

Great. Yeah, traveling this weekend so might be a couple days till I get to it, but happy to add a more prominent example. Also, since I submitted, I noticed some issues with updating the pending_data of the passed LoadedReconciler instance that has to do with whether those attributes get passed by reference or value and then whether the changes “stick” outside the Preprocessor. I think it doesn’t need a big change to fix, but need to dig into what exactly is happening when I have a moment. Thanks

…

On Fri, Dec 6, 2024 at 4:04 PM Eugeniu Plamadeala ***@***.***> wrote: Finally got to try this. Worked pretty well! The example you gave in the comment above was very helpful. It would have taken a lot longer to figure out how to set it up. Could you add an example folder in here <https://github.com/jbms/beancount-import/tree/master/examples> or some text in that README? I haven't reviewed the code itself yet. Tests are failing on list[] vs typing.List[] — Reply to this email directly, view it on GitHub <#243 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJSVVRTLHQRN2OV7U5LCJ32EI3PHAVCNFSM6AAAAABSTM5NMOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRUGY3TAMBYGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Zburatorul]

Did you end up sorting out the reference/copy issue?