Sea ORM integration

[dcorral]

Data Persistence Analysis for Migration to Sea-ORM Database in LRN

The following data is currently written to disk:

1. Logs

• Description: Application logs with timestamps, levels, and messages.
• Persistence Method: Append-only text file.
• File Location: data_dir/logs/logs.txt
• Recommendation: Keep as filesystem. Logs are sequential and not query-intensive; moving to DB would add unnecessary overhead.

2. Channel Peer Data (channel_peer_data)

• Description: Mapping of public keys to socket addresses for channel peers.
• Persistence Method: Text file with lines in format pubkey@address.
• File Location: ldk_data_dir/channel_peer_data

3. PSBT Files (psbt_{txid})

• Description: Partially Signed Bitcoin Transactions for funding transactions.
• Persistence Method: Individual text files containing PSBT strings.
• File Location: ldk_data_dir/psbt_{funding_txid}

4. Configuration Files

• Description: Various configuration values including indexer URL, Bitcoin network, wallet fingerprints, and extended public keys.
• Persistence Method: Individual text files with single values.
• File Locations:
  • storage_dir_path/indexer_url
  • storage_dir_path/bitcoin_network
  • storage_dir_path/wallet_fingerprint
  • storage_dir_path/wallet_account_xpub_colored
  • storage_dir_path/wallet_account_xpub_vanilla
  • storage_dir_path/wallet_master_fingerprint

5. Encrypted Mnemonic

• Description: Encrypted wallet mnemonic phrase.
• Persistence Method: Base64-encoded encrypted string in file.
• File Location: mnemonic_path (user-specified)

6. Revoked Tokens

• Description: List of revoked authentication tokens.
• Persistence Method: Temporary file with token data.
• File Location: Temporary file in auth directory

7. Asset Media Files

• Description: Binary media files for assets (images, etc.).
• Persistence Method: Individual binary files.
• File Location: file_path (user-uploaded)

8. Backup Files

• Description: ZIP archives and encrypted backup files.
• Persistence Method: Compressed/encrypted files.
• File Location: User-specified backup paths

9. Serialized Lightning Data Structures

• Description: Complex binary-serialized data structures for Lightning Network operations.
• Persistence Method: Binary serialization to files.
• File Locations:
  • ldk_data_dir/network_graph (NetworkGraph)
  • ldk_data_dir/inbound_payments (InboundPaymentInfoStorage)
  • ldk_data_dir/outbound_payments (OutboundPaymentInfoStorage)
  • ldk_data_dir/output_spender_txes (OutputSpenderTxes)
  • ldk_data_dir/maker_swaps and taker_swaps (SwapMap)
  • ldk_data_dir/channel_ids (ChannelIdsMap)
  • ldk_data_dir/scorer (ProbabilisticScorer)

Candidates for DB Migration

• Encrypted Mnemonic: Sensitive data requiring secure storage.
• Channel Peer Data: Simple key-value, frequent reads/writes.
• Configuration Files: Static data, better managed in DB.
• Revoked Tokens: Token management with potential for queries and cleanup.
• Payment Information: Inbound/Outbound payments.
• Swap Data: Maker/Taker swaps with potential for complex relationships.
• Channel IDs: Simple mapping that could be queried.

Keep as Files

1. Logs: Append-only, better as files.
2. PSBT Files: Temporary, small files.
3. Asset Media: Large binaries, better on filesystem.
4. Backup Files: Archival, no need for DB.
5. Network Graph & Scorer: Complex binary structures, keep serialized unless specific needs.

[gofman8]

6. Revoked Tokens
   this is actually needs to be stored in db, loosing this tokens leads to security issues

[gofman8]

with the rest I do agree, let's do it! but the only thing I have no idea yet is "PSBT Files". I don't really understand the purpose of this files yet, so this we need to figure out and ask in RLN chat

[dcorral]

*AI generated report

RLN Filesystem Persistence Audit

Goal

Consolidate all critical node state into a single SQLite database file (rln_db). This audit identifies what's currently stored on the filesystem, whether it can be migrated to the database, and what constraints exist due to rust-lightning's direct file access.

---

Current State Summary

Already in Database (rln_db)

Table	Data	Status
mnemonic	Encrypted wallet seed phrase	Done
rgb_config	Key-value store for RGB configuration	Done
channel_peer_data	Peer public keys and socket addresses	Done
revoked_token	Revoked authentication tokens	Done

Still on Filesystem

Everything else. See detailed breakdown below.

---

Filesystem Data Analysis

1. LDK Core State (.ldk/ directory)

These files are managed by rust-lightning's FilesystemStore and MonitorUpdatingPersister.

File/Directory	Description	Can Migrate?	Recommendation
manager	ChannelManager serialized state	No	LDK reads directly via ChannelManagerReadArgs
network_graph	Lightning network routing graph	No	LDK reads directly on startup
scorer	Probabilistic payment routing scorer	No	LDK reads directly on startup
monitors/*	Per-channel monitor state	No	ChainMonitor reads directly
monitor_updates/*	Incremental monitor updates	No	MonitorUpdatingPersister reads directly
archived_monitors/*	Archived channel monitors	No	LDK reads directly

Verdict: These cannot be migrated. LDK's persistence layer (FilesystemStore, MonitorUpdatingPersister) expects files on disk. Changing this would require modifying rust-lightning itself.

---

2. RLN Application State (.ldk/ directory)

These are written via FilesystemStore::write() but read by RLN code, not LDK internals.

File	Description	Can Migrate?	Recommendation
inbound_payments	Payment hash to payment info map	Yes	Migrate to DB
outbound_payments	Payment ID to payment info map	Yes	Migrate to DB
channel_ids	Temporary to final channel ID map	Yes	Migrate to DB
maker_swaps	Maker swap data	Yes	Migrate to DB
taker_swaps	Taker swap data	Yes	Migrate to DB
output_spender_txes	Output spending transaction data	Yes	Migrate to DB

Verdict: These can be migrated. They're read/written via src/disk.rs functions that you control.

Migration approach: Create new tables for each, serialize/deserialize using the same TLV format, or store as JSON/binary blobs.

---

3. RGB Channel & Payment State (.ldk/ directory)

These are read/written by rust-lightning's RGB extensions (rgb_utils/mod.rs).

File Pattern	Description	Can Migrate?	Recommendation
{channel_id}	RGB channel info (contract, amounts)	No	rust-lightning reads directly
{channel_id}.pending	Pending RGB channel state	No	rust-lightning reads directly
{payment_hash}.inbound	Inbound RGB payment info	No	rust-lightning reads directly
{payment_hash}.outbound	Outbound RGB payment info	No	rust-lightning reads directly
consignment_*	RGB consignment files	No	rust-lightning reads directly
*_transfer_info	RGB transfer information	No	rust-lightning reads directly
psbt_*	Partially signed transactions	Temporary	Cleaned up after use

Verdict: These cannot be migrated without modifying rust-lightning. The rgb_utils/mod.rs module reads these files directly:

// rust-lightning/lightning/src/rgb_utils/mod.rs
let serialized_info = fs::read_to_string(rgb_channel_info_path).expect("valid rgb info file");

---

4. RGB Configuration Files (root storage directory)

These are currently in a hybrid state: database is source of truth, but files are synced for rust-lightning compatibility.

File	Description	In DB?	File Required?	Recommendation
indexer_url	Electrum indexer URL	Yes	Yes	Keep sync (rust-lightning reads)
proxy_endpoint	RGB proxy URL	Yes	Yes	Keep sync (rust-lightning reads)
bitcoin_network	Network (mainnet/testnet/regtest)	Yes	Yes	Keep sync (rust-lightning reads)
wallet_fingerprint	BIP32 wallet fingerprint	Yes	Yes	Keep sync (rust-lightning reads)
wallet_account_xpub_colored	xpub for colored assets	Yes	Yes	Keep sync (rust-lightning reads)
wallet_account_xpub_vanilla	xpub for BTC	Yes	Yes	Keep sync (rust-lightning reads)
wallet_master_fingerprint	Master key fingerprint	Yes	Yes	Keep sync (rust-lightning reads)

Verdict: Already correctly implemented. The database is the source of truth, and sync_rgb_config_to_files() keeps files in sync for rust-lightning.

rust-lightning reads these directly:

// rust-lightning/lightning/src/rgb_utils/mod.rs:129
fn _get_rgb_wallet_dir(ldk_data_dir: &Path) -> PathBuf {
    let fingerprint = _read_file_in_parent(ldk_data_dir, WALLET_FINGERPRINT_FNAME);
    _get_file_in_parent(ldk_data_dir, &fingerprint)
}

---

5. Logs

File	Description	Can Migrate?	Recommendation
.ldk/logs/logs.txt	LDK debug logs	No	Keep on filesystem
logs/rln.log.*	RLN application logs (rotating)	No	Keep on filesystem

Verdict: Logs should remain on filesystem. They're append-only, can grow large, and don't need to be backed up with node state.

---

6. RGB-Lib Wallet Data (managed by rgb-lib)

Path	Description	Can Migrate?	Recommendation
{wallet_fingerprint}/	RGB wallet directory	No	Managed by rgb-lib
RGB-lib SQLite DB	Asset contracts, transfers, UTXOs	No	Separate DB, managed by rgb-lib

Verdict: This is managed entirely by rgb-lib. You'd need rgb-lib changes to consolidate this.

---

Migration Priority

High Priority (Should Migrate)

These are RLN-specific data that you fully control:

1. inbound_payments - Payment tracking
2. outbound_payments - Payment tracking
3. channel_ids - Channel ID mapping
4. maker_swaps - Swap state
5. taker_swaps - Swap state
6. output_spender_txes - Transaction tracking

Suggested schema:

-- Option 1: Dedicated tables with proper columns
CREATE TABLE inbound_payments (
    payment_hash BLOB PRIMARY KEY,
    preimage BLOB,
    secret BLOB,
    status INTEGER,
    amt_msat INTEGER,
    created_at INTEGER,
    updated_at INTEGER,
    payee_pubkey BLOB
);

-- Option 2: Simple blob storage (preserves TLV format)
CREATE TABLE ldk_state (
    key TEXT PRIMARY KEY,
    value BLOB
);

Cannot Migrate (rust-lightning constraints)

These require files on disk because rust-lightning reads them directly:

• LDK core: manager, network_graph, scorer, monitors/*
• RGB state: {channel_id}, {payment_hash}.*, consignment_*
• RGB config files (but already synced from DB)

Should Not Migrate

• Logs (append-only, large, not critical state)

---

Backup Strategy

Given the constraints, a complete backup requires:

Option A: Database + LDK Directory

Backup contents:
├── rln_db                    # SQLite database (mnemonic, config, peers, payments, swaps)
└── .ldk/                     # LDK state directory
    ├── manager
    ├── network_graph
    ├── scorer
    ├── monitors/
    ├── {channel_id} files
    └── {payment_hash}.* files

Option B: Database + Selective File Backup

After migrating payments/swaps to DB:

Backup contents:
├── rln_db                    # Complete application state
└── .ldk/                     # Only LDK-required files
    ├── manager               # Critical
    ├── monitors/             # Critical
    ├── {channel_id}          # Critical (RGB)
    └── {payment_hash}.*      # Critical (RGB)

Files that can be regenerated (no backup needed):

• network_graph - Re-downloaded from network
• scorer - Rebuilt over time
• RGB config files - Synced from database

---

Implementation Roadmap

Phase 1: Migrate RLN Application State

1. Create new database tables for payments, swaps, channel IDs
2. Modify src/disk.rs read functions to query database instead of files
3. Modify src/ldk.rs write operations to save to database
4. Add migration logic to import existing file data on first run
5. Remove file-based storage after successful migration

Phase 2: Optimize Backup

1. Create backup command that exports only:
   • rln_db (complete database)
   • .ldk/manager (channel state)
   • .ldk/monitors/* (channel monitors)
   • RGB channel/payment files
2. Create restore command that imports backup and syncs config files

Phase 3: (Future) Coordinate with rust-lightning

If you need further consolidation, propose changes to rust-lightning:

• Abstract rgb_utils file access behind a trait
• Allow custom storage backends for RGB state

---

Summary Table

Data Category	Current Storage	Target Storage	Blocker
Mnemonic	Database	Database	None (done)
RGB Config	Database + Files	Database + Files	rust-lightning reads files
Channel Peers	Database	Database	None (done)
Revoked Tokens	Database	Database	None (done)
Payments (in/out)	Files	Database	None - migrate
Swaps	Files	Database	None - migrate
Channel IDs	Files	Database	None - migrate
Output Spender Txes	Files	Database	None - migrate
LDK Manager	Files	Files	rust-lightning
LDK Monitors	Files	Files	rust-lightning
Network Graph	Files	Files	rust-lightning (can regenerate)
Scorer	Files	Files	rust-lightning (can regenerate)
RGB Channel State	Files	Files	rust-lightning
RGB Payment State	Files	Files	rust-lightning
Logs	Files	Files	Should stay on filesystem
RGB-lib Wallet	Separate DB	Separate DB	rgb-lib

---

Conclusion

You can migrate 6 data types (payments, swaps, channel IDs, output spender txes) to the database, which will consolidate most RLN application state. However, LDK core state and RGB channel/payment state must remain on the filesystem due to rust-lightning's direct file access patterns.

For backups, the recommended approach is:

1. Migrate what you can to rln_db
2. Backup rln_db + critical .ldk/ files (manager, monitors, RGB state)
3. Skip regeneratable files (network_graph, scorer) and logs

[dcorral]

closing this in favor of: RGB-Tools#92