Database hygiene

[granawkins]

In some cases the database will contain outdated records, and the only way to 'update' it is to remove the database (~/.mentat/chroma). One case in particular is if you change between chunker_line and chunker_llm, it won't update the db record.

Potential solutions:

• Add more fields or a checksum to the collection name
• Add chunker-type-specific data so 'Chunker[type].is_complete' checks for a record of that type specifically
• Refresh the database every time 'version' is updated.

[granawkins]

If you cancel while it's chunking, it leaves a corrupt record

[granawkins]

New plan:

• db record fields should be annotator-specific, e.g. 'chunks_llm' or 'chunks_line', so they can maintain a single record.
• put everything in a single collection, tagged to ragdaemon minor version
• add db scripts to backup, inspect, and recover from corrupted records (go thru all records and validate one-by-one).

Conventions to follow:

• A file/chunk record should only depend on itself, not any other nodes. This way if two repos have the same doc (like an empty src/__init__.py, using the same db record is fine.