schedule old production books to be deleted

[elfkuzco]

Rationale

This PR enhances the mill to apply retention rules to prod titles. It groups the prod titles by flavour before applying the rules to determine which files to mark as deleted. Consequently, a new background task is added to the shuttle which actually deletes the files that have been marked for deletion

Chanages

• add new column deletion_date to set the date the file should be deleted
• apply retention rules to files
• add background task to delete files that have been marked for deletion

This closes #141

[codecov]

Codecov Report

❌ Patch coverage is 96.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.06%. Comparing base (12e50ab) to head (f2d2fe9).
⚠️ Report is 5 commits behind head on cms-upgrade.

Files with missing lines	Patch %	Lines
backend/src/cms_backend/mill/processors/title.py	95.00%	1 Missing and 1 partial ⚠️
backend/src/cms_backend/shuttle/delete_files.py	97.91%	0 Missing and 1 partial ⚠️
backend/src/cms_backend/shuttle/main.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##           cms-upgrade     #149      +/-   ##
===============================================
+ Coverage        77.05%   79.06%   +2.00%     
===============================================
  Files               40       41       +1     
  Lines             1386     1533     +147     
  Branches           119      139      +20     
===============================================
+ Hits              1068     1212     +144     
- Misses             280      281       +1     
- Partials            38       40       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[benoit74]

Looks pretty good, I just think there is a bug in book sorting (unless I'm mistaken, but this is not tested so better to at least fix the test to cover it, probably fix the code). Situation to consider is that we could have multiple letters in the period, like 2021_01aa. And 2021_01a <2021_01b < 2021_01c < ... < 2021_01aa < 2021_01bg which is probably not what happens with the natural alphabetical sorting the code is using. Having two letters suffix is going to be extremely rare, but we need to take that into account. Sort expression which is already used in compute_target_filename is suffixes.sort(key=lambda s: (len(s), s))

[benoit74]

I just noticed it looks like we should fix get_next_book_to_move_files_or_none to consider only books which location_kind is not to_delete (or deleted).

[benoit74]

LGTM