feat: enrich zetkin data sources nightly#428
Conversation
There was a problem hiding this comment.
Pull request overview
Adds nightly enrichment scheduling for Zetkin-backed data sources by enqueueing an enrichDataSource job after the nightly Zetkin import job is enqueued.
Changes:
- After enqueueing
importDataSourcefor each Zetkin data source, conditionally enqueueenrichDataSourcewhen the data source has enrichments configured. - Add warning logging if the enrichment job cannot be enqueued.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (source.enrichments.length > 0) { | ||
| try { | ||
| await enqueue("enrichDataSource", source.id, { | ||
| dataSourceId: source.id, | ||
| }); |
There was a problem hiding this comment.
The new enrichment enqueue bypasses the autoEnrich flag on the data source. As written, any Zetkin source with enrichments.length > 0 will get an enrichDataSource job every night even if the user has disabled automatic enrichment. Consider including source.autoEnrich in the condition (or otherwise documenting/encoding why Zetkin should ignore that setting).
There was a problem hiding this comment.
@copilot apply changes based on this feedback
There was a problem hiding this comment.
Fixed in 4011ba7 — the condition now checks source.autoEnrich && source.enrichments.length > 0 so the nightly enrichment job is only enqueued when automatic enrichment is enabled.
| if (source.enrichments.length > 0) { | ||
| try { | ||
| await enqueue("enrichDataSource", source.id, { | ||
| dataSourceId: source.id, | ||
| }); |
There was a problem hiding this comment.
This change adds new behavior (conditionally enqueueing enrichDataSource) but there are no tests covering importZetkinDataSources. Since other job modules have automated tests (e.g. tests/feature/importDataSource.test.ts, tests/feature/enrichDataSource.test.ts), consider adding a unit/feature test that verifies enqueue() is called for enrichment only for sources that should be enriched (and that enqueue failures are handled as expected).
…etkin sources Agent-Logs-Url: https://github.com/commonknowledge/ts-mapped/sessions/2980f2de-1bc4-4c39-96ff-5dea72016cf7 Co-authored-by: joaquimds <12935136+joaquimds@users.noreply.github.com>
No description provided.