feat: Ingest dynamic relationships created

[afonsobspinto]

Closes https://metacell.atlassian.net/browse/SCKAN-429

• Adds custom_ingestion_code property to relationships

• Adds ingestion admin page

• Adds ingestion logs download admin page

• Adds processing relationship custom code as one new step in the transform neurondm neuron stop.

• Splits ingestion in 3 steps (get relationships data, transform neurondm neuron, ingest into the database). Step 1 and 3 have their dedicated django command.

• Creates new docker image for isolated, django independent, neurodm transform (step 2)

• Adds serialization and deserialization to the models used on the neurondm transform the data in argo moves between steps via json files in a shared volume.

• Moves ingestion to argo workflow

• Updates notify task to send different emails depending on the argo task being exited (ingestion vs export)

• Adds new django command to delete temporary old files (30 days old by default) from the ingestion pipeline

• Adds bash script to setup a cron job to run the django command aforementioned every day

2025-10-16.11-46-06.mp4

2025-10-18.15-34-24.mp4

Deployment notes:

• Requires updating cloudharness to release/3.0.0 (previous version was feature/ch-158)

Misc:

• Moves population file filter one step sooner to prevent neurondm processing of non mentioned populations.
• Improves performance of admin page by changing filter_horizontal to raw_id_fields
• Fixes Incorrect timezone setting: %s" % self.TIME_ZONE by changing timezone setting
• Removes silk debug app (as we also have django debug toolbar)
• Updates ingest_statements function for local testing purposes if needed but it's no longer used.
• Splits enums (text choices) and pure_enums to ease the creation of a django independent neurondm script
• Bumps django filters version

[Copilot]

Pull Request Overview

This PR implements a comprehensive ingestion system for dynamic relationships from neurondm, split into a 3-step workflow:

• Step 0: Extract custom relationship definitions from database
• Step 1: Process neurondm data and execute custom code in a lightweight container
• Step 2: Ingest processed data into the database with Django

Key changes include:

• New 3-step Argo workflow architecture with composer-neurondm task container
• Custom ingestion code field on Relationship model for dynamic data extraction
• Serialization/deserialization support for NeuroDM objects to JSON
• Population filtering moved to Step 1 for efficiency
• UI components for ingestion configuration and log file downloads

Reviewed Changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
deployment/codefresh-dev.yaml	Adds composer-neurondm build steps and restructures parallel builds
applications/composer/tasks/notify/notify.py	Extends notification support for ingestion workflows
applications/composer/tasks/neurondm/process_neurondm_standalone.py	New standalone script for Step 1 neurondm processing
applications/composer/tasks/neurondm/Dockerfile	Lightweight container for neurondm processing
applications/composer/backend/composer/models.py	Adds custom_ingestion_code field to Relationship
applications/composer/backend/composer/services/cs_ingestion/neurondm_script.py	Adds custom relationship processing in Step 1
applications/composer/backend/composer/services/cs_ingestion/models.py	Serialization/deserialization for NeuroDM objects
applications/composer/backend/composer/services/cs_ingestion/helpers/statement_helper.py	Processes custom relationships in Step 2
applications/composer/backend/composer/pure_enums.py	Django-independent enum definitions
applications/composer/backend/composer/constants.py	Ingestion file paths configuration
applications/composer/backend/composer/admin.py	UI for custom ingestion code and log downloads

Comments suppressed due to low confidence (1)

applications/composer/backend/composer/services/cs_ingestion/helpers/statement_helper.py:1

• [nitpick] The tuple contains a trailing comma making it a tuple, but this is in the admin.py file (line 318), not statement_helper.py. However, if this is intentional migration from filter_horizontal to raw_id_fields for performance reasons, ensure that the UI still provides adequate usability for selecting multiple triples.

import re

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}