Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
152 commits
Select commit Hold shift + click to select a range
69de5d5
Deploy: removed old .env ref.
vladd-bit Jul 4, 2025
4e950f8
Deploy: change nifi cert mount.
vladd-bit Jul 4, 2025
480347e
NiFi: Changed cert paths.
vladd-bit Jul 4, 2025
a28b42c
Deploy: updated nginx settings to accomodate for nifi-registry.
vladd-bit Jul 6, 2025
0f14151
Security: updated ES certs.
vladd-bit Jul 7, 2025
64235b1
NiFi: scripting cleanup + added new processors for avro binary doc fi…
vladd-bit Jul 7, 2025
310d25b
NiFi: updated available processors, added mime type + conversion of f…
vladd-bit Jul 8, 2025
c35be46
NiFi: updated base64 conversion proc.
vladd-bit Jul 8, 2025
8187128
NiFi: fixed schema issues when converting base64 to string type.
vladd-bit Jul 8, 2025
baa4319
NiFi: script documentation update.
vladd-bit Jul 8, 2025
889425e
NiFi: updated decompression script.
vladd-bit Jul 8, 2025
c429b9d
NiFi: updated decompression script.
vladd-bit Jul 9, 2025
564adf0
NiFi: updated decompression script.
vladd-bit Jul 9, 2025
8c3f8d8
NiFi: script updates.
vladd-bit Jul 9, 2025
8a1a362
NiFi scripts: removed redundant decoding.
vladd-bit Jul 9, 2025
60f24d5
NiFi: rename script.
vladd-bit Jul 9, 2025
0714f4d
NiFi: updated cerner decompression script.
vladd-bit Jul 9, 2025
164f472
NiFi: updated cerner decompression script.
vladd-bit Jul 9, 2025
2332064
NiFi Scripts: added script to parse cg service update.
vladd-bit Jul 11, 2025
222ce93
NiFi: updated conten repository arcbive settings (more flexible).
vladd-bit Jul 11, 2025
e26e514
Updated python pkg requirements in for nifi.
vladd-bit Jul 11, 2025
a9108c3
Deploy: updated service defs. Created separate compose file for dev env.
vladd-bit Jul 11, 2025
6f873ee
Deploy:ocr-service service ref update.
vladd-bit Jul 15, 2025
00bc016
NiFi: script updates.
vladd-bit Jul 15, 2025
0b024ee
NiFi Scripts: updated decompression processor.
vladd-bit Jul 21, 2025
5a8a1d5
Script updates.
vladd-bit Jul 22, 2025
c88fa82
Script updates.
vladd-bit Jul 22, 2025
deff116
NiFi: Cerner script updates.
vladd-bit Jul 22, 2025
cc38fef
NiFi: Cerner script updates.
vladd-bit Jul 22, 2025
66a8cbf
NiFi Scripts: added json decoding options.
vladd-bit Jul 22, 2025
4574031
NiFi: updated content repository properties.
vladd-bit Jul 30, 2025
6d25e42
Removed old submodules.
vladd-bit Jul 30, 2025
d8f7331
Removed old macOS remnants (DS_Store) files.
vladd-bit Jul 30, 2025
64ca7f0
Services: removed old j-hub refs.
vladd-bit Jul 30, 2025
7e0a7ee
Services: update kibana native image.
vladd-bit Jul 30, 2025
5961c9e
Services: pg samples db script update.
vladd-bit Jul 30, 2025
cc3f8e5
Squashed 'services/jupyter-hub/' content from commit 8a67f138
vladd-bit Jul 30, 2025
68d4b20
Merge commit 'cc3f8e59af70ef6135dbc15ee8cbe82f96fdb26c' as 'services/…
vladd-bit Jul 30, 2025
44972d8
Removed subtree.
vladd-bit Jul 30, 2025
0deb9bb
Updated submodules.
vladd-bit Jul 30, 2025
fd5437c
Updated workflows.
vladd-bit Jul 30, 2025
8f0ffd2
Scripts: added submodule update script.
vladd-bit Jul 30, 2025
2b6f60a
Scripts: added submodule update script.
vladd-bit Jul 30, 2025
c86e01b
Script: submodule script updates.
vladd-bit Jul 31, 2025
75163dd
Script: submodule script updates (fixed base url).
vladd-bit Jul 31, 2025
a75f3de
Scripts:updated submodule related scripts.
vladd-bit Jul 31, 2025
cdc0ad5
NiFi: script updates.
vladd-bit Aug 1, 2025
3864424
Added ocr-service to submodules.
vladd-bit Aug 2, 2025
d0e86da
Undo ocr-service submodule add.
vladd-bit Aug 2, 2025
4d7b711
Services: re-added ocr-service submodule.
vladd-bit Aug 2, 2025
c341bba
Services: updated pg samples (simplified db). Deploy: updated service…
vladd-bit Aug 2, 2025
e56e851
Deploy: removed old env file defs from Makefile.
vladd-bit Aug 2, 2025
f6fe499
NiFi: Cleanup old user templates | Services: updated OpenS to 3.1.0.
vladd-bit Aug 5, 2025
0576038
NiFi: moved new temapltes to root template folder.
vladd-bit Aug 5, 2025
90075db
NiFi: added custom boostrab python lib import + avro json decoder.
vladd-bit Aug 5, 2025
3d19ff9
NiFi: processor updates.
vladd-bit Aug 5, 2025
94ca904
Deploy: gitea.env added
vladd-bit Aug 6, 2025
e798ff0
Services: removed trainer submodule.
vladd-bit Aug 7, 2025
9d0b62b
NiFi: updated regex cleanup script for de-id.
vladd-bit Aug 7, 2025
6050b42
NiFi: script updates.
vladd-bit Aug 7, 2025
8bc04f4
NiFi: moved legacy scripts that are still relevant to current script …
vladd-bit Aug 7, 2025
44d83db
Remove submodule services/jupyter-hub
vladd-bit Aug 14, 2025
6518810
Re-added j-hub service.
vladd-bit Aug 14, 2025
71c712c
scripts: added gitea scripts to support produciton deployments.
vladd-bit Aug 14, 2025
91ec9af
deploy: general env update.
vladd-bit Aug 14, 2025
b813018
Deploy: removed refs for some services in services.yml, env var updat…
vladd-bit Aug 18, 2025
f7562ed
NiFi scripts: Updated ann ingestion script with extra deid param.
vladd-bit Aug 18, 2025
b09efd0
Updated .gitmodules.
vladd-bit Aug 20, 2025
a145b3f
Updated Makefile + script.
vladd-bit Aug 21, 2025
20431ca
Deploy: updated scripts + Makefile.
vladd-bit Aug 21, 2025
09dd78d
Deploy: script export update.
vladd-bit Aug 21, 2025
c05c68a
Fixed paths for MCTrainer.
vladd-bit Aug 22, 2025
f85f406
Updated submodule update script.
vladd-bit Aug 27, 2025
e52215e
Update submodules to latest release tags (or main)
vladd-bit Aug 27, 2025
79a2c73
Updated github actions.
vladd-bit Aug 28, 2025
08f06be
Services: ElasticSearch native dockerfile updates.
vladd-bit Aug 28, 2025
91d1939
GHA updates.
vladd-bit Aug 28, 2025
bc05e9f
Update submodules to latest release tags (or main)
vladd-bit Aug 28, 2025
842256f
GHA: Updated submodule sync to point to latest release instead.
vladd-bit Aug 28, 2025
92c303c
GHA: Elasticsearch workflow fix.
vladd-bit Aug 28, 2025
d51534c
GHA: ES build fixes.
vladd-bit Aug 28, 2025
7f28959
Services: Gitea ini update.
vladd-bit Aug 30, 2025
7c681b0
Scripts: updated gitea util scripts.
vladd-bit Aug 30, 2025
919d4b4
Services: gitea config update.
vladd-bit Aug 30, 2025
637e2ab
NiFi: updated to 2.5.0, docker image cleanup.
vladd-bit Sep 1, 2025
2242d8d
NiFi: dockerfile update.
vladd-bit Sep 1, 2025
88b8a5a
Deploy/GHA: nifi docker build + make file shortcuts update.
vladd-bit Sep 1, 2025
44cd671
GHA: nifi docker build fix.
vladd-bit Sep 1, 2025
1142a9a
NiFi: updated processor (added built-in validation)
vladd-bit Sep 1, 2025
30d23a2
NiFi scripts: more field validation to servce response parser.
vladd-bit Sep 1, 2025
e319577
Services: gitea updates.
vladd-bit Sep 2, 2025
6adf186
Scripts: submodule update updates.
vladd-bit Sep 2, 2025
14803f9
Deploy: MCT trainer ref corrections.
vladd-bit Sep 2, 2025
f8bfb88
NiFi: docker image update(prefer cpu pkgs).
vladd-bit Sep 2, 2025
eda5a59
NiFi: updated Docker image.
vladd-bit Sep 2, 2025
84d5528
Scripts: moved old legacy scripts.
vladd-bit Sep 2, 2025
39ed16b
NiFi: added extra pip flag for build speedup.
vladd-bit Sep 2, 2025
348b54a
NiFi: dockerfile pkg cleannup.
vladd-bit Sep 3, 2025
40c0859
Security: cert gen scripts cleanup and update.
vladd-bit Sep 3, 2025
4ce87b7
Security: bash script updates.
vladd-bit Sep 4, 2025
0cdc613
Readme + License markdown & formatting update.
vladd-bit Sep 4, 2025
60418b1
Deploy: removed redundant example script.
vladd-bit Sep 4, 2025
5bb60cf
Deploy: services.yml cleanup cogstack-cohort.
vladd-bit Sep 4, 2025
db9956e
GHA: nifi workflow update.
vladd-bit Sep 4, 2025
e910943
.gitignore update.
vladd-bit Sep 4, 2025
91e10ae
Deploy/Scripts: updated OpenSearch to 3.2.0, updated Elastic native c…
vladd-bit Sep 4, 2025
0228345
Deploy/Scripts: added util scripts makefile shortcuts for freezeing/u…
vladd-bit Sep 4, 2025
064782f
Deploy: makefile corrections.
vladd-bit Sep 4, 2025
ccebf18
Deploy: makefile improvements.
vladd-bit Sep 4, 2025
18af102
NiFi: updated dockerfile + scripts.
vladd-bit Sep 4, 2025
d13503d
.gitignore updated.
vladd-bit Sep 4, 2025
a1433ce
Deploy: makefile + nginx updates + removed older trainer refs in serv…
vladd-bit Sep 16, 2025
08d0464
Scripts: removed redundant old shell scripts.
vladd-bit Sep 16, 2025
c0974c9
NiFi scripts: updated field validation for all processors.
vladd-bit Sep 17, 2025
f4d13ae
Cert updates.
vladd-bit Sep 20, 2025
3a8d42d
Restructured security folder, updated certificates with correct gen c…
vladd-bit Sep 20, 2025
aa14eb2
Scripts: fixed paths.
vladd-bit Sep 21, 2025
e4cbf9e
Deploy/security: path ref fixes in scripts/compose files.
vladd-bit Sep 21, 2025
9efc513
NiFi toolkit updates.
vladd-bit Sep 21, 2025
6c5e9e4
Cert updates.
vladd-bit Sep 22, 2025
71a26f8
Removed cert dupes.
vladd-bit Sep 22, 2025
c42848d
NiFi cert updates.
vladd-bit Sep 22, 2025
c2e8375
Deploy: updated makefile (stop commands).
vladd-bit Sep 22, 2025
040377d
Security: cert updates (sh script fixes).
vladd-bit Sep 22, 2025
40f4425
Security: corrected cert gen for nifi+root-ca, unified SAN settings.
vladd-bit Sep 23, 2025
e9eed76
Security: ES_native script updates.
vladd-bit Sep 24, 2025
c6c65ed
Security: updates native ES security certs + gen scripts.
vladd-bit Sep 24, 2025
fa3963c
Docs: updated tls certif docs.
vladd-bit Sep 24, 2025
31de649
NiFi: moved old schemas to legacy folder, only kept activley used one…
vladd-bit Sep 25, 2025
6fa2db5
NiFi: updated available processors (comments + type lint, etc.)
vladd-bit Sep 25, 2025
34b1e30
Scripts: updated git scripts to reflect recent security dir changes.
vladd-bit Sep 26, 2025
f52b978
Deploy: services.yml cleanup.
vladd-bit Sep 26, 2025
ac387e5
Deploy: services.yml cleanup II.
vladd-bit Sep 26, 2025
4b8715a
Deploy: env files updates (shm sizes).
vladd-bit Sep 26, 2025
1b7e948
Security: updated cert paths to match new service.yml definitions.
vladd-bit Sep 26, 2025
629a9d3
Deploy: services def update (volume issues).
vladd-bit Sep 26, 2025
92f4b5e
Deploy: docker logging configs.
vladd-bit Oct 3, 2025
b783653
Services: updated sameples-db script to include extra MIMIC-III tables.
vladd-bit Oct 3, 2025
1085d2e
NiFi: updated to 2.6.0.
vladd-bit Oct 7, 2025
253a313
Deploy: makefile update (fixed stop commands, missing env var refs)
vladd-bit Oct 7, 2025
1cee044
Deploy: NiFi registry update ref.
vladd-bit Oct 7, 2025
2a2a64d
Added dependabot.
vladd-bit Oct 7, 2025
ffa278a
Merge branch 'main' of https://github.com/CogStack/CogStack-NiFi into…
vladd-bit Oct 7, 2025
918c512
Updated dependabot.
vladd-bit Oct 7, 2025
de94347
Updated dependabot II.
vladd-bit Oct 7, 2025
b138cad
GHA: nifi build updates.
vladd-bit Oct 7, 2025
0aee023
GHA: fixed nifi docker build manifest.
vladd-bit Oct 7, 2025
4d9b863
GHA: workflow updatees (added template for docker builds).
vladd-bit Oct 8, 2025
38840dd
GHA: fixed secret dockerhub login.
vladd-bit Oct 8, 2025
1e2dc41
GHA: nifi nginx docker build update.
vladd-bit Oct 8, 2025
d17f0f2
GHA: updated all workflows with permissions.
vladd-bit Oct 8, 2025
0b65390
Deploy: removed old refs to images that still used CPU_ARCH env.
vladd-bit Oct 8, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
32 changes: 0 additions & 32 deletions .flake8

This file was deleted.

22 changes: 22 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
version: 2
updates:
# Keep GitHub Actions (in .github/workflows) up to date
- package-ecosystem: "github-actions"
directory: "/" # this is correct even if workflows are in .github/workflows
schedule:
interval: "weekly"

# Keep Dockerfile base images (in repo root) up to date
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
registries:
- dockerhub

registries:
dockerhub:
type: "docker-registry"
url: "https://index.docker.io"
username: "${{secrets.DOCKER_HUB_USERNAME}}"
password: "${{secrets.DOCKER_HUB_ACCESS_TOKEN}}"
131 changes: 131 additions & 0 deletions .github/workflows/_docker-template.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
name: Docker Multi-Arch Template

permissions:
contents: read

on:
workflow_call:
inputs:
image:
description: "Full image name, e.g. cogstacksystems/cogstack-nifi"
required: true
type: string
context:
description: "Docker build context path"
required: true
type: string
dockerfile:
description: "Path to Dockerfile"
required: true
type: string
cache_scope:
description: "Cache key (unique per image)"
required: true
type: string
build_args:
description: "Optional Docker build args (newline-separated)"
required: false
default: ""
type: string

concurrency:
group: docker-${{ inputs.image }}-${{ github.ref }}
cancel-in-progress: true

jobs:
# -------- Build per-architecture ----------
build:
runs-on: ${{ matrix.runner }}
strategy:
matrix:
platform: [amd64, arm64]
include:
- platform: amd64
runner: ubuntu-22.04
- platform: arm64
runner: ubuntu-22.04-arm
outputs:
digest: ${{ steps.push.outputs.digest }}

steps:
- uses: actions/checkout@v5
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3

- if: github.event_name != 'pull_request'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

- name: Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ inputs.image }}
tags: |
# vX.Y.Z -> X.Y.Z and X.Y (strip leading "v")
type=semver,pattern={{version}},prefix=v
type=semver,pattern={{major}}.{{minor}},prefix=v
# latest on main and on v-tags
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') }}
# branch tag for non-PR branches
type=ref,event=branch,enable=${{ github.event_name != 'pull_request' }}
# short sha
type=sha,format=short

- name: Build & push (${{ matrix.platform }})
id: push
uses: docker/build-push-action@v6
with:
context: ${{ inputs.context }}
file: ${{ inputs.dockerfile }}
platforms: linux/${{ matrix.platform }}
build-args: ${{ inputs.build_args }}
tags: |
${{ steps.meta.outputs.tags }}-${{ matrix.platform }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=${{ inputs.cache_scope }}-${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ inputs.cache_scope }}-${{ matrix.platform }}
push: ${{ github.event_name != 'pull_request' }}
provenance: false

# -------- Merge both architectures ----------
manifest:
Comment thread Fixed
runs-on: ubuntu-22.04
if: github.event_name != 'pull_request'
needs: build

steps:
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

- name: Docker metadata (again)
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ inputs.image }}
tags: |
type=semver,pattern={{version}},prefix=v
type=semver,pattern={{major}}.{{minor}},prefix=v
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') }}
type=ref,event=branch,enable=${{ github.event_name != 'pull_request' }}
type=sha,format=short

- name: Create multi-arch manifest
shell: bash
run: |
set -euo pipefail
while IFS= read -r ref; do
[[ -z "$ref" ]] && continue
img="${ref%%:*}"
tag="${ref#*:}"
echo "🌀 Creating manifest for ${img}:${tag}"
docker buildx imagetools create \
--tag "${img}:${tag}" \
"${img}:${tag}-amd64" \
"${img}:${tag}-arm64"
done < <(printf "%s" "${{ steps.meta.outputs.tags }}")
Comment thread Fixed
10 changes: 4 additions & 6 deletions .github/workflows/doc-build.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
name: sphinx-doc-build

on:

push:
branches: [ main ]
pull_request:
Expand All @@ -10,21 +9,20 @@ on:
release:
types: [published]

# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:

jobs:
permissions:
contents: read

jobs:
build:

runs-on: ubuntu-latest

env:
working-directory: ./docs

# Steps represent a sequence of tasks that will be executed as part of the job
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v5

- name: Build docs
run: |
Expand Down
69 changes: 0 additions & 69 deletions .github/workflows/docker-dev.yml

This file was deleted.

Loading
Loading