-
Notifications
You must be signed in to change notification settings - Fork 22
Nifi 2.0 script & workflow updates #31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from all commits
Commits
Show all changes
152 commits
Select commit
Hold shift + click to select a range
69de5d5
Deploy: removed old .env ref.
vladd-bit 4e950f8
Deploy: change nifi cert mount.
vladd-bit 480347e
NiFi: Changed cert paths.
vladd-bit a28b42c
Deploy: updated nginx settings to accomodate for nifi-registry.
vladd-bit 0f14151
Security: updated ES certs.
vladd-bit 64235b1
NiFi: scripting cleanup + added new processors for avro binary doc fi…
vladd-bit 310d25b
NiFi: updated available processors, added mime type + conversion of f…
vladd-bit c35be46
NiFi: updated base64 conversion proc.
vladd-bit 8187128
NiFi: fixed schema issues when converting base64 to string type.
vladd-bit baa4319
NiFi: script documentation update.
vladd-bit 889425e
NiFi: updated decompression script.
vladd-bit c429b9d
NiFi: updated decompression script.
vladd-bit 564adf0
NiFi: updated decompression script.
vladd-bit 8c3f8d8
NiFi: script updates.
vladd-bit 8a1a362
NiFi scripts: removed redundant decoding.
vladd-bit 60f24d5
NiFi: rename script.
vladd-bit 0714f4d
NiFi: updated cerner decompression script.
vladd-bit 164f472
NiFi: updated cerner decompression script.
vladd-bit 2332064
NiFi Scripts: added script to parse cg service update.
vladd-bit 222ce93
NiFi: updated conten repository arcbive settings (more flexible).
vladd-bit e26e514
Updated python pkg requirements in for nifi.
vladd-bit a9108c3
Deploy: updated service defs. Created separate compose file for dev env.
vladd-bit 6f873ee
Deploy:ocr-service service ref update.
vladd-bit 00bc016
NiFi: script updates.
vladd-bit 0b024ee
NiFi Scripts: updated decompression processor.
vladd-bit 5a8a1d5
Script updates.
vladd-bit c88fa82
Script updates.
vladd-bit deff116
NiFi: Cerner script updates.
vladd-bit cc38fef
NiFi: Cerner script updates.
vladd-bit 66a8cbf
NiFi Scripts: added json decoding options.
vladd-bit 4574031
NiFi: updated content repository properties.
vladd-bit 6d25e42
Removed old submodules.
vladd-bit d8f7331
Removed old macOS remnants (DS_Store) files.
vladd-bit 64ca7f0
Services: removed old j-hub refs.
vladd-bit 7e0a7ee
Services: update kibana native image.
vladd-bit 5961c9e
Services: pg samples db script update.
vladd-bit cc3f8e5
Squashed 'services/jupyter-hub/' content from commit 8a67f138
vladd-bit 68d4b20
Merge commit 'cc3f8e59af70ef6135dbc15ee8cbe82f96fdb26c' as 'services/…
vladd-bit 44972d8
Removed subtree.
vladd-bit 0deb9bb
Updated submodules.
vladd-bit fd5437c
Updated workflows.
vladd-bit 8f0ffd2
Scripts: added submodule update script.
vladd-bit 2b6f60a
Scripts: added submodule update script.
vladd-bit c86e01b
Script: submodule script updates.
vladd-bit 75163dd
Script: submodule script updates (fixed base url).
vladd-bit a75f3de
Scripts:updated submodule related scripts.
vladd-bit cdc0ad5
NiFi: script updates.
vladd-bit 3864424
Added ocr-service to submodules.
vladd-bit d0e86da
Undo ocr-service submodule add.
vladd-bit 4d7b711
Services: re-added ocr-service submodule.
vladd-bit c341bba
Services: updated pg samples (simplified db). Deploy: updated service…
vladd-bit e56e851
Deploy: removed old env file defs from Makefile.
vladd-bit f6fe499
NiFi: Cleanup old user templates | Services: updated OpenS to 3.1.0.
vladd-bit 0576038
NiFi: moved new temapltes to root template folder.
vladd-bit 90075db
NiFi: added custom boostrab python lib import + avro json decoder.
vladd-bit 3d19ff9
NiFi: processor updates.
vladd-bit 94ca904
Deploy: gitea.env added
vladd-bit e798ff0
Services: removed trainer submodule.
vladd-bit 9d0b62b
NiFi: updated regex cleanup script for de-id.
vladd-bit 6050b42
NiFi: script updates.
vladd-bit 8bc04f4
NiFi: moved legacy scripts that are still relevant to current script …
vladd-bit 44d83db
Remove submodule services/jupyter-hub
vladd-bit 6518810
Re-added j-hub service.
vladd-bit 71c712c
scripts: added gitea scripts to support produciton deployments.
vladd-bit 91ec9af
deploy: general env update.
vladd-bit b813018
Deploy: removed refs for some services in services.yml, env var updat…
vladd-bit f7562ed
NiFi scripts: Updated ann ingestion script with extra deid param.
vladd-bit b09efd0
Updated .gitmodules.
vladd-bit a145b3f
Updated Makefile + script.
vladd-bit 20431ca
Deploy: updated scripts + Makefile.
vladd-bit 09dd78d
Deploy: script export update.
vladd-bit c05c68a
Fixed paths for MCTrainer.
vladd-bit f85f406
Updated submodule update script.
vladd-bit e52215e
Update submodules to latest release tags (or main)
vladd-bit 79a2c73
Updated github actions.
vladd-bit 08f06be
Services: ElasticSearch native dockerfile updates.
vladd-bit 91d1939
GHA updates.
vladd-bit bc05e9f
Update submodules to latest release tags (or main)
vladd-bit 842256f
GHA: Updated submodule sync to point to latest release instead.
vladd-bit 92c303c
GHA: Elasticsearch workflow fix.
vladd-bit d51534c
GHA: ES build fixes.
vladd-bit 7f28959
Services: Gitea ini update.
vladd-bit 7c681b0
Scripts: updated gitea util scripts.
vladd-bit 919d4b4
Services: gitea config update.
vladd-bit 637e2ab
NiFi: updated to 2.5.0, docker image cleanup.
vladd-bit 2242d8d
NiFi: dockerfile update.
vladd-bit 88b8a5a
Deploy/GHA: nifi docker build + make file shortcuts update.
vladd-bit 44cd671
GHA: nifi docker build fix.
vladd-bit 1142a9a
NiFi: updated processor (added built-in validation)
vladd-bit 30d23a2
NiFi scripts: more field validation to servce response parser.
vladd-bit e319577
Services: gitea updates.
vladd-bit 6adf186
Scripts: submodule update updates.
vladd-bit 14803f9
Deploy: MCT trainer ref corrections.
vladd-bit f8bfb88
NiFi: docker image update(prefer cpu pkgs).
vladd-bit eda5a59
NiFi: updated Docker image.
vladd-bit 84d5528
Scripts: moved old legacy scripts.
vladd-bit 39ed16b
NiFi: added extra pip flag for build speedup.
vladd-bit 348b54a
NiFi: dockerfile pkg cleannup.
vladd-bit 40c0859
Security: cert gen scripts cleanup and update.
vladd-bit 4ce87b7
Security: bash script updates.
vladd-bit 0cdc613
Readme + License markdown & formatting update.
vladd-bit 60418b1
Deploy: removed redundant example script.
vladd-bit 5bb60cf
Deploy: services.yml cleanup cogstack-cohort.
vladd-bit db9956e
GHA: nifi workflow update.
vladd-bit e910943
.gitignore update.
vladd-bit 91e10ae
Deploy/Scripts: updated OpenSearch to 3.2.0, updated Elastic native c…
vladd-bit 0228345
Deploy/Scripts: added util scripts makefile shortcuts for freezeing/u…
vladd-bit 064782f
Deploy: makefile corrections.
vladd-bit ccebf18
Deploy: makefile improvements.
vladd-bit 18af102
NiFi: updated dockerfile + scripts.
vladd-bit d13503d
.gitignore updated.
vladd-bit a1433ce
Deploy: makefile + nginx updates + removed older trainer refs in serv…
vladd-bit 08d0464
Scripts: removed redundant old shell scripts.
vladd-bit c0974c9
NiFi scripts: updated field validation for all processors.
vladd-bit f4d13ae
Cert updates.
vladd-bit 3a8d42d
Restructured security folder, updated certificates with correct gen c…
vladd-bit aa14eb2
Scripts: fixed paths.
vladd-bit e4cbf9e
Deploy/security: path ref fixes in scripts/compose files.
vladd-bit 9efc513
NiFi toolkit updates.
vladd-bit 6c5e9e4
Cert updates.
vladd-bit 71a26f8
Removed cert dupes.
vladd-bit c42848d
NiFi cert updates.
vladd-bit c2e8375
Deploy: updated makefile (stop commands).
vladd-bit 040377d
Security: cert updates (sh script fixes).
vladd-bit 40f4425
Security: corrected cert gen for nifi+root-ca, unified SAN settings.
vladd-bit e9eed76
Security: ES_native script updates.
vladd-bit c6c65ed
Security: updates native ES security certs + gen scripts.
vladd-bit fa3963c
Docs: updated tls certif docs.
vladd-bit 31de649
NiFi: moved old schemas to legacy folder, only kept activley used one…
vladd-bit 6fa2db5
NiFi: updated available processors (comments + type lint, etc.)
vladd-bit 34b1e30
Scripts: updated git scripts to reflect recent security dir changes.
vladd-bit f52b978
Deploy: services.yml cleanup.
vladd-bit ac387e5
Deploy: services.yml cleanup II.
vladd-bit 4b8715a
Deploy: env files updates (shm sizes).
vladd-bit 1b7e948
Security: updated cert paths to match new service.yml definitions.
vladd-bit 629a9d3
Deploy: services def update (volume issues).
vladd-bit 92f4b5e
Deploy: docker logging configs.
vladd-bit b783653
Services: updated sameples-db script to include extra MIMIC-III tables.
vladd-bit 1085d2e
NiFi: updated to 2.6.0.
vladd-bit 253a313
Deploy: makefile update (fixed stop commands, missing env var refs)
vladd-bit 1cee044
Deploy: NiFi registry update ref.
vladd-bit 2a2a64d
Added dependabot.
vladd-bit ffa278a
Merge branch 'main' of https://github.com/CogStack/CogStack-NiFi into…
vladd-bit 918c512
Updated dependabot.
vladd-bit de94347
Updated dependabot II.
vladd-bit b138cad
GHA: nifi build updates.
vladd-bit 0aee023
GHA: fixed nifi docker build manifest.
vladd-bit 4d9b863
GHA: workflow updatees (added template for docker builds).
vladd-bit 38840dd
GHA: fixed secret dockerhub login.
vladd-bit 1e2dc41
GHA: nifi nginx docker build update.
vladd-bit d17f0f2
GHA: updated all workflows with permissions.
vladd-bit 0b65390
Deploy: removed old refs to images that still used CPU_ARCH env.
vladd-bit File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| version: 2 | ||
| updates: | ||
| # Keep GitHub Actions (in .github/workflows) up to date | ||
| - package-ecosystem: "github-actions" | ||
| directory: "/" # this is correct even if workflows are in .github/workflows | ||
| schedule: | ||
| interval: "weekly" | ||
|
|
||
| # Keep Dockerfile base images (in repo root) up to date | ||
| - package-ecosystem: "docker" | ||
| directory: "/" | ||
| schedule: | ||
| interval: "weekly" | ||
| registries: | ||
| - dockerhub | ||
|
|
||
| registries: | ||
| dockerhub: | ||
| type: "docker-registry" | ||
| url: "https://index.docker.io" | ||
| username: "${{secrets.DOCKER_HUB_USERNAME}}" | ||
| password: "${{secrets.DOCKER_HUB_ACCESS_TOKEN}}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| name: Docker Multi-Arch Template | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| image: | ||
| description: "Full image name, e.g. cogstacksystems/cogstack-nifi" | ||
| required: true | ||
| type: string | ||
| context: | ||
| description: "Docker build context path" | ||
| required: true | ||
| type: string | ||
| dockerfile: | ||
| description: "Path to Dockerfile" | ||
| required: true | ||
| type: string | ||
| cache_scope: | ||
| description: "Cache key (unique per image)" | ||
| required: true | ||
| type: string | ||
| build_args: | ||
| description: "Optional Docker build args (newline-separated)" | ||
| required: false | ||
| default: "" | ||
| type: string | ||
|
|
||
| concurrency: | ||
| group: docker-${{ inputs.image }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| # -------- Build per-architecture ---------- | ||
| build: | ||
| runs-on: ${{ matrix.runner }} | ||
| strategy: | ||
| matrix: | ||
| platform: [amd64, arm64] | ||
| include: | ||
| - platform: amd64 | ||
| runner: ubuntu-22.04 | ||
| - platform: arm64 | ||
| runner: ubuntu-22.04-arm | ||
| outputs: | ||
| digest: ${{ steps.push.outputs.digest }} | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v5 | ||
| - uses: docker/setup-qemu-action@v3 | ||
| - uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - if: github.event_name != 'pull_request' | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ secrets.DOCKER_HUB_USERNAME }} | ||
| password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }} | ||
|
|
||
| - name: Docker metadata | ||
| id: meta | ||
| uses: docker/metadata-action@v5 | ||
| with: | ||
| images: ${{ inputs.image }} | ||
| tags: | | ||
| # vX.Y.Z -> X.Y.Z and X.Y (strip leading "v") | ||
| type=semver,pattern={{version}},prefix=v | ||
| type=semver,pattern={{major}}.{{minor}},prefix=v | ||
| # latest on main and on v-tags | ||
| type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') }} | ||
| # branch tag for non-PR branches | ||
| type=ref,event=branch,enable=${{ github.event_name != 'pull_request' }} | ||
| # short sha | ||
| type=sha,format=short | ||
|
|
||
| - name: Build & push (${{ matrix.platform }}) | ||
| id: push | ||
| uses: docker/build-push-action@v6 | ||
| with: | ||
| context: ${{ inputs.context }} | ||
| file: ${{ inputs.dockerfile }} | ||
| platforms: linux/${{ matrix.platform }} | ||
| build-args: ${{ inputs.build_args }} | ||
| tags: | | ||
| ${{ steps.meta.outputs.tags }}-${{ matrix.platform }} | ||
| labels: ${{ steps.meta.outputs.labels }} | ||
| cache-from: type=gha,scope=${{ inputs.cache_scope }}-${{ matrix.platform }} | ||
| cache-to: type=gha,mode=max,scope=${{ inputs.cache_scope }}-${{ matrix.platform }} | ||
| push: ${{ github.event_name != 'pull_request' }} | ||
| provenance: false | ||
|
|
||
| # -------- Merge both architectures ---------- | ||
| manifest: | ||
| runs-on: ubuntu-22.04 | ||
| if: github.event_name != 'pull_request' | ||
| needs: build | ||
|
|
||
| steps: | ||
| - uses: docker/setup-buildx-action@v3 | ||
| - uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ secrets.DOCKER_HUB_USERNAME }} | ||
| password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }} | ||
|
|
||
| - name: Docker metadata (again) | ||
| id: meta | ||
| uses: docker/metadata-action@v5 | ||
| with: | ||
| images: ${{ inputs.image }} | ||
| tags: | | ||
| type=semver,pattern={{version}},prefix=v | ||
| type=semver,pattern={{major}}.{{minor}},prefix=v | ||
| type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') }} | ||
| type=ref,event=branch,enable=${{ github.event_name != 'pull_request' }} | ||
| type=sha,format=short | ||
|
|
||
| - name: Create multi-arch manifest | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| while IFS= read -r ref; do | ||
| [[ -z "$ref" ]] && continue | ||
| img="${ref%%:*}" | ||
| tag="${ref#*:}" | ||
| echo "🌀 Creating manifest for ${img}:${tag}" | ||
| docker buildx imagetools create \ | ||
| --tag "${img}:${tag}" \ | ||
| "${img}:${tag}-amd64" \ | ||
| "${img}:${tag}-arm64" | ||
| done < <(printf "%s" "${{ steps.meta.outputs.tags }}") | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.