-
Notifications
You must be signed in to change notification settings - Fork 28
Migration copy2 #1563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
borkarsaish65
wants to merge
86
commits into
ELEVATE-Project:develop
Choose a base branch
from
borkarsaish65:migration_copy2
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Migration copy2 #1563
Changes from all commits
Commits
Show all changes
86 commits
Select commit
Hold shift + click to select a range
0625ef1
feat:new files
borkarsaish65 5344ae3
updated devops files
borkarsaish65 bbd8fc1
jest modification
borkarsaish65 d45e806
package json modification
borkarsaish65 0466e74
savepoint
borkarsaish65 acfb78b
Merge branch 'mentorReadOptimisation' of https://github.com/ELEVATE-P…
borkarsaish65 d1be292
pushing test sequencer
borkarsaish65 aac18ae
updates after addressing pr comments
borkarsaish65 b92293c
kafka file update
borkarsaish65 6998e8e
kafka change
borkarsaish65 821d638
feat:updated docker compose with kafka circleci file
borkarsaish65 aa0e9b3
feat:updated pakcage json and docker compose file
borkarsaish65 4ccdbeb
script for migration script
borkarsaish65 7b61505
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 a78fc1c
migration script fix
borkarsaish65 c9da6d6
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 209c306
restored prev migrations and seeders
borkarsaish65 6a12e90
migrations updated
borkarsaish65 c6e8050
migrations fix check
borkarsaish65 fa6e05d
restored files
borkarsaish65 91591f9
feat:samigrations updates
borkarsaish65 3b6f1a9
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 0914311
savepoint-post-pr-review-1
borkarsaish65 0a6e228
addressed PR comments
borkarsaish65 db6f603
new fix
borkarsaish65 c2d0444
new fix
borkarsaish65 cbaf658
new fix-1
borkarsaish65 9d0b5ff
fix-2
borkarsaish65 82f2a1b
fix-3
borkarsaish65 df2bcd6
fix-3.1
borkarsaish65 a364f77
fix-4
borkarsaish65 f94947d
fix-5
borkarsaish65 e9a1558
fix-6
borkarsaish65 aea283d
fix-7
borkarsaish65 d16933b
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 d8602a2
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 dd992f3
updated migration scripts and seeders
borkarsaish65 e2e3026
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 25544a0
migrations+test integration
borkarsaish65 9834782
savepoint-1
borkarsaish65 7b30729
done the changes
borkarsaish65 4456acd
circleci file updated
borkarsaish65 db052d2
circleci update with correct npm commands
borkarsaish65 6f5178c
copying kafka file is fixed
borkarsaish65 b646efa
copying kafka file is fixed-1
borkarsaish65 74a1ddd
copying kafka file is fixed-2
borkarsaish65 764e094
copying kafka file is fixed-3
borkarsaish65 3b9947f
copying kafka file is fixed-4
borkarsaish65 baf74f4
updated circleci file
borkarsaish65 02964dc
kafka file updated.
borkarsaish65 9db429f
addressed pr comments
borkarsaish65 42b28e6
removed unused variables
borkarsaish65 852b701
enabling citus
borkarsaish65 189db62
correcting the citus code
borkarsaish65 802744f
enabling citus - 2
borkarsaish65 1f53832
enabling citus - 3
borkarsaish65 200164e
enabling citus - 4
borkarsaish65 28b67e6
enabling citus - 5
borkarsaish65 8ccbf94
enabling citus - 6
borkarsaish65 e933acf
citus and distribution placement fixed in circleci
borkarsaish65 c86582c
updated citus config
borkarsaish65 b4a1fa1
updated citus config
borkarsaish65 ecb4ecb
circleci and docker-compose file updated
borkarsaish65 5afd288
test sequencer fix
borkarsaish65 e246735
test sequencer fix -1
borkarsaish65 1ac061d
test sequencer fix -2
borkarsaish65 32e1727
starting user service via circleci
borkarsaish65 d5eee49
savepoint-1 refresh view build
borkarsaish65 b8ffdbb
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 07f8199
savepoint-2 minor fixes
borkarsaish65 3f171a3
savepoint-3 minor fixes
borkarsaish65 f0de7c9
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 30a9384
Merge branch 'feature/int_test12' of https://github.com/borkarsaish65…
borkarsaish65 41ca559
deleted config.json
borkarsaish65 ce89008
undoed migration changes from this PR
borkarsaish65 79e17ae
updated package json file
borkarsaish65 3245c85
Merge branch 'feature/int_test12' of https://github.com/borkarsaish65…
borkarsaish65 b880388
package json updated
borkarsaish65 02c0ce5
Merge branch 'feature/int_test12' of https://github.com/borkarsaish65…
borkarsaish65 6496a73
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 7e0e755
Merge branch 'develop' of https://github.com/ELEVATE-Project/mentorin…
borkarsaish65 a504f0a
savepoint-1
borkarsaish65 2742bdc
savepoint-2
borkarsaish65 1716b3f
savepoint-3
borkarsaish65 586160d
savepoint-2
borkarsaish65 9d2d573
savepoint-3
borkarsaish65 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,64 +1,231 @@ | ||
| version: 2.1 # CircleCI version | ||
| orbs: | ||
| sonarcloud: sonarsource/sonarcloud@1.1.1 | ||
| version: 2.1 | ||
|
|
||
| jobs: | ||
| build: | ||
| machine: #Linux machine instead of docker environment | ||
| machine: | ||
| image: ubuntu-2004:2024.05.1 | ||
| docker_layer_caching: true | ||
| working_directory: ~/mentoring # Default working directory | ||
|
|
||
| working_directory: ~/mentoring | ||
|
|
||
| steps: | ||
| # ------------------------------- | ||
| # Git setup | ||
| # ------------------------------- | ||
| - run: | ||
| name: Configure git to use HTTPS | ||
| command: git config --global url."https://github.com/".insteadOf "git@github.com:" | ||
|
|
||
| - checkout: | ||
| path: ~/mentoring/ | ||
|
|
||
| # ------------------------------- | ||
| # Restore mentoring dependencies | ||
| # ------------------------------- | ||
| - restore_cache: | ||
| key: mentoring-dependency-cache-{{ checksum "src/package.json" }} | ||
|
|
||
| - run: | ||
| name: Install dependencies | ||
| command: cd src/ && npm install #cd ../src/ && pwd && | ||
| name: Install mentoring dependencies | ||
| command: | | ||
| cd src | ||
| npm install | ||
|
|
||
| - save_cache: | ||
| key: mentoring-dependency-cache-{{checksum "src/package.json"}} | ||
| key: mentoring-dependency-cache-{{ checksum "src/package.json" }} | ||
| paths: | ||
| - ./src/node_modules | ||
|
|
||
| # ------------------------------- | ||
| # Disable SSH rewrite | ||
| # ------------------------------- | ||
| - run: | ||
| name: Executing unit test cases | ||
| command: cd src/ && npm run test -- --coverage --collectCoverageFrom="./services/**" | ||
| - store_artifacts: | ||
| path: src/coverage/ | ||
| destination: /coverage/ | ||
| - sonarcloud/scan | ||
| name: Disable SSH rewrite | ||
| command: git config --global --unset url."ssh://git@github.com".insteadOf || true | ||
|
|
||
| # ------------------------------- | ||
| # Clone USER service | ||
| # ------------------------------- | ||
| - run: | ||
| name: Checking prerequisites | ||
| command: |- | ||
| docker-compose --version | ||
| - run: | ||
| name: Cloning user service | ||
| command: cd ../ && git clone https://github.com/ELEVATE-Project/user.git --branch dev --single-branch | ||
| name: Clone User service | ||
| command: | | ||
| cd .. | ||
| git clone https://github.com/ELEVATE-Project/user.git \ | ||
| --branch develop \ | ||
| --single-branch | ||
|
|
||
| - restore_cache: | ||
| key: user-dependency-cache-{{ checksum "../user/src/package.json" }} | ||
|
|
||
| - run: | ||
| name: Install User service dependencies | ||
| command: cd ../user/src/ && npm install #cd ../src/ && pwd && | ||
| command: | | ||
| cd ../user/src | ||
| npm install | ||
|
|
||
| - save_cache: | ||
| key: user-dependency-cache-{{checksum "../user/src/package.json"}} | ||
| key: user-dependency-cache-{{ checksum "../user/src/package.json" }} | ||
| paths: | ||
| - ../user/src/node_modules | ||
|
|
||
| # ------------------------------- | ||
| # Clone optional services | ||
| # ------------------------------- | ||
| - run: | ||
| name: Clone Scheduler service | ||
| command: | | ||
| cd .. | ||
| git clone https://github.com/ELEVATE-Project/scheduler.git \ | ||
| --branch dev \ | ||
| --single-branch || true | ||
|
|
||
| - run: | ||
| name: Clone Communications service | ||
| command: | | ||
| cd .. | ||
| git clone https://github.com/ELEVATE-Project/chat-communications.git \ | ||
| --branch dev \ | ||
| --single-branch || true | ||
|
|
||
| # ------------------------------- | ||
| # Config | ||
| # ------------------------------- | ||
| - run: | ||
| name: Copy config.json | ||
| command: cp dev-ops/integration_test.config.json src/config.json | ||
|
|
||
| # ------------------------------- | ||
| # Start docker-compose | ||
| # ------------------------------- | ||
| - run: | ||
| name: Start docker containers | ||
| command: | | ||
| cd dev-ops | ||
| docker-compose pull || true | ||
| docker-compose build | ||
| docker-compose up -d | ||
| sleep 20 | ||
|
|
||
| # ------------------------------- | ||
| # Wait for healthchecks | ||
| # ------------------------------- | ||
| - run: | ||
| name: Wait for services to be healthy | ||
| command: | | ||
| cd dev-ops | ||
| echo "Waiting for containers to be healthy..." | ||
|
|
||
| for i in {1..30}; do | ||
| unhealthy=$(docker ps --filter "health=unhealthy" --format "{{.Names}}") | ||
| starting=$(docker ps --filter "health=starting" --format "{{.Names}}") | ||
|
|
||
| if [ -z "$unhealthy" ] && [ -z "$starting" ]; then | ||
| echo "All containers are healthy" | ||
| break | ||
| fi | ||
|
|
||
| echo "Waiting... attempt $i" | ||
| docker ps --format "table {{.Names}}\t{{.Status}}" | ||
| sleep 5 | ||
| done | ||
| # ------------------------------- | ||
| # Run migrations (EXPLICIT) | ||
| # ------------------------------- | ||
| - run: | ||
| name: Run mentoring migrations | ||
| command: | | ||
| export MENTORING=$(docker ps --format "{{.Names}}" | grep mentoring) | ||
| echo "Running migrations in $MENTORING" | ||
| docker exec "$MENTORING" npm run db:init | ||
|
|
||
| - run: | ||
| name: Enable Citus and run distributionColumns.psql | ||
| command: | | ||
| cd dev-ops | ||
| set -e | ||
|
|
||
| echo "Enabling Citus extension..." | ||
| docker-compose exec -T citus psql \ | ||
| -U postgres \ | ||
| -d elevate-mentoring \ | ||
| -c "CREATE EXTENSION IF NOT EXISTS citus;" | ||
|
|
||
| echo "Running distributionColumns.psql..." | ||
| docker-compose exec -T citus psql \ | ||
| -U postgres \ | ||
| -d elevate-mentoring \ | ||
| -f /var/src/distributionColumns.psql | ||
|
|
||
| echo "Citus setup completed." | ||
|
|
||
| # ------------------------------- | ||
| # Run seeders (EXPLICIT) | ||
| # ------------------------------- | ||
| - run: | ||
| name: Run mentoring seeders | ||
| command: | | ||
| export MENTORING=$(docker ps --format "{{.Names}}" | grep mentoring) | ||
| echo "Running seeders in $MENTORING" | ||
| docker exec "$MENTORING" npm run db:seed:all | ||
|
|
||
| - run: | ||
| name: Start mentoring app | ||
| command: | | ||
| export MENTORING=$(docker ps --format "{{.Names}}" | grep mentoring) | ||
| echo "Starting mentoring app..." | ||
| docker exec -d "$MENTORING" npm run dev | ||
|
|
||
| - run: | ||
| name: Run user migrations | ||
| command: | | ||
| export USER=$(docker ps --format "{{.Names}}" | grep user) | ||
| echo "Running user migrations in $USER" | ||
| docker exec "$USER" npm run db:init | ||
|
|
||
| - run: | ||
| name: Run user seeders | ||
| command: | | ||
| export USER=$(docker ps --format "{{.Names}}" | grep user) | ||
| docker exec "$USER" npm run db:seed:all | ||
|
|
||
|
|
||
| - run: | ||
| name: Start user service | ||
| command: | | ||
| export USER=$(docker ps --format "{{.Names}}" | grep user) | ||
| echo "Starting user service..." | ||
| docker exec -d "$USER" npm run dev | ||
|
|
||
|
|
||
| # ------------------------------- | ||
| # Run integration tests | ||
| # ------------------------------- | ||
| - run: | ||
| name: Starting the docker containers | ||
| command: |- | ||
| cd dev-ops/ && docker-compose up -d | ||
| name: Run integration tests | ||
| command: | | ||
| cd src | ||
| npm run test:integration | ||
|
|
||
| # ------------------------------- | ||
| # Diagnostics | ||
| # ------------------------------- | ||
| - run: | ||
| name: Running test cases | ||
| command: |- | ||
| cd src/ && npm run test:integration | ||
| name: Print logs on failure | ||
| when: always | ||
| command: | | ||
| cd dev-ops | ||
| docker-compose logs --tail=200 kafka || true | ||
| docker-compose logs --tail=200 citus || true | ||
| docker-compose logs --tail=200 mentoring || true | ||
| docker-compose logs --tail=200 user || true | ||
|
|
||
| - store_test_results: | ||
| path: ./dev-ops/report | ||
|
|
||
| workflows: | ||
| build-and-test: # This is the name of the workflow, | ||
| # Inside the workflow, you define the jobs you want to run. | ||
| build-and-test: | ||
| jobs: | ||
| - build: | ||
| context: | ||
| - SonarCloud | ||
| filters: | ||
| tags: | ||
| only: \b(dev|develop|main)\b | ||
| only: \b(dev|develop|main)\b | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Health check loop silently continues on failure — pipeline will proceed with unhealthy services.
If containers never become healthy within 30 attempts, the
forloop completes without hittingbreak, and execution silently proceeds to the migration step. This will cause confusing downstream failures. Add an explicit failure check after the loop.🐛 Proposed fix: fail the step if containers aren't healthy
echo "Waiting... attempt $i" docker ps --format "table {{.Names}}\t{{.Status}}" sleep 5 done + + # Fail if any container is still not healthy + unhealthy=$(docker ps --filter "health=unhealthy" --format "{{.Names}}") + starting=$(docker ps --filter "health=starting" --format "{{.Names}}") + if [ -n "$unhealthy" ] || [ -n "$starting" ]; then + echo "ERROR: Containers not healthy after 30 attempts: $unhealthy $starting" + docker ps --format "table {{.Names}}\t{{.Status}}" + exit 1 + fi🤖 Prompt for AI Agents