Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
c49c087
feat: add pgcrypto user table encryption migration
junaidferoz May 18, 2026
7ea39b5
fix(db): COALESCE user view insert booleans and require DB_ENCRYPTION…
junaidferoz May 18, 2026
698a7f4
Merge branch 'dev' into feat-156-pgcrypto_encryption
junaidferoz Jun 1, 2026
1d7c03e
fix: remove old implementation of encryption
karimouf Jun 12, 2026
0228f6c
feat: add encryption functionalities
karimouf Jun 12, 2026
fbad854
feat: add hooks for automatic encryption/decryption handling
karimouf Jun 12, 2026
e62f502
feat: add fields to be encrypted
karimouf Jun 12, 2026
07dc3f2
feat: add db encryption key to the env file
karimouf Jun 12, 2026
7674fa8
feat: add migration to encrypt user data
karimouf Jun 12, 2026
873ef11
Merge branch 'dev' into feat-156-pgcrypto_encryption
karimouf Jun 12, 2026
168a0c9
feat: removing extra role button
karimouf Jun 12, 2026
11f43e5
fix: update salt with encryption key
karimouf Jun 12, 2026
a261817
feat: add before bulk create hook
karimouf Jun 12, 2026
95c46ac
fix: remove wrong naming
karimouf Jun 12, 2026
b119ead
fix: remove wrong naming
karimouf Jun 12, 2026
980387a
Merge branch 'dev' into feat-156-pgcrypto_encryption
karimouf Jun 16, 2026
bc4902d
feat: backup command
karimouf Jun 17, 2026
67d7251
feat: initialize new encryption key each server restart
karimouf Jun 17, 2026
b2d778b
fix: add a down method for the encrypted fields
karimouf Jun 18, 2026
770bc09
fix
karimouf Jun 18, 2026
12d5eb2
fix: remove encrption key from env file
karimouf Jun 18, 2026
8355e11
feat: add enable encryption env variable to enable/disable encryption
karimouf Jun 18, 2026
444aadc
feat: add script to change encryption key
karimouf Jun 22, 2026
8250438
feat: change db encryption state when the .env encryption state is ch…
karimouf Jun 22, 2026
b4e5a7e
feat: generate new key if no new key is provided
karimouf Jun 23, 2026
67d5eaa
Merge branch 'dev' into feat-156-pgcrypto_encryption
bingobongomann Jun 23, 2026
8cf9586
feat: enable decrypting before back up
karimouf Jun 23, 2026
681110b
Merge branch 'feat-156-pgcrypto_encryption' of https://github.com/UKP…
karimouf Jun 23, 2026
f693c3e
refactor: db is cloned and then decrypted
karimouf Jun 28, 2026
0c94ad0
Merge branch 'dev' into feat-156-pgcrypto_encryption
karimouf Jul 1, 2026
1029133
fix: fix function docstring
karimouf Jul 1, 2026
bc1772c
feat: unique fields have an addition column in order to preserve the …
karimouf Jul 1, 2026
ea05cc7
feat: add error message handling to handle unique constraint
karimouf Jul 10, 2026
199ec00
fix: add guard for mapping empty arrays
karimouf Jul 11, 2026
73f5397
fic: add warning when changing encryption env variable
karimouf Jul 11, 2026
afb47ac
Merge branch 'dev' into feat-156-pgcrypto_encryption
karimouf Jul 11, 2026
8c5a123
Merge branch 'dev' into feat-156-pgcrypto_encryption
karimouf Jul 22, 2026
faf1de9
refactor: move encryption in helper folder
karimouf Jul 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .env
Original file line number Diff line number Diff line change
Expand Up @@ -65,5 +65,10 @@ PG_STATS_MIN_AGE_MS=1000
# to parameterize the LIMIT.
PG_STATS_TOP_N=10

# WARNING: Changing this value will trigger a full database migration on the next backend restart.
# true → all encryptedFields across all models will be encrypted in-place.
# false → all encryptedFields will be decrypted back to plaintext in-place.
# Do NOT toggle this while the server is running or with an incomplete backup.
ENCRYPTION_ENABLED=true
# session
SESSION_SECRET=secretString
SESSION_SECRET=secretString
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,16 @@ files/*.zip

# database dumps
db_dumps/
backups/

# logs (root + backend)
logs/
backend/sessions
backend/node_modules
backend/logs
backend/coverage
backend/encryption.key
backend/encryption.state

# ignore build files
dist/
Expand Down
51 changes: 46 additions & 5 deletions Makefile
Comment thread
karimouf marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ help:
@echo "make build Create a dockerized production build including frontend, backend, nlp, services"
@echo "make build-clean Clean the environment of production build"
@echo "make docker Start docker images"
@echo "make backup_db CONTAINER=<name/id> Backup the database in the given container"
@echo "make backup CONTAINER=<name/id> Create full backup (DB dump + .env + encryptionkey + files)"
@echo "make backup_db CONTAINER=<name/id> Backup the database (prompts whether to decrypt before dumping)"
@echo "make recover_db CONTAINER=<name/id> DUMP=<name in db_dumps folder> Recover database into container"
@echo "make anonymize_dump CONTAINER=<name/id> DUMP=<name in db_dumps folder> [SEED=<int>] [NUM=<int>] Create anonymized dump (consent-filtered + pseudonymized)"
@echo "make export_dump_files CONTAINER=<name/id> DUMP=<name in db_dumps folder> Archive document files referenced by an existing anonymized dump"
Expand All @@ -37,6 +38,7 @@ help:
@echo "make kill Kill all node instances (only unix)"
@echo "make modules Install npm packages in all utils/modules subdirectories"
@echo "make audit npm audit for frontend, backend, and utils/modules packages"
@echo "make change_encryption_key NEW_KEY=<64-char hex> Re-encrypt all user fields with a new encryption key"

.PHONY: doc
doc: doc_sphinx
Expand Down Expand Up @@ -137,10 +139,28 @@ build-clean:
@docker network rm ${PROJECT_NAME}_default || echo "IGNORING ERROR"

.PHONY: backup_db
backup_db:
@echo "Backing up database"
mkdir -p db_dumps
@docker exec -t $${CONTAINER} pg_dumpall -c -U postgres > db_dumps/dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql
backup_db: backend/node_modules/.uptodate
ifndef CONTAINER
$(error CONTAINER is not set. Usage: make backup_db CONTAINER=<name/id>)
endif
@echo "Backing up database"; \
mkdir -p db_dumps; \
printf "Decrypt DB before backup? [y/N] "; \
read DECRYPT_ANSWER; \
OUTFILE="db_dumps/dump_$$(date +%d-%m-%Y_%H_%M_%S).sql"; \
if [ "$$DECRYPT_ANSWER" = "y" ] || [ "$$DECRYPT_ANSWER" = "Y" ]; then \
SIDECAR="$(POSTGRES_CAREDB)_backup_$$(date +%s)"; \
echo "[backup] Cloning live DB into $$SIDECAR..."; \
docker exec $(CONTAINER) psql -q -U postgres -c "CREATE DATABASE $$SIDECAR TEMPLATE $(POSTGRES_CAREDB)"; \
echo "[backup] Decrypting clone (live DB stays encrypted)..."; \
(cd backend && POSTGRES_CAREDB=$$SIDECAR npm run --silent decrypt-db); \
echo "[backup] Dumping plaintext clone..."; \
docker exec -t $(CONTAINER) pg_dump -c -C -U postgres $$SIDECAR > $$OUTFILE; \
docker exec $(CONTAINER) psql -q -U postgres -c "DROP DATABASE $$SIDECAR"; \
echo "[backup] Done - plaintext dump: $$OUTFILE"; \
else \
docker exec -t $(CONTAINER) pg_dumpall -c -U postgres > $$OUTFILE; \
fi

.PHONY: recover_db
recover_db:
Expand All @@ -153,6 +173,22 @@ else
@cat "db_dumps/$${DUMP}" | docker exec -i $${CONTAINER} psql -U postgres
endif

.PHONY: backup
backup: backup_db
ifndef CONTAINER
$(error CONTAINER is not set. Usage: make backup CONTAINER=<name/id>)
endif
@echo "Creating full backup archive"
@mkdir -p backups
@LATEST_DUMP=$$(ls -t db_dumps/*.sql | head -n 1); \
TIMESTAMP=$$(date +%d-%m-%Y_%H_%M_%S); \
tar -czf "backups/backup_$$TIMESTAMP.tar.gz" \
"$$LATEST_DUMP" \
.env \
backend/encryption.key \
files; \
echo "Backup created: backups/backup_$$TIMESTAMP.tar.gz"

# Internal target: zip document files from a live DB. Requires DB and FILEZIP to be set.
.PHONY: _export_document_files
_export_document_files:
Expand Down Expand Up @@ -215,6 +251,11 @@ export_dump_files:
$(MAKE) _export_document_files CONTAINER=$${CONTAINER} DB=$$SIDECAR FILEZIP=$$FILEZIP; \
docker exec $${CONTAINER} psql -q -U postgres -c "DROP DATABASE $$SIDECAR" > /dev/null

.PHONY: change_encryption_key
change_encryption_key: backend/node_modules/.uptodate
@echo "Changing encryption key..."
@cd backend && npm run --silent change-encryption-key

.PHONY: admin-password
admin-password: backend/node_modules/.uptodate
cd backend && ADMIN_EMAIL="$(ADMIN_EMAIL)" npm run set-admin-password
Expand Down
5 changes: 5 additions & 0 deletions backend/db/MetaModel.js
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,11 @@ module.exports = class MetaModel extends Model {

} catch (err) {
console.log("DB MetaModel Class " + this.constructor.name + " add error in creation: " + err.message);
if (err.name === 'SequelizeUniqueConstraintError') {
const field = Object.keys(err.fields || {})[0] || '';
const fieldName = field.replace(/Hash$/, '');
throw new Error(`A record with this ${fieldName} already exists.`);
}
throw new Error(err.message);
}
}
Expand Down
167 changes: 167 additions & 0 deletions backend/db/migrations/20260612100001-encrypt-user-fields.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
'use strict';

/**
* Encrypt existing plaintext values in user fields: firstName, lastName, email, initialPassword.
* Also populates emailHash from the newly encrypted email value.
*
* Requires DB_ENCRYPTION_KEY to be set in the environment.
* Skips rows where the field already appears encrypted (safe to re-run).
*/

const { encrypt, getKey, initializeEncryptionKey, decrypt } = require('../../utils/helper/encryption');

module.exports = {
async up(queryInterface) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The up() and down() here have a lot of duplicate code. Can we reuse the code from backend/utils/helper/encryption.js instead? It already has encryptAllModels/decryptAllModels (via _applyToAllModels)

const isEncryptionEnabled = process.env.ENCRYPTION_ENABLED === 'true';
if (!isEncryptionEnabled) {
return;
}
initializeEncryptionKey();
const encryptionKey = getKey();
if (!encryptionKey) {
throw new Error(
'DB_ENCRYPTION_KEY must be set before running the user encryption data migration'
);
}

const transaction = await queryInterface.sequelize.transaction();
try {
const users = await queryInterface.sequelize.query(
`SELECT id, "firstName", "lastName", email, "initialPassword", "twoFactorOtp", "totpSecret", "orcidId", "ldapUsername", "samlNameId", "salt" FROM "user"`,
{ type: queryInterface.sequelize.QueryTypes.SELECT, transaction }
);

for (const user of users) {
const updates = {};

if (user.firstName) {
updates.firstName = encrypt(user.firstName);
}
if (user.lastName) {
updates.lastName = encrypt(user.lastName);
}
if (user.email) {
const encryptedEmail = encrypt(user.email);
updates.email = encryptedEmail;
}
if (user.initialPassword) {
updates.initialPassword = encrypt(user.initialPassword);
}
if (user.twoFactorOtp) {
updates.twoFactorOtp = encrypt(user.twoFactorOtp);
}
if (user.totpSecret) {
updates.totpSecret = encrypt(user.totpSecret);
}
if (user.orcidId) {
updates.orcidId = encrypt(user.orcidId);
}
if (user.ldapUsername) {
updates.ldapUsername = encrypt(user.ldapUsername);
}
if (user.samlNameId) {
updates.samlNameId = encrypt(user.samlNameId);
}
if (user.salt) {
updates.salt = encrypt(user.salt);
}

if (Object.keys(updates).length > 0) {
const setClauses = Object.keys(updates)
.map(col => `"${col}" = :${col}`)
.join(', ');

await queryInterface.sequelize.query(
`UPDATE "user" SET ${setClauses} WHERE id = :id`,
{
replacements: { ...updates, id: user.id },
transaction,
}
);
}
}

await transaction.commit();
} catch (err) {
await transaction.rollback();
throw err;
}
},

async down(queryInterface) {
const isEncryptionEnabled = process.env.ENCRYPTION_ENABLED === 'true';
if (!isEncryptionEnabled) {
return;
}
initializeEncryptionKey();
const encryptionKey = getKey();
if (!encryptionKey) {
throw new Error(
'DB_ENCRYPTION_KEY must be set before running the user encryption data migration'
);
}

const transaction = await queryInterface.sequelize.transaction();
try {
const users = await queryInterface.sequelize.query(
`SELECT id, "firstName", "lastName", email, "initialPassword", "twoFactorOtp", "totpSecret", "orcidId", "ldapUsername", "samlNameId", "salt" FROM "user"`,
{ type: queryInterface.sequelize.QueryTypes.SELECT, transaction }
);

for (const user of users) {
const updates = {};

if (user.firstName) {
updates.firstName = decrypt(user.firstName);
}
if (user.lastName) {
updates.lastName = decrypt(user.lastName);
}
if (user.email) {
const decryptedEmail = decrypt(user.email);
updates.email = decryptedEmail;
}
if (user.initialPassword) {
updates.initialPassword = decrypt(user.initialPassword);
}
if (user.twoFactorOtp) {
updates.twoFactorOtp = decrypt(user.twoFactorOtp);
}
if (user.totpSecret) {
updates.totpSecret = decrypt(user.totpSecret);
}
if (user.orcidId) {
updates.orcidId = decrypt(user.orcidId);
}
if (user.ldapUsername) {
updates.ldapUsername = decrypt(user.ldapUsername);
}
if (user.samlNameId) {
updates.samlNameId = decrypt(user.samlNameId);
}
if (user.salt) {
updates.salt = decrypt(user.salt);
}

if (Object.keys(updates).length > 0) {
const setClauses = Object.keys(updates)
.map(col => `"${col}" = :${col}`)
.join(', ');

await queryInterface.sequelize.query(
`UPDATE "user" SET ${setClauses} WHERE id = :id`,
{
replacements: { ...updates, id: user.id },
transaction,
}
);
}
}

await transaction.commit();
} catch (err) {
await transaction.rollback();
throw err;
}
},
};
2 changes: 2 additions & 0 deletions backend/db/models/user.js
Original file line number Diff line number Diff line change
Expand Up @@ -602,6 +602,8 @@ module.exports = (sequelize, DataTypes) => {
sequelize,
modelName: "user",
tableName: "user",
//Keys that require encryption unique set to false by default
encryptedFields: ['firstName', 'lastName', { name: 'email', unique: true }, 'salt', 'initialPassword', 'twoFactorOtp', 'totpSecret', 'orcidId', 'ldapUsername', 'samlNameId'],

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we also need to encrypt extId, but it is currently an Integer. Maybe we should change it to String, but we need to make sure that it will still work everywhere, maybe need a parse to int when requesting Moodle (needs testing).

hooks: {
afterCreate: async (user, options) => {
const {context, transaction} = options;
Expand Down
Loading
Loading