Skip to content

#143: Re implement upload submission files #147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
leoraba wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

#143: Re implement upload submission files #147

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
629bb7f
process new records on active submission
leoraba Jun 10, 2025
2d3eeca
process edit submitted records on submission
leoraba Jun 10, 2025
f2ae3ed
fix build and validation issue
leoraba Jun 10, 2025
f7aae2f
unit test parser submission records
leoraba Jun 10, 2025
8241149
clean up code
leoraba Jun 10, 2025
ac1ead7
update submission data
leoraba Jun 10, 2025
6643abd
upload files
leoraba Jun 11, 2025
3c7139c
file utils unit tests
leoraba Jun 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.schema
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ DB_NAME=
DB_PASSWORD=
DB_PORT=
DB_USER=
FILES_LIMIT_SIZE=
ID_CUSTOM_ALPHABET=
ID_CUSTOM_SIZE=
ID_USELOCAL=
Expand Down
35 changes: 18 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,23 +56,24 @@ A model-agnostic, tabular data submission system designed to manage and validate

### Environment Variables

| Name | Description | Default |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------- |
| `ALLOWED_ORIGINS` | Specifies a list of permitted origins for Cross-Origin Resource Sharing (CORS). These origins, separated by commas, are allowed to make requests to the server, ensuring only trusted domains can access resources. (Example: https://www.example.com,https://subdomain.example.com) | |
| `AUDIT_ENABLED` | Ensures that any modifications to the submitted data are logged, providing a way to identify who made changes and when they were made. | true |
| `CORS_ENABLED` | Controls whether the CORS functionality is enabled or disabled. | false |
| `DB_HOST` | Database Hostname | |
| `DB_NAME` | Database Name | |
| `DB_PASSWORD` | Database Password | |
| `DB_PORT` | Database Port | |
| `DB_USER` | Database User | |
| `ID_CUSTOM_ALPHABET` | Custom Alphabet for local ID generation | '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ' |
| `ID_CUSTOM_SIZE` | Custom size of ID for local ID generation | 21 |
| `ID_USELOCAL` | Generate ID locally | true |
| `LECTERN_URL` | Schema Service (Lectern) URL | |
| `LOG_LEVEL` | Log Level | 'info' |
| `PLURALIZE_SCHEMAS_ENABLED` | This feature automatically convert schema names to their plural forms when handling compound documents. Pluralization assumes the words are in English | true |
| `PORT` | Server Port | 3030 |
| Name | Description | Default |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| `ALLOWED_ORIGINS` | Specifies a list of permitted origins for Cross-Origin Resource Sharing (CORS). These origins, separated by commas, are allowed to make requests to the server, ensuring only trusted domains can access resources. (Example: https://www.example.com,https://subdomain.example.com) | |
| `AUDIT_ENABLED` | Ensures that any modifications to the submitted data are logged, providing a way to identify who made changes and when they were made. | true |
| `CORS_ENABLED` | Controls whether the CORS functionality is enabled or disabled. | false |
| `DB_HOST` | Database Hostname | |
| `DB_NAME` | Database Name | |
| `DB_PASSWORD` | Database Password | |
| `DB_PORT` | Database Port | |
| `DB_USER` | Database User | |
| `FILES_LIMIT_SIZE` | Limit upload file size in string or number. <br>Supported units and abbreviations are as follows and are case-insensitive: <br> - b for bytes<br> - kb for kilobytes<br>- mb for megabytes<br>- gb for gigabytes<br>- tb for terabytes<br>- pb for petabytes<br>Any other text is considered as byte | '10mb' |
| `ID_CUSTOM_ALPHABET` | Custom Alphabet for local ID generation | '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ' |
| `ID_CUSTOM_SIZE` | Custom size of ID for local ID generation | 21 |
| `ID_USELOCAL` | Generate ID locally | true |
| `LECTERN_URL` | Schema Service (Lectern) URL | |
| `LOG_LEVEL` | Log Level | 'info' |
| `PLURALIZE_SCHEMAS_ENABLED` | This feature automatically convert schema names to their plural forms when handling compound documents. Pluralization assumes the words are in English | true |
| `PORT` | Server Port | 3030 |

### Script Commands

Expand Down
3 changes: 3 additions & 0 deletions apps/server/src/config/app.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ export const appConfig: AppConfig = {
pluralizeSchemasName: getBoolean(process.env.PLURALIZE_SCHEMAS_ENABLED, true),
},
},
files: {
limitSize: process.env.FILES_LIMIT_SIZE || '10mb',
},
idService: {
useLocal: getBoolean(process.env.ID_USELOCAL, true),
customAlphabet: process.env.ID_CUSTOM_ALPHABET || '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ',
Expand Down
38 changes: 26 additions & 12 deletions apps/server/swagger/submission-api.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,19 +168,26 @@
summary: Add new data to a submission for the specified category. Returns an Active Submission containing the newly created records
tags:
- Submission
consumes:
- multipart/form-data
parameters:
- $ref: '#/components/parameters/path/CategoryId'
- $ref: '#/components/parameters/query/EntityName'
- $ref: '#/components/parameters/query/Organization'
requestBody:
description: The JSON payload containing the data to be added to the submission
description: Files to be submitted. Accepted file extensions are '.tsv' and '.csv'.
required: true
content:
application/json:
multipart/form-data:
schema:
type: array
items:
type: object
type: object
properties:
files:
type: array
items:
type: string
format: binary
required:
- files
responses:
200:
description: Submission accepted
Expand All @@ -200,19 +207,26 @@
summary: Modifies existing data for a submission. Returns an Active Submission containing the records that will be updated
tags:
- Submission
consumes:
- multipart/form-data
parameters:
- $ref: '#/components/parameters/path/CategoryId'
- $ref: '#/components/parameters/query/EntityName'
- $ref: '#/components/parameters/query/Organization'
requestBody:
description: The JSON payload containing the data to be added to the submission
description: Files to be submitted. Accepted file extensions are '.tsv' and '.csv'.
required: true
content:
application/json:
multipart/form-data:
schema:
type: array
items:
type: object
type: object
properties:
files:
type: array
items:
type: string
format: binary
required:
- files
responses:
200:
description: Edit Data request accepted
Expand Down
7 changes: 7 additions & 0 deletions packages/data-provider/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,23 +37,30 @@
"@overture-stack/lectern-client": "2.0.0-beta.4",
"@overture-stack/lyric-data-model": "workspace:^",
"@overture-stack/sqon-builder": "^1.1.0",
"bytes": "^3.1.2",
"csv-parse": "^5.6.0",
"dotenv": "^16.4.5",
"drizzle-orm": "^0.29.5",
"express": "^4.19.2",
"firstline": "^2.0.2",
"lodash-es": "^4.17.21",
"multer": "^2.0.1",
"nanoid": "^5.0.7",
"pg": "^8.12.0",
"plur": "^5.1.0",
"winston": "^3.13.1",
"zod": "^3.23.8"
},
"devDependencies": {
"@types/bytes": "^3.1.5",
"@types/chai-as-promised": "^8.0.1",
"@types/deep-freeze": "^0.1.5",
"@types/express": "^4.17.21",
"@types/express-serve-static-core": "^4.19.5",
"@types/firstline": "^2.0.4",
"@types/lodash": "^4.17.7",
"@types/lodash-es": "^4.17.12",
"@types/multer": "^1.4.13",
"@types/pg": "^8.11.6",
"@types/qs": "^6.9.15",
"chai-as-promised": "^8.0.0",
Expand Down
5 changes: 5 additions & 0 deletions packages/data-provider/src/config/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ export type FeaturesConfig = {
recordHierarchy: RecordHierarchyConfig;
};

export type FilesConfig = {
limitSize: string;
};

export type SchemaServiceConfig = {
url: string;
};
Expand All @@ -43,6 +47,7 @@ export type AppConfig = {
auth: AuthConfig;
db: DbConfig;
features?: FeaturesConfig;
files: FilesConfig;
idService: IdServiceConfig;
logger: LoggerConfig;
onFinishCommit?: (resultOnCommit: ResultOnCommit) => void;
Expand Down
Loading