Typescript module server by ciaranschutte · Pull Request #936 · overture-stack/arranger

ciaranschutte · 2025-04-12T13:01:01Z

converts all files to TS
ensures type is correctly in imports
uses underscore to denote unused param, supported in TS as unused param eg. (_req, res)...
add getCaughtErrorMessage
- type guards for caught thrown errors
moves ES client to module and define consistent type from "new api" folder (this type returns typings from api calls)
- which also means ES functions require more strict typing overall eg. providing full set of object properties
ConfigObject is no longer a partial type
config becomes an extended object which complicates types, leaving most of this as any for now
config "prop drilling" has a lot of duplicated typing for now, refactoring config has been discussed

Further improvements:

location of types. Decide if they are colocated or a distinct "types" folder

* initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* add agg mode env config * env config to strip out section of gql schema, typedef + resolver * prettier * prettier * Update modules/server/.env.schema Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * Update modules/server/src/mapping/createConnectionResolvers.ts Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * alphabetize * move up import * change AGG_ONLY_MODE to enabled DOCUMENT_HITS * add empty line at EOF * add comment * fix typo * fix inverted logic --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* add conditional prop to schema * conditional schema field * conditional add dataMasking schema fields * add data masking logic * add agg only conditional to resolvers * clarify todo * get optional aggregations field and values for hits resolution * add mask func stub * fix missing prop from spread * masking logic * conditional resolve hits * move thresholding higher on resolver chain * add falsey belowThreshold value instead of null * data mask threshold env var * update wrong process.env path * seperate resolver creation from routing * check for undefiend value in lookup * clarify comment * add threshold value explanation * move types to common file * typing --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* fix build error * single endpoint * fix GQL type name collisions * move data mask threshold env. ensure data masking applies even if hits.total is not queried * change belowThreshold to relation object * surface api errors to axios responses * add relation logic and agg merging for network search * read data mask threshold min from config * check for valid env val * remove unused func * fix TS build errors * fix schema merge error --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

* cleanup utils * reorg types, decompose file, improve comments * clean up types * cleanup types, redundant code and fix a couple of bad merge files * fix enum * cleanup test types * type fix * explicit type import, use path alias --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

ciaranschutte · 2025-04-12T13:21:31Z

modules/server/src/esClient/index.ts

+import { Client, type ClientOptions } from '@elastic/elasticsearch';
+
+export const buildEsClient = (esHost = '', esUser = '', esPass = '') => {
+	if (!esHost) {
+		console.error('no elasticsearch host was provided');
+	}
+
+	const esConfig: ClientOptions = {
+		node: esHost,
+	};
+
+	if (esUser) {
+		if (!esPass) {
+			console.error('ES user was defined, but password was not');
+		}
+		esConfig['auth'] = {
+			username: esUser,
+			password: esPass,
+		};
+	}
+
+	return new Client(esConfig);
+};
+
+export const buildEsClientViaEnv = (ES_HOST: string, ES_USER: string, ES_PASS: string) => {
+	return buildEsClient(ES_HOST, ES_USER, ES_PASS);
+};


move ES client to module

README.md

justincorrigible · 2025-04-14T12:57:23Z

modules/server/src/esClient/index.ts

+import { Client, type ClientOptions } from '@elastic/elasticsearch';
+
+export const buildEsClient = (esHost = '', esUser = '', esPass = '') => {
+	if (!esHost) {


wouldn't it be fair to provide a default value for this?
ES has a common 9200 port, and localhost is a usual

justincorrigible · 2025-04-14T13:02:43Z

modules/server/src/graphqlRoutes.ts

+import type { Client } from '@elastic/elasticsearch';
+import type { SQONFilter } from '#sqon/index.js';
+import { getCaughtErrorMessage } from '#utils/caughtError.js';
+import type { GraphQLSchema } from 'graphql';


please, would you add these in the order/position that keeps consistency with code already present?
I've found that using this Microsoft extension for ESLint helps a lot keeping track of this and other "consistency" items https://marketplace.visualstudio.com/items/?itemName=dbaeumer.vscode-eslint

handy extension thank you 👍 they become "out of sync" with VSCode "import this file" which we now have after ts-ify 👍

justincorrigible · 2025-04-14T13:07:00Z

modules/server/src/graphqlRoutes.ts

 		};
 	} catch (error) {
-		const message = error?.message || error;
+		const message = getCaughtErrorMessage(error);


👍 one more from my ToDo:s, thanks for handling it!

Ciarán Schütte and others added 30 commits March 23, 2025 22:56

Pr feedback - doc updates (#879)

687a5ff

* update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

Feat/resolve stats (#883)

5ecee76

* add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

use Math methods (#900)

c111736

Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

Minor comment updates (#905)

c0cf685

* improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca>

Update modules/server/src/app.js

959d8b1

Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

make horrible ajax.ts console logging somewhat useful

2949a67

justincorrigible and others added 22 commits March 25, 2025 18:06

improve release process

49c5096

fix paths and imports for new build config

bcbf899

update tests to todos

fd53c02

fix some imports

ddd36e3

fix build issues with lodash and missing config flag

7417058

Merge branch 'main' into feature/federated

98aa919

fix paths

ed5f625

comment out test files, broken with tsx vs jest

5a5275c

fix path

122df8f

fix path

874a6d6

improve release process

22f45ed

Merge branch 'main' into feature/federated

1c8829d

allow toolbar options customization

eaf5813

fix setsIndex conflict resolutioon misses

83065d2

default for ENABLE_DOCUMENT_HITS env flag is true

9bf0154

Merge branch 'main' into feature/federated

5dac0e2

default enable document hits to true

1fabba7

add ts watch command

a2233eb

Add Express types to server and type dependency

669e3e7

remove unused es dir, create new esClient module

5754252

type fixes

f1427a4

Merge remote-tracking branch 'origin/main' into typescript_module_server

193ebb7

ciaranschutte commented Apr 12, 2025

View reviewed changes

justincorrigible reviewed Apr 14, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

justincorrigible reviewed Apr 14, 2025

View reviewed changes

Base automatically changed from feature/federated to main April 22, 2025 13:04

justincorrigible force-pushed the main branch from 85433e2 to 4ae6089 Compare May 5, 2025 12:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Typescript module server#936

Typescript module server#936
ciaranschutte wants to merge 52 commits intomainfrom
typescript_module_server

ciaranschutte commented Apr 12, 2025 •

edited

Loading

Uh oh!

ciaranschutte Apr 12, 2025

Uh oh!

Uh oh!

justincorrigible Apr 14, 2025

Uh oh!

justincorrigible Apr 14, 2025

Uh oh!

ciaranschutte Apr 14, 2025

Uh oh!

justincorrigible Apr 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ciaranschutte commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ciaranschutte Apr 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

justincorrigible Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

justincorrigible Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

ciaranschutte Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

justincorrigible Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ciaranschutte commented Apr 12, 2025 •

edited

Loading