Release 3.0.0 by vivianzheng404 · Pull Request #32 · jas34/scheduledwf

vivianzheng404 · 2023-06-23T14:53:14Z

This version is compatible to conductor-boot v3.13.5
Besides mySQL, additionally supports PostgreSQL for persistent storage

jas34

Apart from inline comments, the following are the general comments:

Add author in new files as well as add since={version}.
Update the author with your name in all of the touched files.
We need to align of code style/format. If needed I can provide one which has been originally used in the older releases.
Update README.md for the addition of Postgres persistence layer. Currently, it mentions only MySQL implementation.

jas34 · 2023-07-03T11:15:21Z

-			<groupId>junit</groupId>
-			<artifactId>junit</artifactId>
-			<version>4.13.1</version>
+			<groupId>org.junit.jupiter</groupId>


Conductor test cases have been built using junit. Any reason for upgrading from junit to junit-jupiter?
If not then I would recommend to stick to the dependency used by conductor.

Run mvn --settings settings.xml -P bintray clean install

junit returns error

[ERROR] TestEngine with ID 'junit-vintage' failed to discover tests
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
[ERROR] TestEngine with ID 'junit-vintage' failed to discover tests
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:631)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
[ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)

While org.junit.jupiter, build is successful.

See the post here https://stackoverflow.com/questions/59900637/error-testengine-with-id-junit-vintage-failed-to-discover-tests-with-spring

I have tried to run the mvn --settings settings.xml -P bintray clean install with dependency junit instead of org.junit.jupiter and the build was successful. I am attaching build logs also. Can you recheck some system-level or application-level settings?
build_sucess_logs_with_junit.txt

junit works when I add dependency junit-vintage-engine version 5.7.0+. Version 5.6.2 as your build logs shows, does not work. Refer to https://stackoverflow.com/questions/70452633/org-junit-platform-commons-junitexception-testengine-with-id-junit-jupiter-fa/74501390#74501390

The newer version of junit-vintage-engine is 5.10.0 https://repo1.maven.org/maven2/org/junit/vintage/junit-vintage-engine/ Besides junit, I am going to add dependency junit-vintage-engine version 5.10.0.

jas34 · 2023-07-03T11:49:40Z

+		</dependency>
+
+
+		<dependency>


This dependency should be only needed in the module scheduled-postgres-persistence just like conductor-mysql-persistence dependency inside scheduled-mysql-persistence

Without conductor-postgres-persistence dependency here, I got error

java.lang.NoClassDefFoundError: com/netflix/conductor/common/run/WorkflowTestRequest
at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?]
at java.lang.Class.privateGetDeclaredConstructors(Class.java:3137) ~[?:?]
at java.lang.Class.getDeclaredConstructors(Class.java:2357) ~[?:?]
at org.springframework.boot.context.properties.ConfigurationPropertiesBindConstructorProvider.findConstructorBindingAnnotatedConstructor(ConfigurationPropertiesBindConstructorProvider.java:62) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.context.properties.ConfigurationPropertiesBindConstructorProvider.getBindConstructor(ConfigurationPropertiesBindConstructorProvider.java:48) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.context.properties.ConfigurationPropertiesBean$BindMethod.forType(ConfigurationPropertiesBean.java:311) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.context.properties.ConfigurationPropertiesBeanDefinitionValidator.validate(ConfigurationPropertiesBeanDefinitionValidator.java:63) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.context.properties.ConfigurationPropertiesBeanDefinitionValidator.postProcessBeanFactory(ConfigurationPropertiesBeanDefinitionValidator.java:45) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:291) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:175) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
at io.github.jas34.scheduledwf.ScheduledWorkflowConductor.main(ScheduledWorkflowConductor.java:34) [classes/:?]
Caused by: java.lang.ClassNotFoundException: com.netflix.conductor.common.run.WorkflowTestRequest
at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]

Can you make it work without adding conductor-postgres-persistence here?

Thanks for the information. I will work for sure work on this item.

After removing this dependence from parent pom:

I am able to build.

I am able to run the application.

Can you elaborate you are getting this error on the application start or while running testes?

After removing conductor-postgres-persistence from parent pom, the application is able to start, but it cannot function as expected.

Is the application started?
Yes. IntelliJ UI shows the app is running.

The console shows this error message in the beginning:

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'workflowSweeper' defined in URL [jar:file:/Users/vivianzheng/.m2/repository/com/netflix/conductor/conductor-core/3.13.5/conductor-core-3.13.5.jar!/com/netflix/conductor/core/reconciliation/WorkflowSweeper.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'workflowExecutor' defined in URL [jar:file:/Users/vivianzheng/.m2/repository/com/netflix/conductor/conductor-core/3.13.5/conductor-core-3.13.5.jar!/com/netflix/conductor/core/execution/WorkflowExecutor.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'deciderService' defined in URL [jar:file:/Users/vivianzheng/.m2/repository/com/netflix/conductor/conductor-core/3.13.5/conductor-core-3.13.5.jar!/com/netflix/conductor/core/execution/DeciderService.class]: Unsatisfied dependency expressed through constructor parameter 4; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'systemTaskRegistry' defined in URL [jar:file:/Users/vivianzheng/.m2/repository/com/netflix/conductor/conductor-core/3.13.5/conductor-core-3.13.5.jar!/com/netflix/conductor/core/execution/tasks/SystemTaskRegistry.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'START_WORKFLOW' defined in URL [jar:file:/Users/vivianzheng/.m2/repository/com/netflix/conductor/conductor-core/3.13.5/conductor-core-3.13.5.jar!/com/netflix/conductor/core/execution/tasks/StartWorkflow.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultValidator' defined in class path resource [org/springframework/boot/autoconfigure/validation/ValidationAutoConfiguration.class]: Invocation of init method failed; nested exception is javax.validation.ValidationException: Unable to parse null at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE] at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]

then console keeps generating the same response:

Caused by: java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed. at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:96) ~[HikariCP-3.4.5.jar:?] at com.netflix.conductor.postgres.dao.PostgresBaseDAO.getWithTransaction(PostgresBaseDAO.java:117) ~[conductor-postgres-persistence-3.13.5.jar:3.13.5] at com.netflix.conductor.postgres.dao.PostgresBaseDAO.lambda$getWithRetriedTransactions$3(PostgresBaseDAO.java:145) ~[conductor-postgres-persistence-3.13.5.jar:3.13.5] at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329) ~[spring-retry-1.3.0.jar:?] at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:209) ~[spring-retry-1.3.0.jar:?] at com.netflix.conductor.postgres.dao.PostgresBaseDAO.getWithRetriedTransactions(PostgresBaseDAO.java:145) ~[conductor-postgres-persistence-3.13.5.jar:3.13.5] ... 9 more

Does the app function as expected?
No. None of the endpoints works. Error: connect ECONNREFUSED ....:8080.

Also, the swagger UI refused to connect. localhost:8080/swagger-ui.html

Tests are successful. mvn --settings settings.xml -P bintray clean install This command runs fine.

To reproduce this, try schedule some workflows, then shut down some or all workflows. From the repeating error message java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed and postgres.dao.PostgresBaseDAO.getWithTransaction, I think you may need to have some data/scheduled workflows in Postgres to reproduce.

In my case, I have 3 shutdown scheduled workflow, 1 running workflow.

No conductor-postgres-persistence from parent pom
Behavior described as above.

With conductor-postgres-persistence from parent pom
The console keeps generating the same log

86405 [pool-8-thread-1] DEBUG io.github.jas34.scheduledwf.execution.DefaultSchedulerManager [] - No workflow left to be scheduled. 86406 [pool-8-thread-1] DEBUG io.github.jas34.scheduledwf.execution.DefaultSchedulerManager [] - No running process found for shutdown with managerRef=fc656dcd-ed26-468e-b741-4b9da5b03910, with names=[check-conductor-health-workflow2, check-conductor-health-workflow, check-conductor-health-workflow3]

The console is flooded with repeating message, if you feel like this needs future improvement, can you create a work item?

jas34 · 2023-07-03T12:05:07Z

+
+		<dependency>
+			<groupId>org.springframework.retry</groupId>
+			<artifactId>spring-retry</artifactId>


This dependency should not be declared explicitly.

This dependency is needed by modules scheduled-mysql-persistence and scheduled-postgres-persistence.

We still do not need to declare this in these modules because they will be available transitively from conductor-mysql-persistence and conductor-postgres-persistence respectively.

If you remove the spring-retry block here, you will see error "java: package org.springframework.retry.support does not exist". Code won't run.

Without the dependency here, the
import org.springframework.retry.support.RetryTemplate; will not work.

If you use IntelliJ IDE, you can try to comment out spring-retry dependency, click, File -> Invalidate Caches -> Select all boxes, after the IDE reopens, and see if the code runs. I see error "java: package org.springframework.retry.support does not exist"

Can you make it work without adding spring-retry here?

Thanks for the information. I will work for sure work on this item.

Your findings are correct. This dependency is not available from conductor modules because everywhere it has been defined with scope compileOnly. Check this line

We should do the following:

This dependency is mostly relevant for persistence modules. We shall not declare these in parent pom.

Let's add this dependency in each module - scheduled-mysql-persistence and scheduled-postgres-persistence.

Conductor is using version 1.3.3, we shall also opt for the same.

Removing spring-retry from parent pom and adding to scheduled-mysql-persistence and scheduled-postgres-persistence do not work.

The application failed to start, error message:

com.netflix.conductor.core.config.ConductorCoreConfiguration.onTransientErrorRetryTemplate(ConductorCoreConfiguration.java:118)

The following method did not exist:

org.springframework.retry.support.RetryTemplateBuilder org.springframework.retry.support.RetryTemplate.builder()

From the error message, ConductorCoreConfiguration class from package com.netflix.conductor.core.config uses RetryTemplate.

spring-retry should not be removed from parent pom. I updated the version to be 1.3.3.

jas34 · 2023-07-04T08:21:47Z

+                            .addParameter(def.getStatus() != null ? def.getStatus().name() : null)
+                            .addJsonParameter(def).addParameter(def.getCronExpression()).executeUpdate());
+        } else {
+            // @formatter:off


Remove this comment from here as well as MySQLScheduledWfMetaDataDao.java#L110

vivianzheng404 · 2023-07-12T19:55:02Z

Apart from inline comments, the following are the general comments:

Add author in new files as well as add @SInCE.

Update the author with your name in all of the touched files.

We need to align of code style/format. If needed I can provide one which has been originally used in the older releases.

Update README.md for the addition of Postgres persistence layer. Currently, it mentions only MySQL implementation.

Reply:

I am curious about the reason for adding @SInCE
Done
Is the style/format referring to the formatter in this comment ? https://github.com/jas34/scheduledwf/pull/32/files#r1250801445 I am not aware, please elaborate.
Done

This reverts commit 1ac05c0.

This reverts commit 57ed3c9.

jas34 · 2023-07-15T06:31:09Z

-			<groupId>junit</groupId>
-			<artifactId>junit</artifactId>
-			<version>4.13.1</version>
+			<groupId>org.junit.jupiter</groupId>


I have tried to run the mvn --settings settings.xml -P bintray clean install with dependency junit instead of org.junit.jupiter and the build was successful. I am attaching build logs also. Can you recheck some system-level or application-level settings?
build_sucess_logs_with_junit.txt

jas34 · 2023-07-15T07:10:20Z

+
+		<dependency>
+			<groupId>org.springframework.retry</groupId>
+			<artifactId>spring-retry</artifactId>


Your findings are correct. This dependency is not available from conductor modules because everywhere it has been defined with scope compileOnly. Check this line

We should do the following:

This dependency is mostly relevant for persistence modules. We shall not declare these in parent pom.

Let's add this dependency in each module - scheduled-mysql-persistence and scheduled-postgres-persistence.

Conductor is using version 1.3.3, we shall also opt for the same.

jas34 · 2023-07-15T08:12:36Z

+		</dependency>
+
+
+		<dependency>


After removing this dependence from parent pom:

I am able to build.

I am able to run the application.

Can you elaborate you are getting this error on the application start or while running testes?

jas34 · 2023-07-15T08:33:09Z

 			<artifactId>spring-boot-starter</artifactId>
 			<version>${spring-boot-version}</version>
+
+			<exclusions>


Thanks for the correct findings. Let's keep the exclusion and add spring-boot-starter-log4j2 which is also used by conductor-server.

jas34 · 2023-07-15T08:34:16Z

 * 
 * @since v2.0.0
- * @author Jasbir Singh
+ * @author Jasbir Singh Vivian Zheng


I believe there is no change in this file, please remove this change.

jas34 · 2023-07-15T08:41:24Z

 * Date: 26/09/21-5:13 pm
 * @since v2.0.0
- * @author Jasbir Singh
+ * @author Jasbir Singh Vivian Zheng


Use either of the formats to add author:

@author Jasbir Singh, Vivian Zheng - use coma ',' in between names
or

@author Jasbir Singh \n @author Vivian Zheng - new line in between names.

Added add new line and since.

jas34 · 2023-07-15T11:47:35Z

    public DefaultSchedulerManager(ScheduledWfMetadataDAO scheduledWfMetadataDAO,
-            ScheduledProcessRegistry processRegistry, MetadataDAO metadataDAO, IndexScheduledWfDAO indexDAO,
-            WorkflowSchedulingAssistant schedulingAssistant) {
+                                   ScheduledProcessRegistry processRegistry, MetadataDAO metadataDAO, IndexScheduledWfDAO indexDAO,


Code format change

Removed the space.

jas34 · 2023-07-15T11:48:45Z

Apart from inline comments, the following are the general comments:

Add author in new files as well as add @SInCE.

Update the author with your name in all of the touched files.

We need to align of code style/format. If needed I can provide one which has been originally used in the older releases.

Update README.md for the addition of Postgres persistence layer. Currently, it mentions only MySQL implementation.

Reply:

I am curious about the reason for adding @SInCE

Done

Is the style/format referring to the formatter in this comment ? https://github.com/jas34/scheduledwf/pull/32/files#r1250801445 I am not aware, please elaborate.

Done

Even I am not aware of the person who got tagged (:P). My intention was to add Since for Java docs.
I am referring to formatting change like this.

jas34 and others added 9 commits January 10, 2023 08:41

[maven-release-plugin] prepare release v2.0.1

40be001

[maven-release-plugin] rollback the release of v2.0.1

5e95949

[maven-release-plugin] prepare release v2.0.1

11ed38b

[maven-release-plugin] rollback the release of v2.0.1

0b8c199

[maven-release-plugin] prepare release v2.0.1

03e7cfb

[maven-release-plugin] prepare for next development iteration

640dce0

init

ce811f2

Upgrade conductor 3.13.5 and postgres

34d4e77

Delete .gitignore

4eda29c

jas34 self-requested a review July 3, 2023 10:57

jas34 requested changes Jul 4, 2023

View reviewed changes

vivianzheng010 added 2 commits July 12, 2023 13:00

Code review 1

2cfa971

Merge remote-tracking branch 'origin/release-2.0.2' into release-2.0.2

9f88fb1

jas34 self-requested a review July 13, 2023 05:56

vivianzheng010 added 4 commits July 13, 2023 11:56

Resolve merge conflict of refactor 1

57ed3c9

Refactor 1

1ac05c0

Revert "Refactor 1"

ac1f28c

This reverts commit 1ac05c0.

Revert "Resolve merge conflict of refactor 1"

876ac72

This reverts commit 57ed3c9.

This was referenced Jul 15, 2023

Follow Exception Handling Design Similar To Conductor #33

Open

Guides on using different persistent storage: PostgreSQL #26

Closed

Get Priority Parameter As A Part Of Scheduled Workflow Definition #34

Open

Deprecate IDGenerator_.java #35

Open

jas34 reviewed Jul 15, 2023

View reviewed changes

jas34 self-requested a review August 13, 2023 13:53

vivianzheng404 changed the title ~~Release 2.0.2~~ Release 3.0.0 Aug 13, 2023

Refactor 1

92ab58e

Conversation

vivianzheng404 commented Jun 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jas34 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vivianzheng404 Jul 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vivianzheng404 Jul 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vivianzheng404 Aug 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vivianzheng404 Aug 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vivianzheng404 Aug 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vivianzheng404 commented Jul 12, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vivianzheng404 commented Jun 23, 2023 •

edited

Loading

jas34 left a comment •

edited

Loading

vivianzheng404 Jul 12, 2023 •

edited

Loading

vivianzheng404 Jul 12, 2023 •

edited

Loading

vivianzheng404 Aug 12, 2023 •

edited

Loading

vivianzheng404 Aug 12, 2023 •

edited

Loading

vivianzheng404 Aug 13, 2023 •

edited

Loading

vivianzheng404 Aug 13, 2023 •

edited

Loading