[DSIP-79][Task] Add datavines task for data quality by zixi0825 · Pull Request #16863 · apache/dolphinscheduler

zixi0825 · 2024-12-01T02:49:28Z

Purpose of the pull request

See #16113 for detail.

close #16113

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

… dev-datavines # Conflicts: # dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss

… dev-datavines

# Conflicts: # dolphinscheduler-task-plugin/pom.xml # dolphinscheduler-ui/src/locales/en_US/project.ts # dolphinscheduler-ui/src/locales/zh_CN/project.ts

SbloodyS · 2026-04-15T06:23:24Z

+@TestMethodOrder(MethodOrderer.MethodName.class)
+@DolphinScheduler(composeFiles = "docker/datavines-task/docker-compose.yaml")
+@DisableIfTestFails
+public class DatavinesTaskE2ETest extends BaseWorkflowE2ETest {


You should add this to

dolphinscheduler/.github/workflows/e2e.yml

Lines 137 to 138 in 5314ac0

- name: DolphinDBDataSourceE2ETest

class: org.apache.dolphinscheduler.e2e.cases.DolphinDBDataSourceE2ETest

SbloodyS · 2026-04-15T06:23:51Z

+
+    @Test
+    @Order(10)
+    void testRunDatavinesTasks_SuccessCase() {


It's better to add a failed test case.

SbloodyS · 2026-04-15T06:25:06Z

+      MASTER_MAX_CPU_LOAD_AVG: 400
+      MASTER_RESERVED_MEMORY: 0.01
+      WORKER_TENANT_AUTO_CREATE: 'true'


These configurations need to be consistent with other test cases.

SbloodyS · 2026-04-15T06:26:29Z

+        parse(doPost(address + DatavinesTaskConstants.JOB_EXECUTION_KILL + jobExecutionId, token));
+    }
+
+    public static JsonNode parse(String res) {


It's better to use JSONUtils instead of creating a new one.

Copilot

Pull request overview

Adds a new Datavines task type to DolphinScheduler to support triggering Datavines data quality jobs from workflows (UI configuration + worker task plugin + docs/E2E coverage).

Changes:

UI: register DATAVINES task type, add form fields (address/jobId/token/failureBlock), and add DAG icon + i18n strings.
Backend: add dolphinscheduler-task-datavines plugin module implementing task channel, parameters, and HTTP calls to Datavines OpenAPI.
Docs/E2E: add Datavines task docs (EN/ZH) and an E2E workflow test with a mock server.

Reviewed changes

Copilot reviewed 33 out of 38 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss	Adds Datavines task icon + hover icon styling in the DAG.
dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts	Adds `DATAVINES` to UI task type union and task type metadata map.
dolphinscheduler-ui/src/views/projects/task/components/node/types.ts	Extends task parameter typing to include Datavines fields.
dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-datavines.ts	New UI task node model + form composition for Datavines.
dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts	Registers `DATAVINES` to the task form factory map.
dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts	Serializes Datavines form fields into `taskParams`.
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-datavines.ts	New Datavines-specific form fields (address/jobId/token/failureBlock + custom params).
dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts	Exports the new Datavines field builder.
dolphinscheduler-ui/src/store/project/types.ts	Adds `DATAVINES` to the store-side task type union.
dolphinscheduler-ui/src/store/project/task-type.ts	Adds `DATAVINES` to store task type metadata map.
dolphinscheduler-ui/src/locales/zh_CN/project.ts	Adds Chinese i18n strings for Datavines form fields.
dolphinscheduler-ui/src/locales/en_US/project.ts	Adds English i18n strings for Datavines form fields.
dolphinscheduler-task-plugin/pom.xml	Adds `dolphinscheduler-task-datavines` module to task plugins build.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/pom.xml	New Datavines task plugin module (deps + shade config).
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTask.java	Implements the Datavines remote task execution/tracking/cancel logic.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesParameters.java	Defines Datavines task parameters and validation.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTaskChannel.java	Implements parameter parsing + task creation for the plugin.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTaskChannelFactory.java	Registers the Datavines task channel via SPI.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTaskConstants.java	Adds Datavines OpenAPI endpoints/status/constants.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTaskException.java	Adds a Datavines-specific runtime exception.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../utils/RequestUtils.java	Adds HTTP helper utilities for calling Datavines OpenAPI.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/test/java/.../DatavinesTaskTest.java	Adds unit tests for task behavior (with static mocking).
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/test/java/.../DatavinesTaskChannelTest.java	Adds unit tests for channel parsing and task creation.
dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/test/java/.../DatavinesTaskChannelFactoryTest.java	Adds unit tests for SPI factory name and instantiation.
dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml	Includes the Datavines task artifact in the “task-all” bundle.
dolphinscheduler-e2e/.../docker/datavines-task/mockserver-config.json	MockServer expectations for Datavines OpenAPI endpoints used in E2E.
dolphinscheduler-e2e/.../docker/datavines-task/docker-compose.yaml	E2E compose to run standalone DS + mock Datavines server.
dolphinscheduler-e2e/.../pages/project/workflow/task/DatavinesTaskForm.java	Adds Selenium page object for filling Datavines task form.
dolphinscheduler-e2e/.../pages/project/workflow/WorkflowForm.java	Adds `DATAVINES` task type wiring to workflow form helper.
dolphinscheduler-e2e/.../cases/tasks/DatavinesTaskE2ETest.java	Adds E2E test that creates/runs a workflow with Datavines task.
dolphinscheduler-api/src/main/resources/task-type-config.yaml	Adds `DATAVINES` under a new `dataQuality` group (and updates ML list).
docs/docs/zh/guide/task/datavines.md	Adds Chinese documentation page for Datavines task.
docs/docs/en/guide/task/datavines.md	Adds English documentation page for Datavines task.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+
+    @Override
+    public boolean checkParameters() {
+        return StringUtils.isNotEmpty(this.address) && StringUtils.isNotEmpty(this.jobId);


+    public static String doGet(String url, String token) {
+        String result = "";
+        HttpClient httpClient = HttpClientBuilder.create().build();
+        HttpGet httpGet = null;
+        try {
+            URIBuilder uriBuilder = new URIBuilder(url);
+            URI uri = uriBuilder.build();
+            httpGet = new HttpGet(uri);
+            httpGet.setHeader("Authorization", "Bearer " + token);
+            log.info("access url: {}", uri);
+            HttpResponse response = httpClient.execute(httpGet);
+            if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
+                result = EntityUtils.toString(response.getEntity());
+                log.info("datavines task succeed with results: {}", result);
+            } else {
+                log.error("datavines task terminated,response: {}", response);
+            }
+        } catch (IllegalArgumentException ie) {
+            log.error("datavines task terminated: {}", ie.getMessage());
+        } catch (Exception e) {
+            log.error("datavines task terminated: ", e);
+        } finally {
+            if (null != httpGet) {
+                httpGet.releaseConnection();
+            }
+        }
+        return result;


+    void submitApplicationExecutesJobSuccessfully() {
+        when(taskExecutionContext.getTaskParams())
+                .thenReturn("{\"address\":\"http://localhost\",\"jobId\":\"1\",\"token\":\"token\"}");
+        datavinesTask.init();
+        assertDoesNotThrow(() -> datavinesTask.submitApplication());
+    }
+
+    @Test
+    void trackApplicationStatusJobExecutionSuccessSetsExitCodeSuccess() throws TaskException {
+        JsonNode executeJobResult = RequestUtils.parse("{\"code\":\"200\",\"data\":\"1\"}");
+        JsonNode executeStatus = RequestUtils.parse("{\"code\":\"200\",\"data\":\"SUCCESS\"}");
+        JsonNode executeResult = RequestUtils.parse("{\"code\":\"200\",\"data\":\"1\"}");
+        try (MockedStatic<RequestUtils> requestUtilsStatic = Mockito.mockStatic(RequestUtils.class)) {
+            when(taskExecutionContext.getTaskParams())
+                    .thenReturn("{\"address\":\"http://localhost\",\"jobId\":\"1\",\"token\":\"token\"}");
+            datavinesTask.init();
+
+            requestUtilsStatic.when(() -> RequestUtils.executeJob(Mockito.any(), Mockito.any(), Mockito.any()))
+                    .thenReturn(executeJobResult);
+            datavinesTask.submitApplication();
+
+            requestUtilsStatic
+                    .when(() -> RequestUtils.getJobExecutionStatus(Mockito.any(), Mockito.any(), Mockito.any()))
+                    .thenReturn(executeStatus);
+
+            requestUtilsStatic
+                    .when(() -> RequestUtils.getJobExecutionResult(Mockito.any(), Mockito.any(), Mockito.any()))
+                    .thenReturn(executeResult);
+
+            datavinesTask.trackApplicationStatus();
+            Assertions.assertEquals(EXIT_CODE_SUCCESS, datavinesTask.getExitStatusCode());
+        }
+    }
+
+    @Test
+    void trackApplicationStatusJobExecutionFailureSetsExitCodeFailure() throws TaskException {
+        when(taskExecutionContext.getTaskParams())
+                .thenReturn("{\"address\":\"http://localhost\",\"jobId\":\"1\",\"token\":\"token\"}");
+        datavinesTask.init();
+        datavinesTask.submitApplication();
+        datavinesTask.trackApplicationStatus();
+        Assertions.assertEquals(EXIT_CODE_FAILURE, datavinesTask.getExitStatusCode());


+  jobId?: string
+  token?: string
+  failureBlock?: string


+        if (result instanceof MissingNode || result instanceof NullNode) {
+            errorHandle(DatavinesTaskConstants.API_ERROR_MSG);
+            isCorrect = false;
+        } else if (result.get(DatavinesTaskConstants.API_RESULT_CODE)
+                .asInt() != DatavinesTaskConstants.API_RESULT_CODE_SUCCESS) {
+            errorHandle(result.get(DatavinesTaskConstants.API_RESULT_MSG));


+package org.apache.dolphinscheduler.plugin.task.datavines;
+
+/**
+ * Custom DinkyTaskConstants


+|-------------------|-------------------------------------------------------------------------------------------------------|
+| Datavines Address | The URL for the Datavines service, e.g., http://localhost:5600.                                       |
+| Datavines Job ID  | The unique job id for a datavines job.                                                                |
+| Datavines token   | The Datawines service access token can be obtained through token management on the Datavines service. |


    - 'OPENMLDB'
    - 'DVC'
    - 'SAGEMAKER'
+    - 'PYTORCH'


+import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;


+                // Use address-taskId as app id
+                setAppIds(String.format(DatavinesTaskConstants.APPIDS_FORMAT, address, this.jobExecutionId));


sonarqubecloud · 2026-04-16T06:51:13Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 60%)

See analysis details on SonarQube Cloud

zixi0825 added 2 commits December 1, 2024 09:43

[Feature][Task] Add datavines task for data quality

faa2ae7

[Fix][Task] Fix conflicts

8394ea5

zixi0825 requested review from EricGao888, Gallardot, SbloodyS, caishunfeng and songjianet as code owners December 1, 2024 02:49

github-actions bot added UI ui and front end related backend document labels Dec 1, 2024

github-actions bot assigned zixi0825 Dec 1, 2024

[Fix][Task] Fix format issue

501b9ff

SbloodyS added the DSIP label Dec 1, 2024

SbloodyS added this to the 3.3.0 milestone Dec 1, 2024

SbloodyS changed the title ~~[Feature][Task] Add datavines task for data quality~~ [DSIP-79][Task] Add datavines task for data quality Dec 1, 2024

zixi0825 added 8 commits December 1, 2024 11:10

[Fix][Task] Fix img issue

bce54a5

[Fix][Task] Add header

6a12fe2

[Fix][Task] Fix docs issue

6976842

[Fix][Task] Fix package issue

0db891c

[Fix][Task] Fix format issue

dac3c28

[Fix][Task] Fix conflicts

0c1d483

Merge branch 'dev' of https://github.com/apache/dolphinscheduler into…

28fdfd6

… dev-datavines # Conflicts: # dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss

[Fix][Task] Fix code style issue

8ad0a2a

zixi0825 force-pushed the dev-datavines branch from 9267125 to 8ad0a2a Compare January 28, 2025 01:36

zixi0825 added 4 commits January 28, 2025 09:43

[Fix][UI] Fix code style issue (1)

27067af

[Fix][Task] Fix build error

d496c2c

[Test][Task] Add datavines task tests

603b667

Merge branch 'dev' of https://github.com/apache/dolphinscheduler into…

5d4691c

… dev-datavines

zixi0825 force-pushed the dev-datavines branch from 27c3e67 to 5d4691c Compare January 29, 2025 02:37

github-actions bot added the test label Jan 29, 2025

zixi0825 added 13 commits April 15, 2026 10:39

[Fix][Task] Fix format issue

58a032f

[Fix][Task] Fix conflicts

efcb5b6

[Fix][Task] Fix code style issue

d5a8b38

[Fix][UI] Fix code style issue (1)

376be7d

[Fix][Task] Fix build error

a51e615

[Test][Task] Add datavines task tests

92037e4

[Fix][Task] Fix code style

2685a93

[Fix][Task] Fix code style issue

88b73fd

[Test][Task] Add unit test

280f315

[Fix][Task] Fix issues

de3a71f

[Fix][Task] Fix issues

dfe8eb0

[Feature][Task] Add datavines task e2e

3ec722e

Merge remote-tracking branch 'origin/dev-datavines' into dev-datavines

7c67501

# Conflicts: # dolphinscheduler-task-plugin/pom.xml # dolphinscheduler-ui/src/locales/en_US/project.ts # dolphinscheduler-ui/src/locales/zh_CN/project.ts

zixi0825 requested a review from ruanwenjun as a code owner April 15, 2026 02:42

github-actions bot added docker kubernetes CI&CD labels Apr 15, 2026

zixi0825 added 2 commits April 15, 2026 11:55

[Feature][Task] Add datavines task test

3acaa23

Merge branch 'dev' into dev-datavines

42ad057

github-actions bot removed docker kubernetes CI&CD labels Apr 15, 2026

SbloodyS requested changes Apr 15, 2026

View reviewed changes

SbloodyS requested a review from Copilot April 15, 2026 06:28

Copilot started reviewing on behalf of SbloodyS April 15, 2026 06:28 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

zixi0825 added 2 commits April 16, 2026 14:37

[Fix][Task] Fix datavines task test

9b02337

Merge remote-tracking branch 'origin/dev-datavines' into dev-datavines

0304312

github-actions bot added the CI&CD label Apr 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DSIP-79][Task] Add datavines task for data quality#16863

[DSIP-79][Task] Add datavines task for data quality#16863
zixi0825 wants to merge 64 commits intoapache:devfrom
zixi0825:dev-datavines

zixi0825 commented Dec 1, 2024 •

edited by SbloodyS

Loading

Uh oh!

SbloodyS Apr 15, 2026

Uh oh!

SbloodyS Apr 15, 2026

Uh oh!

SbloodyS Apr 15, 2026

Uh oh!

SbloodyS Apr 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

sonarqubecloud bot commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	- name: DolphinDBDataSourceE2ETest
	class: org.apache.dolphinscheduler.e2e.cases.DolphinDBDataSourceE2ETest

		import static org.junit.Assert.assertThrows;
		import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;

		// Use address-taskId as app id
		setAppIds(String.format(DatavinesTaskConstants.APPIDS_FORMAT, address, this.jobExecutionId));

Conversation

zixi0825 commented Dec 1, 2024 • edited by SbloodyS Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the pull request

Brief change log

Verify this pull request

Pull Request Notice

Uh oh!

SbloodyS Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

SbloodyS Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

SbloodyS Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

SbloodyS Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

sonarqubecloud bot commented Apr 16, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zixi0825 commented Dec 1, 2024 •

edited by SbloodyS

Loading