[Feature Request]: Data Validator - Enable Rules as a Static Schema Definition

[dmainou]

What would you like to happen?

Summary

Enhance the Data Validator transform to support loading validation rules from a Static Schema Definition (Metadata Perspective) using a JSON document.

Instead of manually configuring every validation rule inside the transform, users would be able to reference a JSON schema stored in the Metadata Perspective. The transform would interpret the schema and create the validation rules at runtime.

This would allow validation logic to be externalised from the pipeline while remaining version-controlled, reusable and understandable by non-Hop users.

---

Problem

The current Data Validator requires all validation rules to be manually configured within the transform.

This becomes difficult when:

• Validation rules are generated externally.
• Multiple test suites require different validation sets.
• The same validation logic is reused across many pipelines.
• Governance teams need to review or approve validation rules.
• Validation rules evolve independently of pipeline logic.

Changing validation rules currently requires editing the pipeline itself.

---

Proposed Solution

Add support for selecting a Static Schema Definition from the Metadata Perspective.

The selected schema would contain the validator definitions in JSON.

The Data Validator would:

1. Load the JSON schema.
2. Convert it internally into the existing validator definitions.
3. Execute validations exactly as it does today.

This preserves full backwards compatibility while allowing validations to be managed independently of the pipeline.

---

Example

Current XML representation

<validator_field>
  <allowed_value>
    <value>AAA</value>
    <value>ABC</value>
  </allowed_value>
  <conversion_mask>ddmm</conversion_mask>
  <data_type>String</data_type>
  <data_type_verified>Y</data_type_verified>
  <decimal_symbol>.</decimal_symbol>
  <end_string>@#$</end_string>
  <end_string_not_allowed>df</end_string_not_allowed>
  <error_code>Error1</error_code>
  <error_description>Error 1</error_description>
  <grouping_symbol>,</grouping_symbol>
  <is_sourcing_values>Y</is_sourcing_values>
  <max_length>5</max_length>
  <max_value>234</max_value>
  <min_length>234</min_length>
  <min_value>234</min_value>
  <name>String1</name>
  <null_allowed>Y</null_allowed>
  <only_null_allowed>Y</only_null_allowed>
  <only_numeric_allowed>Y</only_numeric_allowed>
  <regular_expression>.*</regular_expression>
  <regular_expression_not_allowed>.*</regular_expression_not_allowed>
  <sourcing_field>field</sourcing_field>
  <sourcing_transform>src</sourcing_transform>
  <start_string>34</start_string>
  <start_string_not_allowed>QW</start_string_not_allowed>
  <validation_name>Sample1</validation_name>
</validator_field>

<validator_field>
  <allowed_value>
    <value>df</value>
    <value>df</value>
  </allowed_value>
  <conversion_mask>234rt</conversion_mask>
  <data_type>ert</data_type>
  <data_type_verified>Y</data_type_verified>
  <decimal_symbol>234</decimal_symbol>
  <end_string>2wer</end_string>
  <end_string_not_allowed>wer</end_string_not_allowed>
  <error_code>Error2</error_code>
  <error_description>Error2</error_description>
  <grouping_symbol>34rt</grouping_symbol>
  <is_sourcing_values>Y</is_sourcing_values>
  <max_length>234r</max_length>
  <max_value>we</max_value>
  <min_length>wer</min_length>
  <min_value>wer</min_value>
  <name>String2</name>
  <null_allowed>Y</null_allowed>
  <only_null_allowed>Y</only_null_allowed>
  <only_numeric_allowed>Y</only_numeric_allowed>
  <regular_expression>we</regular_expression>
  <regular_expression_not_allowed>we</regular_expression_not_allowed>
  <sourcing_field>dfv</sourcing_field>
  <start_string>we</start_string>
  <start_string_not_allowed>er</start_string_not_allowed>
  <validation_name>Sample2</validation_name>
</validator_field>

Equivalent JSON

[
  {
    "allowedValue": [
      "AAA",
      "ABC"
    ],
    "conversionMask": "ddmm",
    "dataType": "String",
    "dataTypeVerified": true,
    "decimalSymbol": ".",
    "endString": "@#$",
    "endStringNotAllowed": "df",
    "errorCode": "Error1",
    "errorDescription": "Error 1",
    "groupingSymbol": ",",
    "isSourcingValues": true,
    "maxLength": 5,
    "maxValue": "234",
    "minLength": 234,
    "minValue": "234",
    "name": "String1",
    "nullAllowed": true,
    "onlyNullAllowed": true,
    "onlyNumericAllowed": true,
    "regularExpression": ".*",
    "regularExpressionNotAllowed": ".*",
    "sourcingField": "field",
    "sourcingTransform": "src",
    "startString": "34",
    "startStringNotAllowed": "QW",
    "validationName": "Sample1"
  },
  {
    "allowedValue": [
      "df",
      "df"
    ],
    "conversionMask": "234rt",
    "dataType": "ert",
    "dataTypeVerified": true,
    "decimalSymbol": "234",
    "endString": "2wer",
    "endStringNotAllowed": "wer",
    "errorCode": "Error2",
    "errorDescription": "Error2",
    "groupingSymbol": "34rt",
    "isSourcingValues": true,
    "maxLength": "234r",
    "maxValue": "we",
    "minLength": "wer",
    "minValue": "wer",
    "name": "String2",
    "nullAllowed": true,
    "onlyNullAllowed": true,
    "onlyNumericAllowed": true,
    "regularExpression": "we",
    "regularExpressionNotAllowed": "we",
    "sourcingField": "dfv",
    "startString": "we",
    "startStringNotAllowed": "er",
    "validationName": "Sample2"
  }
]

---

Benefits

Reusable validation

The same validation schema can be used across many pipelines without duplication.

Easier testing

Entire validation suites can be swapped simply by selecting a different Static Schema Definition.

For example:

• Development validations
• Integration validations
• Production validations
• Client-specific validations

No pipeline changes required.

Reusable validation pipelines

A generic validation pipeline could be built where only three metadata objects change:

• Input schema
• Validation schema
• Output schema

This enables highly reusable, metadata-driven pipelines.

Better governance

JSON is significantly easier for business users, governance teams and data stewards to review than transform XML.

Validation rules can be:

• Peer reviewed
• Version controlled
• Automatically generated
• Validated using JSON Schema
• Managed outside Hop

Easier upgrades

Validation logic can evolve independently from pipeline logic.

Changing validation rules would not require opening, modifying or redeploying pipelines.

Better interoperability

External applications could generate validation definitions programmatically, allowing Hop to consume validation rules from governance platforms, metadata repositories or code generators.

This also opens the possibility of generating validation schemas from other metadata sources or enterprise data catalogues.

---

Backwards Compatibility

This feature could be completely optional.

Existing Data Validator transforms would continue to work exactly as they do today.

Users could choose between:

• Embedded validator definitions (current behaviour)
• Static Schema Definition (JSON)

Issue Priority

Priority: 2

Issue Component

Component: Hop Gui

[jballment]

While this can currently be implemented using a metadata injection; metadata injection adds a layer of complexity to the ETL process that could very easily be removed with the suggested change.

[CarlosJuncher03]

Another aspect worth examining is the "data validation" transformation's ability to handle moderate workloads. Currently, because validation is performed row-by-row, a bottleneck arises; furthermore, the use of external sources for validation limits scalability, and memory consumption appears to be quite high. It might be possible to implement a more efficient approach, such as batch processing or parallel execution. Just a thought.

Regarding parallel execution: I’m not sure if it’s a bug, but if I use a "Table Input" step for data validation and enable parallel execution, the step fails to locate the data from the "Table Input."

[hansva]

Split that to a separate ticket

[CarlosJuncher03]

I am going to open it, though; I just mentioned it here to share the idea—sorry.