[Feature Request]: Enhanced Configuration File Selection in Environment Properties

[dsanderbi]

What would you like to happen?

I would like to request an enhancement for the way configuration files are added in Environment Properties.

Instead of selecting individual files manually, it would be great if the selection worked similarly to the JSON Input transform.

Specifically, it would be helpful to have the following options:

• File/Directory Selection (allowing users to select an entire folder)
• Wildcard (RegExp) to filter specific files
• Exclude Wildcard to ignore certain files
• Required flag for mandatory files
• Include Subfolders option to process files in subdirectories

This would greatly improve usability and flexibility when managing configuration files.

Currently:

After feature implementation:

NOTE:
It would also be great to have this functionality for the hop-conf script.

Example:

$ sh hop-conf.sh \
     --environment-create \
     --environment test-file \
     --environment-project test-file \
     --environment-purpose=Testing\
     --environment-config-file-directory="<YOUR_ENV_CONFIG_PATH>/config"
     --environment-config-file-wildcard="*.json"

Issue Priority

Priority: 2

Issue Component

Component: Hop Gui, Component: Hop Config

[hansva]

Great idea!

[dsanderbi]

@sramazzina
It would also be great if the option were available for Docker as well.

[sramazzina]

Ciao @dsanderbi, is not clear to me what do you mean by saying It would also be great if the option were available for Docker as well. Can you give elaborate a bit more about that?

[dsanderbi]

Hi @sramazzina,

The current Apache Hop Docker image already provides HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS.
This variable allows you to specify a comma separated list of config file paths.

To avoid picking those files one by one it would be great if you could use the same functionality as in the GUI (or the command line)
allowing you to simply specify a folder and a regex/wildcard to pick the files automatically.

Something like this:

docker run  --rm \
  -e HOP_PROJECT_NAME=test \
  -e HOP_ENVIRONMENT_NAME=test \
  -e HOP_LOG_LEVEL=BASIC
  -e HOP_ENVIRONMENT_CONFIG_FILE_DIRECTORY="/config" \
  -e HOP_ENVIRONMENT_CONFIG_FILE_WILDCARD="*.json" \
  -e HOP_ENVIRONMENT_CONFIG_FILE_EXCLUDE="*secret*.json" \
  -e HOP_ENVIRONMENT_CONFIG_INCLUDE_SUBFOLDERS=true \
  apache/hop:latest

Since these values still need to be passed to hop-config.sh anyway, someone else could handle that in a separate ticket.

The main question is how this would be represented in load-and-execute.sh in combination with HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS.

So far, what was your idea for implementing this, in the context of hop-config.sh?

Does the environment-config-files parameter still exist, or do you plan to replaced by the new options like
environment-config-file-directory?
Or can be used both parameters, like this:

$ sh hop-conf.sh \
     --environment-create \
     --environment test-file \
     --environment-project test-file \
     --environment-purpose=Testing\
     --environment-config-file-directory="<YOUR_ENV_CONFIG_PATH>/config"
     --environment-config-file-wildcard="*.json"
     --environment-config-files="config_path_1/file1.json,config_path2/sub_path/file2.json"  # already exists

[sramazzina]

@dsanderbi thanks for clarifying. One last question related to your idea of using wildcards in the GUI dialog. The dialog has a button to edit a selected environment file. If we use the wildcards to choose more than one file at a time in an efficient way, how can we edit them using the edit button if they are specified in that way? I hope you understand what I mean

[dsanderbi]

Hi @sramazzina, very good point, i hadn't considered that at all.

The only thing that comes to my mind is that, when adapting the environment, there could be an option to specify a RegExp or a single file like in the get file names transform.
And then when hitting Add all files get loaded in the section below, where you can adapt the files.

After the Add

Would something like that make sense?