Add support for skipping partial shell script creation.

[eladrion]

Pull Request Overview

Implement a configuration option create_partial_scripts (false by default) and only create partial scripts with missing tool information as comment if set to true. Otherwise, log output file and missing tools as error.
Also, the created scripts contain the same header (generation info) as CWL files, for example

Related Issue

Resolves #50

Changes Introduced

• Added configuration related parts in APERunConfig and APEConfigTagFactory
• Added generation info in SolutionGraphFactory
• Updated writeExecutableWorkflows in APE to react on the configuration option
• Updated tests to use create_partial_scripts with value true to behave as before

How Has This Been Tested?

Locally using test suite.

Checklist

• I have referenced a related issue.
• I have followed the project’s style guidelines.
• My changes include tests, if applicable.
• All tests pass locally.
• I have added myself to the CITATION.cff file, if not already present.

[eladrion]

Hi @vedran-kasalica, could you take a look into this solution? Creating non-functional scripts by default felt wrong to me, so I only do it if explicitly desired. Otherwise, I log an error message with information about the tool names where the code is missing.

[vedran-kasalica]

Thanks @eladrion, that’s a good idea, I will have a look at it (I am currently not able to generate the .sh files, the library just skips generating them, but it might be something wrong on my end).

I’m wondering whether the tag could be used for all workflow outputs (cwl, snakemake), and not only for .sh files. Otherwise, the scope of the field might not be clear.

On a more general note, the .sh outputs are quite outdated (they were used as a proof of concept and never in practice).

[eladrion]

I think the tag could be reused for CWL. CWL requires a cwl_reference instead of codefor the implementation. I have to look into the Snakemake part whether this tag could be used there, too.

[eladrion]

@vedran-kasalica, @CGru21: I exteded the support for skipping partial shell scripts to CWL and Snakemake export. A CWL workflow is seen as partial, if the CWL file reference is null while a Snakemake workflow is seen as partial, if there is no code entry in the tool annotation. What are your opinions about that?

[eladrion]

@vedran-kasalica Currently, I first check (if partial implementations shall be skipped), whether there is missing information about CWL reference or code. This requires iterating over the modules. And if all information is present, the workflow is generated, which again includes iterating over the modules. I would propose that I propagate the "skipping" flag to the creation of the workflow and if true throw a custom exception that is then caught. This way, we would iterate only once. What do you think about that? And would it be sufficient to then throw instantly when encountering a missing definition or should I track all missing definitions and then throw with a list of the missing implementations?

[eladrion]

Rerunning Github action on the current basis of main.

[eladrion]

@CGru21 Could you look into this code and review it? Also, here is a sample config JSON:

{
  "ontology_path": "https://raw.githubusercontent.com/Workflomics/tools-and-domains/main/domains/edam.owl",
  "ontologyPrefixIRI": "http://edamontology.org/",
  "toolsTaxonomyRoot": "operation_0004",
  "dataDimensionsTaxonomyRoots": ["data_0006", "format_1915"],
  "tool_annotations_path": "https://raw.githubusercontent.com/Workflomics/tools-and-domains/main/domains/proteomics/tools.json",
  "constraints_path": "https://raw.githubusercontent.com/Workflomics/tools-and-domains/main/domains/proteomics/constraints.json",
  "strict_tool_annotations": "true",
  "timeout_sec": "120",
  "solutions_dir_path": ".",
  "create_partial_scripts": "false",
  "solution_length": {
    "min": 1,
    "max": 6
  },
  "solutions": "10",
  "number_of_execution_scripts": "10",
  "number_of_generated_graphs": "10",
  "number_of_cwl_files": "10",
  "number_of_snakemake_files": "10",
  "debug_mode": "false",
  "use_workflow_input": "all",
  "use_all_generated_data": "one",
  "tool_seq_repeat": "false",
  "inputs": [
    {
      "data_0006": ["data_0943"],
      "format_1915": ["format_3244"]
    },
    {
      "data_0006": ["data_2976"],
      "format_1915": ["format_1929"]
    }
  ],
  "outputs": [
    {
      "data_0006": ["data_3753"],
      "format_1915": ["format_3464"]
    }
  ]
}

The config yields 7 files for CWL and no files for shell scripts and Snakemake.

[CGru21]

I looked at the snakemake part and everything works fine with my examples.

[eladrion]

I looked at the snakemake part and everything works fine with my examples.

Hi @CGru21, could you then approve, so I can merge?

[eladrion]

Aaah, don't need that. Merging now