feat: Add the possibility to use the dirac-cwl project's job wrapper

[Loxeris]

Adds slights changes to create_job_wrapper() to use the experimental Job Wrapper for CWL jobs.
If the job executable is set to dirac-cwl-exec in the JDL, the new job wrapper is used, else the defaultWrapperLocation is used like before.
The dirac-cwl project is cloned locally on the pilot to get the latest version of the job wrapper before being run.

See DIRACGrid/dirac-cwl#44

BEGINRELEASENOTES

*WorkloadManagementSystem
NEW: Experimental job wrapper for cwl workflows

ENDRELEASENOTES

[aldbr]

Looks good to me!
Note for the other reviewer: the new job wrapper is experimental and is not expected to be executed at a large scale. It's only meant to be used for a few jobs in parallel for now.

@Loxeris have you tested it using the SingularityComputingElement?
The JobAgent can use the SingularituComputingElement instead of the InProcessComputingElement. In that case, the job wrapper is generated in a slightly different way:

DIRAC/src/DIRAC/Resources/Computing/SingularityComputingElement.py

Lines 211 to 222 in 11bc43d

	result = createJobWrapper(
	wrapperPath=tmpDir,
	rootLocation=self.__innerdir,
	jobID=jobDesc.get("jobID", 0),
	jobParams=jobDesc.get("jobParams", {}),
	resourceParams=jobDesc.get("resourceParams", {}),
	optimizerParams=jobDesc.get("optimizerParams", {}),
	pythonPath="python",
	log=log,
	logLevel=logLevel,
	extraOptions="" if self.__installDIRACInContainer else "/tmp/pilot.cfg",
	)

[Loxeris]

@Loxeris have you tested it using the SingularityComputingElement? The JobAgent can use the SingularituComputingElement instead of the InProcessComputingElement. In that case, the job wrapper is generated in a slightly different way

@aldbr I believe the computing element used by the CTAO test instance I did end-to-end tests on is SSHComputingElement. I didn't try using a different type of computing element.

[fstagni]

Just to say that that PR must first be tested using the DIRAC certification setup (instructions not written down yet, will be there for the hackathon next week)

[aldbr]

@Loxeris managed to test the PR in certification.
There are still missing pieces to properly report the status of the jobs (status is stuck in Matched), but this will be handled later in the dirac-cwl repo.
The pilot output reported the job as Done though.

[aldbr]

Is there any reason why this job.json file is located in 3 different locations?
As we are still working on the cwl hints, this workflow could potentially break at some point, so we could try to have it only in one place to avoid having to modify too many files at the same time.

[fstagni]

oops, sorry that was my fault. This is the only location needed -- please @Loxeris remove the other 2.