RCDS will support github tags - add that support to jobsub

[shreyb]

Overview

Dave Dykstra, Andrew McNab, and I had a discussion about a proposed feature DUNE asked RCDS to add, which would allow RCDS to download a repository for particular tag from github, tar it up, and then publish the tarball to RCDS. The proposed RCDS API for a client to publish such a tarball is:

publish?cid={vo}/{hash}&source=github.com/{owner}/{repo}

where the {hash} should be the appropriate commit hash for the desired tag. The CID would both serve as the hash that RCDS uses to keep track of the tarball, and also the guide to RCDS as to which commit of a repository to pull down.

Details

Thus jobsub will need to do the following (or something like it):

1. Modify the --tar-file-name flag so we have an additional URI, maybe git:// that takes a repository/tag combo. For example: git://github.com/fermitools/jobsub_lite:1.12.
2. jobsub should know, with this URI, to NOT create or check a tarball, but rather to run the github API https://api.github.com/repos/{owner}/{repo_name}/git/ref/tags/{tag}. The response is a JSON document, whose object.sha entry contains the commit hash for that tag. See, for example, https://api.github.com/repos/fermitools/jobsub_lite/git/ref/tags/1.12
3. This commit hash will be used to construct the CID for the TarballPublisherHandler instantiation. Like before, the CID will be something like vo/hash; e.g. fermilab/1bf8d373dc04024ae714e072ca059846937a0add.
4. The RCDS API call to publish the tarball will then need to be a little different than before, as mentioned above in the Overview. jobsub will have to parse the user-given repo/tag to create the API call:
   publish?cid={cid from (3)}&source={user-given-repo}

From then on, the logic is the same. The default behavior would be for jobsub to check for the successful publish of the tarball before proceeding; or the user could skip that check with --skip-check rcds.

[shreyb]

@DrDaveD opened this issue for the RCDS work: cvmfs-contrib/cvmfs-user-pub#15

[retzkek]

1. Please be absolutely sure that we want to go down the road of resolving tags to commit hashes. It's extra plumbing and instills/supports poor habits in users.
2. Review GitHub API rate limits and consider if we're going to run into any, and how to mitigate that
3. "Resolving GitHub URLs" is something many tools do, so it's worth a prior art check for useful libraries or snippets. Spack maybe has something?
4. We should support a "generic" git URL option, which includes the hash, as a base case, allowing immediate use beyond GitHub.

[DrDaveD]

1. Please be absolutely sure that we want to go down the road of resolving tags to commit hashes. It's extra plumbing and instills/supports poor habits in users.

What do you mean, what poor habits? It still requires a tag, it is just using the hash internally.

2. Review [GitHub API rate limits](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api) and consider if we're going to run into any, and how to mitigate that

Oh, very good to know. There should be only one call needed per job batch submission, but the rate limits are very low if you're unauthenticated, only 60 per hour. Jobsub would probably need to use a personal access token which allows up to 5000 requests per hour. That should be enough unless a job submission script is abusive and makes many separate calls to submit a small number of jobs each, so maybe a tag to hash cache will need to be kept to avoid the api lookup.

3. "Resolving GitHub URLs" is something many tools do, so it's worth a prior art check for useful libraries or snippets. [Spack](https://github.com/spack/spack) maybe has something?

What do you mean by "Resolving GitHub URLs"?

4. We should support a "generic" git URL option, which includes the hash, as a base case, allowing immediate use beyond GitHub.

I don't know what you mean by that either, although we have been discussing via chat today to be able to at least support tagged github release assets in addition to tagged source code. We're leaning toward making the RCDS API source parameter be essentially a full URL to download, but having the RCDS configuration list the allowed server names. Initially that will be only github.com but others could be added by the operations team on request.

[retzkek]

What do you mean, what poor habits?

Specifying mutable dependencies. Especially when reproducibility is important! But this is not my hill to die on, so I leave it at that.

What do you mean by "Resolving GitHub URLs"?

Given a project/repo and tag ref resolve it to a URL to download the source tarball. OK for jobsub's purposes we only care about getting the commit hash, the main point is not reinventing from scratch working with the GitHub API (and maybe others)

We're leaning toward making the RCDS API source parameter be essentially a full URL to download

That's what I mean.

[DrDaveD]

the main point is not reinventing from scratch working with the GitHub API

Looking up a hash from a tag is such a simple interaction that it does not seem to me to be worth introducing a library dependency.

[retzkek]

Looking up a hash from a tag is such a simple interaction

right, no complications or edge cases to think about...

maybe a tag to hash cache will need to be kept

oh

I agree it would need to be a compelling case to add a library dependency, but there's other options down to reviewing code/comments to see what cases one may have missed. I included that as a reminder since we have a bad track record of forging ahead with blinders on and not considering edge or error conditions.

Frankly that's why I think we shouldn't even do a hash lookup and leave that to users.

[DrDaveD]

Frankly that's why I think we shouldn't even do a hash lookup and leave that to users.

Oh I didn't realize that's what you were trying to advocate for. That's a possible approach. The disadvantage though is that it leaves out the requirement to make a tag, which is also an important habit to encourage.