This is several components that work together.
All http endpoints have a token issued and written to a configuration file directory found at /keys.d/ inside the containers with the http servers. The keys.d file format is a yaml dicionary, where the key is a token to be presented to conduct an action, followed by a list of regular expressions that match what the token is allowed to do (including HTTP method and path). This directory is rescanned on every request, with an in-memory cache of 60 seconds.
Our ability to clone a repo may depend on the upstream existing. Let's have a cache so we can get updates and keep references (tags, branches), locally.
To do that we have a small go program that sits on top of a disk. That disk stores all the various refs for all the cloned repositories that we are told about. These are stored in a single bare git repository using a git namespace per source repository to deduplicate storage as much as reasonable.
To tell the go program to update and make sure the ref is up to date, hit the endpoint: POST /sync/$reponame/$ref.
To add a repo to the cache, hit PUT /sync/$reponame with the body being a json object as such: {"url": "git@github.com:owner/repo.git", "ssh-reading-private-key": "SSH_KEY_HERE"}. SSH keys are stored content-addressed (the sha256sum of the ssh key is the filename, and the repository is configured to use that ssh key for any updates.
To fetch from the cache, use GET /download/$reponame/$ref.tar.gz and git archive will produce the latest tarball of the reference for download.
The Cache is two parts. One is a go program, and the other is a dockerfile running the one program with git installed, and a volume mount at /repos for the git repositories we cache.
This is a tiny go program on top of a volume. The abstraction used in all our go programs for filesystem access is fs.FS, to allow portability. To install an artifact, PUT /$path and the $path can have multiple slashes in it. To get the artifact back, it's simply GET /$path. On GET, the "modification time" of the file is updated to allow a cleaning process to remove obsolete artifacts.
All content is stored hash-addressed, with the sha256sum pointing at the entry, and the relevant paths storing only the reference to the filename
You can also upload containers to Artifacts and use it as an OCI registry.
All PUT calls have the variable reftype with them. If the reftype is branch then the reftype must always be branch. If the reftype is tag then the uploaded artifact can not be changed later, and is marked immutable and all future PUT calls will fail.
This is a temporal enabled go application that pulls a tarball from the Cache service, runs make build in the directory (along with any environment variables that were passed in in the Env key), runs make upload to upload, and then runs make deploy. Finally the working directory is fully removed. The clone, build, upload, deploy, and clean steps are performed with strict affinity so the same worker gets all of them.
This is a hook that is installed on a git repository that calls KICKOFF /$repository/$ref when a push happens to the git repository. Like all other HTTP actions, this requires a bearer token installed. This triggers the overall building temporal workflow. When a subsequent KICKOFF is received for a workflow already in progress, the workflow is sent a signal. The workflow then cancels any existing build, and re-runs from the cache command.
An example way to kickoff:
curl --data @./.vaelci.json -H "Authorization: Bearer $token" -X KICKOFF http://localhost:8083/$repo/$ref
The data for a kickoff is
{
"repository": "repo-name",
"ref": "v0.1.0",
"build-pattern": "MakeBuildUpload",
"compat-patch": ""
}
where the compat-patch will be fed to git apply on the source tree before the build is triggered. Yes, git apply on a random directory does work. This feature is meant to only be used to carry local CI compatibility patches if the upstream is not suitable, such as an upstream you do not control but wish to have your CI build anyway.
There is also a way to set off webhooks from the major git hosting platforms. The webhook URL will include repo but not ref, ref will be deduced from the webhook payload. Authorization token may need to be passed as a URL parameter ?token=aaa.