GitLab Integration: Technical Exploration Summary

[luhenry]

This document summarizes the technical exploration of integrating RISE RISC-V
Runners with GitLab, covering authorization models, job detection strategies,
runner execution models, and security considerations. Each approach is evaluated
in context of the existing riscv-runner-app architecture: a webhook frontend
writes job state to PostgreSQL, a scheduler performs demand matching and
provisions ephemeral runner pods on bare-metal RISC-V Kubernetes nodes.

---

1. Authorization and Onboarding

GitLab has no equivalent of a GitHub App, there is no mechanism where RISE
registers one application and external organizations "install" it with scoped
permissions. Two authorization paths were explored.

1.1 OAuth 2.0 Authorization Code Flow (GitLab.com)

RISE registers a user-owned OAuth 2.0 application on GitLab.com. External
group owners authorize it via a standard OAuth consent flow. RISE receives
an access token and refresh token, which it uses to create runners and
(optionally) webhooks via the GitLab API.

How it works:

• RISE registers an OAuth app at Edit Profile → Applications on GitLab.com.
• The app is configured with a redirect URI pointing to RISE's onboarding portal.
• When an admin clicks "Connect GitLab," the portal redirects to GitLab's
  /oauth/authorize endpoint with the requested scopes.
• The admin authorizes; GitLab redirects back with an authorization code.
• RISE exchanges the code for an access token (2-hour lifetime) and refresh
  token (single-use, returns a new refresh token on each use).
• RISE uses the access token to call POST /api/v4/user/runners to create a
  group runner, receiving a glrt- runner authentication token.

Scopes required:

Scope	Purpose	When needed
create_runner	Create runners via API	Always

Pros:

• Self-service onboarding portal experience, admin clicks a button, authorizes,
  done.
• Single app registration serves all GitLab.com users.
• Scoped to the authorizing user's permissions, can't exceed what the admin
  can do.

Cons:

• The token acts on behalf of the individual user, not a service account. If
  that user loses Owner role or leaves the org, the integration breaks.
• Access tokens expire every 2 hours; refresh tokens are single-use. Requires
  robust token refresh logic in the scheduler.
• No GitLab equivalent of Azure DevOps's multi-tenant Entra ID service principal
  model, the token doesn't represent an "app identity" scoped to a group.

Mitigation for user-binding risk: Recommend admins authorize using a
dedicated service account rather than their personal account.

1.2 Personal Access Token / Group Access Token (Self-Hosted and GitLab.com)

The admin creates a PAT or Group Access Token with create_runner scope and
provides it to RISE's portal.

How it works:

• Admin goes to Edit Profile → Access Tokens (PAT) or Group → Settings →
  Access Tokens (Group Access Token).
• Creates a token with create_runner scope, Owner role (for group runners)
  or Maintainer role (for project runners).
• Pastes the token into RISE's onboarding portal along with the group ID and
  instance URL.
• RISE uses the token to call POST /api/v4/user/runners.

Pros:

• Works for both GitLab.com and self-hosted instances.
• Simple, no OAuth flow, no redirect URI, no token refresh.
• Group Access Tokens create a bot user that survives individual user churn.
• This is the approach AWS CodeBuild uses for self-hosted GitLab — proven model.
• The PAT can be revoked immediately after onboarding if RISE only needs the
  glrt- runner token going forward.

Cons:

• Not self-service in the same way as OAuth, the admin must manually create
  the token and paste it.
• PATs have a maximum lifetime of 365 days (400 with feature flag). Group
  Access Tokens have the same constraint.
• On GitLab.com, Group Access Tokens require Premium or Ultimate subscription.
  PATs work on all tiers but are tied to a personal account.
• If RISE needs ongoing API access (e.g., for polling or webhook management),
  the token must remain valid and stored.

For self-hosted instances: OAuth is impractical because each self-hosted
instance would need its own OAuth app registration (admin must go to Admin →
Applications → New application, enter RISE's redirect URI, and share the
resulting Application ID and Client Secret). PAT is the recommended approach.

1.3 The glrt- Runner Authentication Token

Both authorization paths produce a glrt- prefixed runner authentication token.
This token is what runner pods use to authenticate with GitLab.

Lifetime: Does not expire by default. Expiration is only enforced if a
GitLab instance administrator has configured a "Runners expiration time" in
Admin → Settings → CI/CD. On GitLab.com, this is currently not enforced (the
token_expires_at field in the API response is null).

Renewal: Can be reset via the Runners API (POST /runners/:id/reset_authentication_token
using a PAT, or POST /runners/reset_authentication_token using the current
token value). When automatic rotation is configured on the GitLab instance, the
gitlab-runner binary handles renewal transparently, but this only works for
long-lived runner processes, not ephemeral pods.

Reusability: A single glrt- token can be used by multiple runner
processes simultaneously. Each process gets a unique system_id automatically.
This is fundamentally different from GitHub's JIT config, which is single-use.

---

2. Job Detection

Three strategies were explored for detecting when a GitLab CI job needs a
RISC-V runner.

2.1 Webhooks (Event-Driven)

GitLab sends HTTP POST notifications to RISE's endpoint when job status changes.

How it works:

• A webhook is configured at the group or project level, subscribed to
  Job events, pointing at RISE's glfe endpoint.
• The payload includes build_status (created, pending, running,
  success, failed, canceled), build_id, project_id, and tag
  information.
• The webhook is verified using a shared secret sent in the X-Gitlab-Token
  header.
• glfe writes pending jobs to PostgreSQL, waking the scheduler via
  LISTEN/NOTIFY, identical to the existing ghfe pattern for GitHub.

Requirements:

• Creating webhooks programmatically requires api scope on the OAuth token
  or PAT (no narrower scope covers webhook management). The admin must then
  do it manually.
• Group webhooks require Premium or Ultimate on GitLab.com. Project
  webhooks are available on all tiers but require per-project setup.
• RISE must expose a public HTTPS endpoint for receiving webhooks.

Pros:

• Instant job detection, sub-second latency from job queued to RISE awareness.
• Maps directly to the existing riscv-runner-app architecture (webhook →
  frontend → DB → scheduler → pod).
• No polling overhead, no API rate limit concerns.
• AWS CodeBuild uses this same model for their GitLab runner integration.

Cons:

• Group webhooks are gated behind Premium tier, Free tier orgs need
  per-project webhooks, increasing onboarding friction.
• Known GitLab issue: the created → pending transition doesn't always fire
  a webhook. May receive created then running with no pending in between.
• Known GitLab issue (#505691): the tag_list field in the webhook payload
  may show runner tags instead of job tags at pending status. This affects
  RISE's ability to determine if a pending job is a RISE job from the webhook
  payload alone.
• GitLab auto-disables webhooks after 4 consecutive failures, the endpoint
  must be highly available.
• No webhook retry mechanism, GitLab does not retry failed webhook deliveries
  (unlike GitHub or Stripe).

2.2 API Polling

The scheduler periodically queries the GitLab Jobs API to find pending jobs.

How it works:

• The scheduler calls GET /api/v4/projects/:id/jobs?scope[]=pending&scope[]=created
  for each monitored project.
• If any returned jobs have RISE tags (e.g., saas-linux-medium-riscv64),
  the scheduler writes them to PostgreSQL and proceeds with demand matching.
• Requires an OAuth token with read_api scope, or a PAT with read_api.

Pros:

• No Premium tier requirement.
• No webhook infrastructure to maintain.
• Works identically for GitLab.com and self-hosted.

Cons:

• No group-level pending jobs endpoint, must poll per project. If an entity
  has 50 projects, that's 50 API calls per cycle.
• Latency: poll interval (15–30s) + pod startup time (20–40s) = 35–70 seconds
  from job queued to execution start.
• API rate limit pressure, scales linearly with the number of monitored
  projects across all entities.
• Requires the admin to declare which projects use RISE runners at onboarding
  time (or RISE must enumerate all projects in the group).

2.3 Long-Polling via POST /api/v4/jobs/request (Native Runner Protocol)

The gitlab-runner binary natively polls GitLab for jobs using a long-poll
mechanism built into the GitLab Workhorse server.

How it works:

• The runner sends POST /api/v4/jobs/request with its glrt- token.
• If no jobs are available, GitLab Workhorse holds the HTTP connection open
  (default: up to 50 seconds), subscribing to a Redis PubSub channel.
• When a new job is queued for a matching runner, Redis notifies Workhorse,
  which releases the connection and forwards to Rails.
• Rails assigns the job to the runner and returns a 201 with the full job
  payload (including a short-lived job_token).
• If no job arrives within the long-poll duration, a 204 is returned and the
  runner immediately re-polls.

Critical constraint: This endpoint atomically claims the job. The
runner that receives the 201 response IS the runner that must execute the job.
There is no way to "peek" at the queue without claiming. This makes it
impossible to use this endpoint purely for detection in a scheduler that
provisions a separate worker. If the scheduler polls that API, then it must
keep sending heartbeats while a pod is launched to execute the job. And there
is no support in gitlab-runner to execute a job that it didn't claim itself.

Pros:

• Sub-second latency, effectively instant notification when a job is queued.
• No webhook infrastructure needed.
• No API rate limits, this is the protocol GitLab is designed to handle at
  scale (their hosted runners use it).
• Works on all GitLab tiers.

Cons:

• Detection and execution are coupled, cannot separate "know a job exists"
  from "commit to running it."
• Requires a running gitlab-runner process to be always-on for each token.
• In a multi-token setup (many entities), a single process holds all tokens,
  creating a security risk if the process is compromised.

---

3. Runner Execution Models

Four execution models were explored for running the actual CI job on RISE's
RISC-V infrastructure.

3.1 Docker Executor with run-single

A pod runs gitlab-runner run-single --executor docker --max-builds 1, which
claims one job, executes it in a docker container, streams logs back to
GitLab, and exits.

How it works:

• The pod starts, run-single polls GitLab via POST /api/v4/jobs/request.
• On receiving a job (201 response), it spawns a container, runs the CI
  script stages (before_script, script, after_script) in that container.
• Captures stdout/stderr from the container, buffers it, and sends it to
  GitLab via periodic trace updates (using the job_token).
• Reports final status and exits. Pod terminates.

Pros:

• Self-contained, no manager pod, no sidecar, no external proxy. The pod
  handles everything: claiming, executing, log streaming, status reporting.
• Simple to deploy, one binary, one command, no config file needed (all
  params via CLI flags or env vars).
• Full hardware access, the CI script runs directly in the kubernetes pod.

Cons:

• Detection and execution are coupled (same process claims and runs).
• Each pod must contain the glrt- runner token (security concern for
  multi-tenant environments, though mitigated by using one token per entity
  per pod).
• If used with webhooks/polling for detection, there's a time gap between
  detection and pod startup during which no heartbeat is sent to GitLab for
  the claimed job, because the job hasn't been claimed yet at detection time.

3.2 Kubernetes Executor (Manager-Worker Split)

A persistent manager pod claims jobs and creates ephemeral worker pods on
the RISC-V nodes. The manager handles all GitLab communication; the worker
pod only executes the CI script.

How it works:

• The manager pod runs gitlab-runner run with a config.toml containing
  all entity tokens, using executor = "kubernetes".
• It polls GitLab via long-poll across all configured tokens.
• When a job is claimed, the manager calls the K8s API to create a worker pod.
• The worker pod contains a build container (runs the CI script) and a helper
  container (handles git clone, artifact upload using the job_token).
• The manager uses kube attach to stream logs from the worker pod's helper
  container, and forwards them to GitLab as trace updates.
• The manager stays connected for the entire job duration.

Pros:

• Security isolation: The glrt- tokens live only on the manager pod
  (on the amd64 control plane). Worker pods on riscv64 nodes receive only the
  short-lived job_token. A malicious CI script cannot access other entities'
  runner tokens.
• Built-in demand detection: The manager handles long-polling, no
  webhooks or API polling needed. Job pickup latency is sub-second.
• Heartbeat during provisioning: The manager sends heartbeats to GitLab
  while the worker pod is being scheduled, preventing job timeout.
• GitLab's official, supported, and actively developed executor for K8s.

Cons:

• Bypasses the riscv-runner-app scheduler. The manager creates worker
  pods independently, there's no unified demand matching across GitHub,
  GitLab, and Azure DevOps.
• The manager is a single point of failure, if it dies, all in-flight GitLab
  jobs lose their log proxy.
• The manager must remain connected via kube attach for the entire duration
  of every job, it's an active proxy, not a fire-and-forget launcher.
• Capacity sharing with GitHub jobs must be managed via K8s resource quotas
  or namespace-level limits rather than through the unified scheduler.
• Requires gitlab-runner-helper container image for riscv64 (used for git
  clone and artifact handling in the worker pod).

3.3 Warm Poller Pool (Explored and Rejected)

Pre-provision N pods per entity, each running run-single in long-poll mode.
When a pod finishes a job and exits, the scheduler replaces it immediately.

How it works:

• A K8s Deployment maintains N replicas per entity.
• Each pod runs gitlab-runner run-single and long-polls until a job arrives.
• Job pickup latency is sub-second (the pod is already running).
• On job completion, the pod exits and K8s replaces it.

Why it was rejected: RISE has a limited number of bare-metal machines
shared across all platforms (GitHub, GitLab, Azure DevOps) and many
organizations. Pre-provisioning pods per entity would exhaust capacity on idle
pollers, starving other platforms. The machines are a shared pool, they cannot
be dedicated per-entity.

3.4 Custom Split Architecture (Scheduler Claims, Worker Executes)

The scheduler (on amd64) claims jobs via POST /api/v4/jobs/request, sends
heartbeats while provisioning, then hands off the job payload to a worker pod
(on riscv64) which executes and streams logs.

This heavily mirrors how the Kubernetes executor works. The main difference would
be in how the logs are streamed back to Gitlab, going straight from the runners
to Gitlab, bypassing the manager in the matter. The scheduler (equivalent to the
manager) would only poll for the job, send heartbeats while the pod has not been
launched, and offload the whole heartbeat, setup, and pushing the log to the
worker.

How it works:

• The scheduler runs a multi-token polling loop using the glrt- tokens
  for all entities.
• When a job is claimed (201 response), the scheduler receives the full job
  payload including the job_token.
• The scheduler sends heartbeat/trace updates to GitLab while provisioning a
  worker pod (using the internal trace update protocol).
• The serialized job payload (containing only the job_token, not the
  glrt- runner token) is passed to the worker pod.
• The worker pod deserializes the payload, executes the CI script, streams
  logs, and reports completion.

Pros:

• Maps to the existing riscv-runner-app architecture, the scheduler
  remains the single control point for all platforms.
• Security isolation, worker pods receive only the job_token, not the
  runner token.
• Unified demand matching and capacity management across GitHub, GitLab, and
  Azure DevOps.
• No webhooks required, the scheduler uses long-poll for detection.

Cons:

• Requires implementing the GitLab runner-to-server protocol (job request,
  trace updates, status reporting). This protocol is internal and
  undocumented, it's not part of GitLab's public REST API.
• The trace update endpoint (used for log streaming and heartbeats) is only
  documented in the gitlab-runner Go source code (network/gitlab.go).
• An unofficial Python library (gitlab_runner_api) implements this protocol
  but hasn't been updated in 6 years.
• AWS CodeBuild's implementation of this protocol is proprietary and closed-source.
• Risk of protocol changes between GitLab versions (though in practice, the
  protocol is stable because all runners depend on it).
• Requires either forking gitlab-runner to add a --job-response flag for
  feeding pre-claimed job data, or reimplementing the executor in Python.

Key source files for understanding the protocol:

• network/gitlab.go, HTTP client implementing RequestJob, UpdateJob,
  PatchTrace (the trace update/heartbeat method)
• commands/single.go, the run-single main loop
• common/build.go, Build.Run() orchestration
• common/network.go, the Network interface definition

---

4. Runner Tags

GitLab uses tags for job-to-runner matching. Two types exist:

• Runner tags: Set when the runner is created (via API or UI). Declare what
  the runner can handle. RISE would use saas-linux-medium-riscv64.
• Job tags: Set in .gitlab-ci.yml by the user. Declare what the job needs.

Matching rule: a job can only run on a runner that has all the tags the job
specifies (AND logic, the runner must be a superset of the job's tags).

Setting run_untagged=false on the runner ensures it only picks up jobs that
explicitly request RISE tags, preventing it from consuming unrelated jobs.

---

5. Config Security

The config.toml file contains glrt- tokens for all entities. With the shell
executor, the CI job runs in the same process/filesystem context as the runner,
a malicious .gitlab-ci.yml could read the config file and steal every org's
runner token.

Mitigations explored:

Approach	How	Limitation
K8s Secret on tmpfs	Mount config as a K8s Secret volume (memory-backed)	File still readable by the CI script
Per-entity run-single	Each pod has only ONE token for ONE entity via env var	No multi-token config file exists; requires external demand detection
Kubernetes executor	Manager holds tokens; worker gets only job_token	Bypasses the unified scheduler
Custom split	Scheduler holds tokens; worker gets serialized job payload	Requires implementing the internal protocol

The fundamental constraint: with the shell executor, the CI script has full
access to the runner's environment. The only way to protect tokens from
malicious jobs is to ensure the tokens never exist in the worker pod's
environment, which requires a manager/worker split of some kind.

---

6. Comparison Matrix

Dimension	Webhooks + run-single	API Polling + run-single	K8s Executor	Custom Split
Job detection latency	< 1s	15–30s	< 1s	< 1s
GitLab tier requirement	Premium (group webhooks)	None	None	None
Requires webhook endpoint	Yes	No	No	No
Token security	One token per pod (safe)	One token per pod (safe)	Tokens on manager only (safest)	Tokens on scheduler only (safest)
Unified scheduler	Yes	Yes	No, bypasses scheduler	Yes
Protocol complexity	Standard webhook + public API	Public API only	Built-in (gitlab-runner)	Internal undocumented protocol
Pod startup in critical path	Yes (adds 20–40s latency)	Yes	Yes, but manager sends heartbeats	Manager sends heartbeats during startup
Self-hosted support	Yes	Yes	Yes	Yes
Maintenance burden	Low, uses stock gitlab-runner	Low, uses stock gitlab-runner	Low, uses stock gitlab-runner	High, must track internal protocol changes