ci: reduce Acceptance Tests runtime from ~1h40m (runner queue saturation + per-job rebuilds)

[DeepDiver1975]

Problem

The Acceptance Tests workflow currently takes ~1h40m. Analysis of run 27106300746 shows the dominant cost is not test execution time — it's runner queue saturation and redundant per-job work.

Timeline (measured)

Phase	Start offset	Duration	Note
Workflow queued → detect-changes starts	+0m	9m37s	Runner queue wait
detect-changes runs	+9m37s	58s
detect-changes done → build-and-test starts	+10m35s	14m23s	Runner queue wait
build-and-test runs	+24m58s	8m57s	Builds oCIS, lint, unit tests
Gate open → first test job starts	+33m55s	6m11s	Runner queue
Gate open → last test job starts	+33m55s	56m46s	55 jobs competing for runners
Last test job finishes	+101m17s	626s actual test

Total ~101 minutes. Runner queue wait alone accounts for 65+ minutes.

---

Issue 1: Every test job rebuilds oCIS from source

run-github.py line 381 calls make -C ocis build inside every one of the 55 test matrix jobs. The build-and-test job already builds the binary — then discards it. Each downstream job re-runs:

• actions/setup-go (~21s)
• apt-get install libssl-dev libnghttp2-dev … libvips-dev (~36s, even on cache hit)
• make -C ocis build + make -C tests/ociswrapper build
• make ci-node-generate (IDP/web assets)
• composer install (PHP deps)

Fix: Upload the built binaries and generated assets as an artifact from build-and-test, download them in each test job. This removes the need for setup-go, setup-node/pnpm, libcurl/libvips, and the compile step from all 55 jobs.

# In build-and-test, after building:
- uses: actions/upload-artifact@v4
  with:
    name: ocis-binaries
    path: |
      ocis/bin/ocis
      tests/ociswrapper/bin/ociswrapper
      services/web/assets/
      services/idp/assets/
    retention-days: 1

# In each test job, replace the build steps with:
- uses: actions/download-artifact@v4
  with:
    name: ocis-binaries
- run: chmod +x ocis/bin/ocis tests/ociswrapper/bin/ociswrapper

PHP composer deps could also be uploaded/restored, or kept since composer is fast.

---

Issue 2: 55 jobs launched simultaneously — pool exhaustion

After build-and-test finishes, all 55 test jobs are queued at once. The runner pool can't supply them simultaneously, so jobs are delayed by up to 56 minutes waiting for a runner slot. This is the single biggest contributor to total runtime.

Fix A: Tier execution (fail-fast cheap → expensive)

Add a tier-1-smoke job group with the fast protocol/smoke tests (cs3api=55s, litmus=102s, wopi-builtin=86s, wopi-cs3=104s, cliCommands=668s). Use fail-fast: true so a failure in tier-1 cancels its siblings immediately. Make tier-2-full depend on tier-1-smoke — if tier-1 fails, the 49-job expensive wave never launches.

tier-1-smoke:
  needs: [build-and-test]
  strategy:
    fail-fast: true
    matrix:
      suite: [cs3api, litmus, wopi-builtin, wopi-cs3, "cliCommands,apiServiceAvailability"]

tier-2-full:
  needs: [build-and-test, tier-1-smoke]
  strategy:
    fail-fast: false
    matrix:
      suite: [... all remaining suites ...]

On a broken PR, this stops the run after ~12 minutes instead of 100 minutes.

Fix B: Align matrix groups with the existing Python logical groups

run-github.py already defines logical test groups that share services (email, tika, antivirus, etc.). The matrix currently uses 32 individual suite entries for local-api-tests, but the runner script already handles comma-separated suite lists. Aligning the matrix with the existing groups reduces concurrent runner demand by ~60%:

Current (32 single-suite jobs)	Proposed (grouped)	Combined duration
apiGraph + apiDownloads + apiGraphGroup	1 job: apiGraph,apiDownloads,apiGraphGroup (matches groupAndSearch1)	~617s
apiSpacesDavOperation + apiDownloads + apiAsyncUpload + apiDepthInfinity + apiArchiver + apiActivities	2 jobs (matches davOperations, split for time)	~600s each
apiSharingNgShares + apiReshare + apiSharingNgPermissions	1-2 jobs (matches sharingNg1)	~1345s → 2 jobs
apiSharingNgDriveLinkShare + apiSharingNgItemLinkShare + apiSharingNgLinkShareManagement	1 job (matches sharingNgLinkShare)	~978s

This reduces from 32 to ~14-15 local-api runner slots while keeping per-job runtime under ~700s.

---

Issue 3: detect-changes queue wait (~10 minutes for a 58-second job)

detect-changes waits ~9m37s for a runner to run a 58-second script. One option is to use a paths-filter action directly in the workflow trigger condition, or use concurrency groups more aggressively to avoid needing a separate job for this.

---

Measured job durations (for grouping reference)

Top 10 slowest test jobs (total wall time including setup):

Job	Total	Actual test step
apiSharingNgShares	834s	752s
apiGraphUser	796s	736s
apiSpacesShares	789s	730s
coreApiWebdavProperties	776s	664s
coreApiSharePublicLink2	762s	677s
apiSpaces	683s	603s
coreApiShareOps (grouped)	674s	568s
cliCommands,apiServiceAvailability	668s	629s
apiSharingNgItemInvitation	648s	591s
apiSearchContent	626s	540s

Setup overhead (checkout + setup-go + setup-node + pnpm + setup-php + libcurl) per job: ~55-90s — this overhead disappears if binaries are pre-built.

---

Projected impact

Fix	Complexity	Success path savings	Failure path savings
Upload/download binary artifact	Medium	~2min per job setup eliminated; removes need for Go/Node toolchain in test jobs	Same
Align matrix with existing Python groups (32→14 jobs)	Low	Reduces peak runner demand ~40%; last job starts ~25m earlier	Big
Tier 1/2 split (fail-fast)	Low	Minimal on success	Stops 49-job wave on broken PRs
Fix detect-changes queue	Low	~10m off the front	Same

Combined realistic estimate: 35-45 minutes off the 1h40m runtime on the success path, mostly from runner queue reduction. Failure path could be ~30min total instead of ~100min.

[mklos-kw]

Good ideas!

Ad. Issue 2: Fix B

I'd hold off on optimizing the run-github.py scripts for now. There's an MVP in progress to migrate to GHA native steps, which have built-in feature toggles:

• feat: GHA MVP mklos-kw/ocis#2
• GHA reusable actions mklos-kw/ocis#3
• https://github.com/mklos-kw/ocis-github-actions

That work could also incorporate ideas from Fix B, but it would be better to collect CI run data first to validate the expected improvements before optimizing further. I already tried Fix B some time ago, but in PoC is was not so easy - I'll bring back the diff and rerun again to see what failed then.

Edit: I found old PoC and run again: https://github.com/owncloud/ocis/actions/runs/27137797674/job/80094582806?pr=12392

Fail root cause: Two behat processes hit the same OCIS instance simultaneously and corrupt shared state (same test users, files, locks).

Each parallel lane needs its own OCIS instance. The script currently starts one instance and one copy of each 3rd-party service. To run N suites in parallel it needs:

1. Port allocation — a unique port offset per lane (OCIS HTTP, OCIS gRPC, ociswrapper, Tika, ClamAV, each WOPI service)
2. N OCIS instances — separate config dir + data dir per lane, each launched with its lane's port set
3. N service sets — each 3rd-party container started with --name tika-1 -p 9998:9998 style distinct host ports per lane
4. Per-lane TEST_SERVER_URL / OCIS_WRAPPER_URL — behat for lane N must point at lane N's OCIS

[DeepDiver1975]

only issue 1 was taken care of

[mklos-kw]

Ad. Issue 2: Fix A

From last 200 runs, 2026-05-20 → 2026-06-09. After stripping noise 19 test-failure runs:

Job	Failures
apiSearchContent	5
e2e-search	3
e2e-part-2, e2e-part-3	2 each
apiSettings, apiGraphUser	2 each
everything else	1 each
litmus, cs3api, wopi-builtin, wopi-cs3	0

The protocol validators never fail - they won't catch regressions early. Tier-1 split is worth doing for fail-fast, but the canary/Tier 1 choice is arbitrary - with a bias toward e2e and apiSearchContent.

[dj4oC]

Before this can be acted on, the following needs to be resolved:

• Issue 1 (pre-built binary artifact) is already implemented per the maintainer comment — confirm the remaining scope is only Issue 2 and Issue 3.
• Decision needed on Issue 2 Fix B: optimize run-github.py matrix grouping now, or defer to the in-progress GHA-native migration MVP (a maintainer asked to hold off on run-github.py changes).
• Issue 2 Fix B has an unresolved blocker documented in-thread: parallel suites corrupt shared OCIS state; enabling it requires per-lane port allocation, a separate OCIS instance per lane, and per-lane 3rd-party services (Tika/ClamAV/WOPI). Confirm whether that infra is in scope here.
• Issue 2 Fix A tier-1 smoke selection is undecided: the failure-rate data shows the protocol validators (litmus/cs3api/wopi-*) never fail, so define which suites actually constitute tier-1.
• Confirm which sub-parts remain actionable in owncloud/ocis vs. the separate GHA-native migration effort (mklos-kw/ocis-github-actions).

🤖 Automated by Claude Code

---

Generated by Claude Code