Skip to content

fix(tests): align marketplace e2e guard's credential filter with runtime#344

Open
johnhain-msft wants to merge 3 commits into
ianphil:masterfrom
johnhain-msft:fix/marketplace-credential-stores-7
Open

fix(tests): align marketplace e2e guard's credential filter with runtime#344
johnhain-msft wants to merge 3 commits into
ianphil:masterfrom
johnhain-msft:fix/marketplace-credential-stores-7

Conversation

@johnhain-msft

Copy link
Copy Markdown

Summary

tests/e2e/electron/electronApp.ts's canAccessRepo accepts stored Copilot credentials that the Chamber runtime filters out. The result: marketplace e2e specs can be allowed to run when the runtime cannot actually reach the marketplace repo, causing the specs to fail with a runtime "Unable to access marketplace …" error instead of skipping cleanly.

This PR aligns the test guard's credential set with the runtime's by reusing listStoredGitHubCredentials from @chamber/services. Test-side change only — no production code, no version bump, no CHANGELOG entry.

Closes #343

Root cause

canAccessRepo iterated every entry returned by keytar.findCredentials('copilot-cli'). The Chamber runtime reads from the same keytar service but filters via listStoredGitHubCredentials (packages/services/src/auth/AuthService.ts:36), which keeps only https://github.com:<login> accounts. GitHubRegistryClient and GitHubReleaseAssetClient never see EMU/enterprise-shape accounts (https://github.com/enterprises/microsoft:<login>).

So when a developer's keychain holds an EMU credential that can reach the private marketplace repo but no chamber-format credential that can, the guard probes the EMU token, returns true, and lets the spec run. The runtime then fails with:

Unable to access marketplace agency-microsoft/genesis-minds. Chamber tried the public GitHub API and any stored Chamber GitHub credentials.

Approach — same keytar service, same filter

Test-side only. The guard now uses the exact same filter as the runtime.

Files changed

tests/e2e/electron/chamberRepoAccess.ts                | 69 ++++++  (new)
tests/e2e/electron/electronApp.ts                      | 51 +-
tests/regression/electronAppChamberRepoAccess.test.ts  | 151 +++++  (new)

tests/e2e/electron/chamberRepoAccess.ts (new)

Exports canAccessRepoWithChamberCredentials(nwo, credentialStore, options?). Top-level imports listStoredGitHubCredentials and CredentialStore from @chamber/services. Anon fetch first, then iterates the filtered set. A credential-store rejection is swallowed (returns false) to mirror GitHubRegistryClient.safeCredentials() at packages/services/src/genesis/GitHubRegistryClient.ts:143.

tests/e2e/electron/electronApp.ts

canAccessRepo becomes a thin delegation to the helper. Two invariants preserved:

  • Lazy import('keytar') is kept. Per CHANGELOG Desktop smoke fails on Windows: keytar lock + PR #244 sidebar selector regressions + ProfileMarkdownEditor viewport #250, eagerly loading keytar.node in the Playwright runner process pins the addon and breaks electron-forge's rebuild on Windows (EPERM on unlink).
  • Static import of the helper (not await import('./helper')). Playwright's TypeScript loader handles static .ts imports of sibling files in the test graph, but routes await import('./helper') through Node's native loader, which then rejects the helper's top-level ESM import statements with Cannot use import statement outside a module. The static-import invariant is only safe while @chamber/services has zero transitive keytar imports — that invariant is verified and called out in the JSDoc on canAccessRepo as a tripwire for future regressions.

Inline canFetchRepo was removed (no other callers).

The single helper covers both runtime paths (GitHubRegistryClient.withCredentialStore and GitHubReleaseAssetClient.withCredentialStore) because both go through listStoredGitHubCredentials.

Tests

tests/regression/electronAppChamberRepoAccess.test.ts — 5 vitest cases, no Electron required:

Case Asserts
anon-public Returns true when anon fetch(/repos/{nwo}) succeeds.
chamber-credential-succeeds Returns true when a single https://github.com:<login> token reaches the repo.
only-EMU-can-reach-repo (bug repro) Returns false when only EMU-format tokens can reach the repo.
no-credential-can-reach-repo Returns false when neither anon nor any Chamber-format credential works.
credential-store-errors Returns false when findCredentials rejects (parity with runtime safeCredentials).

The fake satisfies the full CredentialStore interface; setPassword / deletePassword throw to assert they're never called by the helper.

Verification

Static checks against current master

  • npx tsc --noEmit — clean
  • npm run lint — clean (TS + ESLint + deps:check (446 modules, 0 violations) + the new lint:yaml and lint:md)
  • npm test — 164 files, 1801 tests pass (1796 baseline + 5 new)

Empirical A/B repro

On a workstation with the exact bug-repro credential state — chamber-format credential present but lacking access (HTTP 404 against /repos/agency-microsoft/genesis-minds), EMU credential present and granting access (HTTP 200):

Code Spec behavior Outcome
master (pre-fix) Spec runs ~1.7 min — canAccessRepo returns true via EMU probe → Electron launches → runtime tries to add marketplace FAIL with the bug: Unable to access marketplace agency-microsoft/genesis-minds. Chamber tried the public GitHub API and any stored Chamber GitHub credentials.
This branch (post-fix) Spec skips in <1s — canAccessRepo returns false because EMU is filtered out and chamber-format can't reach the repo Clean skip: Stored Chamber GitHub credentials cannot access agency-microsoft/genesis-minds.

Same machine, same credential state, only the test-side guard changed. The runtime error in the pre-fix row is the literal error string from issue #343.

Notes for the reviewer

  • Why not just filter for the active login in the guard? That would tie the test guard to Chamber's active-account state, which is a runtime concept. The runtime's marketplace path is not active-login-scoped — it iterates the entire filtered credential set via safeCredentials(). Mirroring that exact filter keeps the guard's behavior identical to the runtime's by construction.
  • Endpoint parity. The runtime uses /git/trees for actual marketplace fetches; the guard uses /repos/{nwo}. Closing that gap is a separate concern and was intentionally left out.
  • AbortController on the guard's fetch calls was considered and left out for the same reason — separate concern, not in scope for this fix.

Out of scope

Anything other than aligning the two credential paths.

johnhain-msft and others added 3 commits May 18, 2026 20:40
The Playwright Electron test guard `canAccessRepo` previously iterated every
`keytar.findCredentials('copilot-cli')` entry — including EMU/enterprise-shape
accounts (`https://github.com/enterprises/microsoft:<login>`) that Chamber's
runtime filters out. As a result, the guard could return true on the strength
of an EMU token that the in-app marketplace fetcher would never use, opening
the test gate so specs ran but then failed in the runtime with `Unable to
access marketplace <nwo>`.

Reuse `listStoredGitHubCredentials` from `@chamber/services` to filter the
guard's credential candidates to the same Chamber-format
`https://github.com:<login>` set the runtime considers. The guard and the
runtime now consult the same filtered credential set by construction.

- New `tests/e2e/electron/chamberRepoAccess.ts` exports
  `canAccessRepoWithChamberCredentials(nwo, credentialStore, options?)`. Pure,
  unit-testable, no Electron dependency.
- `tests/e2e/electron/electronApp.ts:canAccessRepo` is now a thin
  keytar+helper delegation. The lazy `import('keytar')` is preserved
  (CHANGELOG ianphil#250 — Windows EPERM on `keytar.node`). The helper is imported
  statically because Playwright's TS loader handles static `.ts` imports but
  routes dynamic `import('./helper')` through Node's native loader, which
  rejects the helper's top-level ESM `import` statements. Static import is
  safe only as long as `@chamber/services` has no `keytar` transitive
  imports (verified at commit time); the JSDoc on `canAccessRepo` documents
  this invariant.
- Inline `canFetchRepo` is removed (no other callers).
- New `tests/regression/electronAppChamberRepoAccess.test.ts` covers
  anon-public, chamber-credential-succeeds, only-EMU-can-reach-repo (bug
  repro), no-credential-can-reach-repo, and credential-store-errors (parity
  with runtime's `safeCredentials` swallow).

Verified by re-running the four affected Electron Playwright specs
(`genesis-landing-add-marketplace`, `genesis-marketplace-aggregation`,
`settings-marketplace-management`, `a365-release-tools-smoke`): all skip
cleanly with the documented skip reasons. Before this fix, the first three
would have run and then failed in runtime.

No production credential-store change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Marketplace e2e test guard's credential filter diverges from runtime

1 participant