📖 Scorecard v6: OSPS Baseline conformance proposal and 2026 roadmap

[justaugustus]

What kind of change does this PR introduce?

Documentation: Scorecard v6 proposal and 2026 roadmap.

Scorecard v6 evolves Scorecard from a scoring tool to an open source security
evidence engine. The primary initiative for 2026 is adding
OSPS Baseline conformance evaluation as the
first use case that proves this architecture.

Mission: Scorecard produces trusted, structured security evidence for the
open source ecosystem.

Scorecard accepts diverse inputs about a project's security practices,
normalizes them through probe-based analysis, and packages the resulting
evidence in interoperable formats for downstream tools to act on. Check scores
(0-10) and conformance labels (PASS/FAIL/UNKNOWN) are parallel evaluation
layers over the same probe evidence, produced in a single run. v6 is additive —
existing checks, probes, scores, and output formats are preserved.

The goal of this PR is to create a collaboration/decision-making nexus for
Scorecard and WG ORBIT tooling maintainers to ensure that we build interfaces
that easily interact with other tools and minimize duplication of work across
our maintainers and others in the OpenSSF ecosystem.

Key changes that warrant a major version:

1. New evaluation layer — conformance labels alongside existing check scores
2. Framework-agnostic architecture — probe evidence can be composed against
   OSPS Baseline or other frameworks via pluggable mapping definitions
3. Interoperable output formats — in-toto, Gemara, OSCAL Assessment Results
   alongside existing JSON and SARIF
4. Probe catalog as public interface — probe definitions become a consumable
   artifact for external tools

Key decisions resolved:

• Architecture (CQ-19): Option C (hybrid) — Scorecard owns probe execution
  and conformance evaluation; interoperability at the output layer only
• Output formats (CQ-18): Enriched JSON, in-toto (SVR + Baseline
  Predicate), Gemara (via security-baseline), OSCAL Assessment Results
• Mapping model (CQ-17/CQ-23): Two layers — check-level relations upstream
  in security-baseline, probe-level mappings in Scorecard
• Ecosystem positioning (CQ-13): All downstream consumers (Privateer,
  AMPEL, Minder, Darnit) are equal

Open questions requiring Steering Committee resolution:

• OQ-1/CQ-22: Attestation identity model (blocking)

• OQ-2: Enforcement detection scope

• PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

Scorecard produces 0-10 check scores and structured probe findings. There is no
OSPS Baseline conformance evaluation capability and no public 2026 roadmap.

What is the new behavior (if this is a feature change)?**

This PR adds documentation only (no code changes):

• docs/ROADMAP.md — Public 2026 roadmap with phased delivery plan

• openspec/changes/osps-baseline-conformance/proposal.md — Detailed proposal
  covering architecture, scope, phased delivery, and ecosystem positioning

• openspec/changes/osps-baseline-conformance/decisions.md — Reviewer feedback,
  open questions, maintainer responses, and decision priority analysis

• Tests for the changes have been added (for bug fixes/features)

N/A — documentation only.

Which issue(s) this PR fixes

NONE

Special notes for your reviewer

This PR is structured as two companion documents:

• proposal.md contains the proposal itself (motivation, architecture,
  scope, phased delivery, ecosystem positioning, success criteria)
• decisions.md tracks all reviewer feedback, open questions, maintainer
  responses, and the decision priority analysis

The control-by-control coverage analysis
is maintained separately.

Feedback from Eddie Knight (ORBIT WG TSC Chair), Adolfo García Veytia (AMPEL),
Mike Lieberman, and Felix Lange has been incorporated. See decisions.md for the
full feedback log and how it informed the proposal.

Does this PR introduce a user-facing change?

NONE

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.67%. Comparing base (353ed60) to head (45f4da6).
⚠️ Report is 321 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4952      +/-   ##
==========================================
+ Coverage   66.80%   69.67%   +2.87%     
==========================================
  Files         230      251      +21     
  Lines       16602    15654     -948     
==========================================
- Hits        11091    10907     -184     
+ Misses       4808     3873     -935     
- Partials      703      874     +171     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justaugustus]

@ossf/scorecard-maintainers @ossf/scorecard-fe-maintainers @eddie-knight @puerco @evankanderson @mlieberman85 — based on conversations from this week with various WG ORBIT-adjacent maintainers, I'm tossing this early draft up for review.

Feel free to comment away while I work through this!

[eddie-knight]

Hey @justaugustus, thanks for leading this collaboration! Looking forward to hammering this out.

Some things to clarify:

1. Regarding mappings between Baseline catalog<->Scorecard checks, it is possible to easily put that into a new file with Scorecard maintainers as codeowners, pending approval on that proposal from baseline maintainers.
2. There is not an "OSPS output format," and even the relevant Gemara schemas (which are quite opinionated) are still designed to support output in multiple output formats within the SDK, such as SARIF. I would expect that you'd keep your current output logic, and then maybe add basic Gemara json/yaml as another option.
3. There is a stated goal of not duplicating the code from the plugin ossf/pvtr-github-repo-scanner, but the implementation plan as it's currently written does require duplication. In the current proposal, there would not be a technical relationship between the two codebases.
4. There is cursory mention of a scorecard catalog extraction, which I'm hugely in favor of, but I don't see an implementation plan for that.

An alternative plan would be to for us to spend a week consolidating checks/probes into the pvtr plugin (with relevant CODEOWNERS), then update Scorecard to selectively execute the plugin under the covers.

This would allow us to:

• Extract the Scorecard control catalog for independent versioning and still easily connect it to the code
• Instantly integrate Gemara to scorecard
• Allow Scorecard to optionally run the existing Baseline checks from Scorecard
• Allow LFX Insights and other pvtr users to optionally run Scorecard checks from Privateer
• Simplify contribution overhead for each individual scorecard check
• And also improve the quality of both codebases through shared logic

[justaugustus]

Reviewed this in today's Steering meeting and I think this is moving in the right direction, based on the initial feedback.

@spencerschrock and @jeffmendoza intend to leave reviews.

---

@GeauxJD @SecurityCRob — for awareness.

[puerco]

Amazing, two nits and two comments but it LGTM.

[puerco]

This point is key. Baseline looks for outcomes. Compliance can be supported by Scorecard probe data.

The baseline control can be a 1:1 map to a probe's data, other times it will be a composite set of probes. If you add new probes to look for something new that's useful to test a baseline control, we just need to add another composition definition to say OSPS-XX-XXX can be [probe X] or [probe set 1] or [probe set 2].

This is akin to the way checks work now, but by generalizing it, the probe data can inform other framework testing tools, beyond baseline.

[justaugustus]

This is akin to the way checks work now, but by generalizing it, the probe data can inform other framework testing tools, beyond baseline.

Agreed. The current composition of probes comprise the "Scorecard checks":

scorecard/probes/entries.go

Lines 79 to 165 in 4dbf142

	// SecurityPolicy is all the probes for the
	// SecurityPolicy check.
	SecurityPolicy = []ProbeImpl{
	securityPolicyPresent.Run,
	securityPolicyContainsLinks.Run,
	securityPolicyContainsVulnerabilityDisclosure.Run,
	securityPolicyContainsText.Run,
	}
	// DependencyToolUpdates is all the probes for the
	// DependencyUpdateTool check.
	DependencyToolUpdates = []ProbeImpl{
	dependencyUpdateToolConfigured.Run,
	}
	Fuzzing = []ProbeImpl{
	fuzzed.Run,
	}
	Packaging = []ProbeImpl{
	packagedWithAutomatedWorkflow.Run,
	}
	License = []ProbeImpl{
	hasLicenseFile.Run,
	hasFSFOrOSIApprovedLicense.Run,
	}
	Contributors = []ProbeImpl{
	contributorsFromOrgOrCompany.Run,
	}
	Vulnerabilities = []ProbeImpl{
	hasOSVVulnerabilities.Run,
	}
	CodeReview = []ProbeImpl{
	codeApproved.Run,
	}
	SAST = []ProbeImpl{
	sastToolConfigured.Run,
	sastToolRunsOnAllCommits.Run,
	}
	DangerousWorkflows = []ProbeImpl{
	hasDangerousWorkflowScriptInjection.Run,
	hasDangerousWorkflowUntrustedCheckout.Run,
	}
	Maintained = []ProbeImpl{
	archived.Run,
	hasRecentCommits.Run,
	issueActivityByProjectMember.Run,
	createdRecently.Run,
	}
	CIIBestPractices = []ProbeImpl{
	hasOpenSSFBadge.Run,
	}
	BinaryArtifacts = []ProbeImpl{
	hasUnverifiedBinaryArtifacts.Run,
	}
	Webhook = []ProbeImpl{
	webhooksUseSecrets.Run,
	}
	CITests = []ProbeImpl{
	testsRunInCI.Run,
	}
	SBOM = []ProbeImpl{
	hasSBOM.Run,
	hasReleaseSBOM.Run,
	}
	SignedReleases = []ProbeImpl{
	releasesAreSigned.Run,
	releasesHaveProvenance.Run,
	}
	BranchProtection = []ProbeImpl{
	blocksDeleteOnBranches.Run,
	blocksForcePushOnBranches.Run,
	branchesAreProtected.Run,
	branchProtectionAppliesToAdmins.Run,
	dismissesStaleReviews.Run,
	requiresApproversForPullRequests.Run,
	requiresCodeOwnersReview.Run,
	requiresLastPushApproval.Run,
	requiresUpToDateBranches.Run,
	runsStatusChecksBeforeMerging.Run,
	requiresPRsToChangeCode.Run,
	}
	PinnedDependencies = []ProbeImpl{
	pinsDependencies.Run,
	}
	TokenPermissions = []ProbeImpl{
	hasNoGitHubWorkflowPermissionUnknown.Run,
	jobLevelPermissions.Run,
	topLevelPermissions.Run,
	}

"probe sets" or "compositions" seems like the right way to approach it without introducing too much additional vocabulary (or layers of complexity).

[puerco]

I'm assuming conformance here means "framework compliance".

This is cool, but also ensure that Scorecard's view of the world can be used at the check and probe level to enable projects and organizations to evaluate adherence to other frameworks. Especially useful for internal/unpublished variants (profiles) of frameworks that organizations define.

[justaugustus]

Agreed — the conformance engine should support arbitrary frameworks and organizational profiles. The probe findings are framework-agnostic by design; OSPS Baseline is just the first (non-"checks") evaluation layer over them.

The same probe evidence can be composed differently for other frameworks, as you suggested above.
We'll make this explicit in the proposal.

[puerco]

The current predicate type is the full scorecard run evaluation. For completeness' sake, it would be nice to have one type for a list of check evaluations and one for probe evaluations.

These are only useful, though, if they have more data than what an SVR has to offer, so I would wait until there is an actual need for them.

[justaugustus]

Right. Probe-level findings are available via --format=probe but have no in-toto wrapper today. Agree that dedicated predicate types for check and probe evaluations are worth considering once there's a concrete need beyond what SVR provides.

Maybe we assume that you want a check-based or probe-based predicate type depending on the run and output options the user provides?

This might suggest the need for a --framework or --evaluation option?

[puerco]

From reading the proposal, wouldn't Scorecard rather become a consumer of control catalogs?

[justaugustus]

Both directions — Scorecard would consume the OSPS Baseline catalog (via security-baseline) for [one type of] conformance evaluation, and Scorecard's own probe definitions (probes/*/def.yml) are already machine-readable YAML with structured metadata.

The "extraction plan" is about packaging those existing definitions for consumption so that tools like other parts of the Scorecard codebase or external tools, like AMPEL, can discover what Scorecard evaluates and compose mappings against it, if needed.

[FelixOliverLange]

Really cool to see in general, as it may also help make output more understandable for users (e.g. pass / fail statements). From my perspective, it would be important to preserve the valuable insights generated by Scorecard, which is also why I have a question on how less mapped checks feed back into the ecosystem. But of course, I'm also only a user of Scorecard.

[FelixOliverLange]

When it comes to ecosystem alignment, will existent Scorecard checks like "Maintained" also feed back into other parts of the ecosystem (as right now, "Maintained" is mainly referenced as precondition)? Checks like "Maintained" can help users of open-source projects to e.g. identify abandoned projects, so would be nice to preserve in a prominent manner.

[justaugustus]

But of course, I'm also only a user of Scorecard.

@FelixOliverLange — User feedback is important to let us know whether we're building the right thing. I appreciate you taking the time to comment! 🚀

Existing checks (like Maintained) would be fully preserved; check scores (0-10) and conformance labels (PASS/FAIL/UNKNOWN) are parallel evaluation layers over the same probe evidence. (See the three-tier evaluation model in the proposal.)

Checks that don't map to OSPS Baseline controls would continue to produce scores as they do today.
The Maintained check's probes also serve double duty: the conformance layer uses them as preconditions
(via the applicability engine) and as evidence toward maintenance-related controls.

No existing check would be deprioritized or removed.

[SecurityCRob]

I am VERY supportive of this effort. Thanks for taking the time to think this through, and thanks to all the contributors for the thoughtful and respectful conversation! scorecard++

[AdamKorcz]

Why does Scorecard need to operate within the ORBIT WG ecosystem? Perhaps add a bit about what the ORBIT WG ecosystem is - that may clear it up.

[AdamKorcz]

Suggestion: Expand MVVSR here.

[AdamKorcz]

A bit of a general question: What are the reasons of adding framework conformance to Scorecard itself instead of having a standalone tool to which we can feed Scorecard findings and where the standalone tool then gives a verdict about framework conformance?

[AdamKorcz]

Would be good with a definition of downstream tools here.

[AdamKorcz]

Is this ("Processing model") the current dataflow and the following section "Three-tier evaluation model" the intended?

[spencerschrock]

As a whole, I support the evidence-based focus. I like the 6 listed design principles.
I didn't quite get a chance to go through all the decisions.md file, as it's quite a lot of feedback to parse through

[spencerschrock]

Is this going to drift into policy/enforcement? Does this conflict with our goal of evidence only?

[spencerschrock]

I'd say this could be observable if it just needs the right token. If this is run in the context of an OSPO self-observation I think it's fine.

[spencerschrock]

to be clear, in this situation "evaluation" or "conformance" layer, just means output format?

[spencerschrock]

What's the value in upstreaming check-level relations? I think mapping probes to baseline controls is fine.

[spencerschrock]

extracting to where?

[spencerschrock]

What sort of maintenance toil will this involve? Do existing controls get updated, or just new ones added?

[spencerschrock]

I would say the cron has additional barriers, cost of writing/serving more data. I have no concerns with the action

[spencerschrock]

this mapping file lives in which repo?

[spencerschrock]

I agree with these principles

[spencerschrock]

This seems unrelated to this change. Other than I assume you using it to generate some of these docs?

[AdamKorcz]

Where is Option A?

[AdamKorcz]

Not sure if this is entirely correct. Currently, I wouldn't say that Scorecard can produce conformance results, but perhaps I am understanding the context of "constraints" incorrectly here; Are these current constraints or are they constraints that should exits with the conformance layer in Scorecard?

[AdamKorcz]

Would be nice to make this more explicit: What is the success criteria for here? The proposal or the implementation?

[justaugustus]

@spencerschrock @AdamKorcz — thanks so much for the feedback!

I'm working on some changes locally and will push them up by tomorrow.