Skip to content

feat(#3529): move ThresholdConfig from MetricProvider to individual Metrics#3560

Open
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/3529-refactor-metric-provider
Open

feat(#3529): move ThresholdConfig from MetricProvider to individual Metrics#3560
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/3529-refactor-metric-provider

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Refactor the scorecard MetricProvider interface to move threshold configuration from provider-level methods to individual Metric objects.

Core interface changes:

  • Add threshold: ThresholdConfig field to the Metric type
  • Remove getMetricType(), getMetric(), getMetricThresholds(),
    calculateMetric(), and getMetricIds() from MetricProvider
  • Make getMetrics() and calculateMetrics() required on MetricProvider
  • Consolidate single/batch provider code paths into batch-only

Provider migrations:

  • All providers (dependabot, filecheck, github, jira, openssf, sonarqube)
    updated to the new interface with thresholds on metric objects
  • ThresholdResolver API: resolveProviderThresholds(provider)
    resolveMetricThresholds(metric, providerId)
  • ThresholdResolver API: resolveEntityThresholds(entity, provider)
    resolveEntityThresholds(entity, metric, providerId)
  • mergeEntityAndProviderThresholds signature updated similarly
  • PullMetricsByProviderTask consolidated to always use calculateMetrics()
  • MetricProvidersRegistry uses provider.getMetrics() for all operations

All tests updated. API reports regenerated.

Note: DatabaseMetricValues tests could not run (missing better-sqlite3 native binding in sandbox). All other 91 test suites (976 tests) passed.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com


Closes #3529

Post-script verification

  • Branch is not main/master (agent/3529-refactor-metric-provider)
  • Secret scan passed (gitleaks — b57a430ea0c48b4dff885dc812072d5eccb6e0a2..HEAD)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

…etrics

Refactor the scorecard MetricProvider interface to move threshold
configuration from provider-level methods to individual Metric objects.

Core interface changes:
- Add `threshold: ThresholdConfig` field to the `Metric` type
- Remove `getMetricType()`, `getMetric()`, `getMetricThresholds()`,
  `calculateMetric()`, and `getMetricIds()` from MetricProvider
- Make `getMetrics()` and `calculateMetrics()` required on MetricProvider
- Consolidate single/batch provider code paths into batch-only

Provider migrations:
- All providers (dependabot, filecheck, github, jira, openssf, sonarqube)
  updated to the new interface with thresholds on metric objects
- ThresholdResolver API: `resolveProviderThresholds(provider)` →
  `resolveMetricThresholds(metric, providerId)`
- ThresholdResolver API: `resolveEntityThresholds(entity, provider)` →
  `resolveEntityThresholds(entity, metric, providerId)`
- mergeEntityAndProviderThresholds signature updated similarly
- PullMetricsByProviderTask consolidated to always use calculateMetrics()
- MetricProvidersRegistry uses provider.getMetrics() for all operations

All tests updated. API reports regenerated.

Note: DatabaseMetricValues tests could not run (missing better-sqlite3
native binding in sandbox). All other 91 test suites (976 tests) passed.

Closes #3529

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@rhdh-gh-app

rhdh-gh-app Bot commented Jun 24, 2026

Copy link
Copy Markdown

Missing Changesets

The following package(s) are changed by this PR but do not have a changeset:

  • @red-hat-developer-hub/backstage-plugin-scorecard-backend-module-dependabot
  • @red-hat-developer-hub/backstage-plugin-scorecard-backend-module-filecheck
  • @red-hat-developer-hub/backstage-plugin-scorecard-backend-module-github
  • @red-hat-developer-hub/backstage-plugin-scorecard-backend-module-jira
  • @red-hat-developer-hub/backstage-plugin-scorecard-backend-module-openssf
  • @red-hat-developer-hub/backstage-plugin-scorecard-backend-module-sonarqube
  • @red-hat-developer-hub/backstage-plugin-scorecard-backend
  • @red-hat-developer-hub/backstage-plugin-scorecard-common
  • @red-hat-developer-hub/backstage-plugin-scorecard-node

See CONTRIBUTING.md for more information about how to add changesets.

Changed Packages

Package Name Package Path Changeset Bump Current Version
@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-dependabot workspaces/scorecard/plugins/scorecard-backend-module-dependabot none v0.2.13
@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-filecheck workspaces/scorecard/plugins/scorecard-backend-module-filecheck none v0.1.11
@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-github workspaces/scorecard/plugins/scorecard-backend-module-github none v2.7.9
@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-jira workspaces/scorecard/plugins/scorecard-backend-module-jira none v2.7.9
@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-openssf workspaces/scorecard/plugins/scorecard-backend-module-openssf none v0.2.13
@red-hat-developer-hub/backstage-plugin-scorecard-backend-module-sonarqube workspaces/scorecard/plugins/scorecard-backend-module-sonarqube none v0.1.8
@red-hat-developer-hub/backstage-plugin-scorecard-backend workspaces/scorecard/plugins/scorecard-backend none v2.7.9
@red-hat-developer-hub/backstage-plugin-scorecard-common workspaces/scorecard/plugins/scorecard-common none v2.7.9
@red-hat-developer-hub/backstage-plugin-scorecard-node workspaces/scorecard/plugins/scorecard-node none v2.7.9

@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.00000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.81%. Comparing base (b57a430) to head (1874c31).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3560      +/-   ##
==========================================
- Coverage   53.82%   53.81%   -0.01%     
==========================================
  Files        2277     2277              
  Lines       87047    87014      -33     
  Branches    24387    24375      -12     
==========================================
- Hits        46850    46826      -24     
+ Misses      38660    38651       -9     
  Partials     1537     1537              
Flag Coverage Δ *Carryforward flag
adoption-insights 83.70% <ø> (ø) Carriedforward from b57a430
ai-integrations 67.95% <ø> (ø) Carriedforward from b57a430
app-defaults 69.79% <ø> (ø) Carriedforward from b57a430
augment 46.39% <ø> (ø) Carriedforward from b57a430
boost 68.41% <ø> (ø) Carriedforward from b57a430
bulk-import 72.46% <ø> (ø) Carriedforward from b57a430
cost-management 14.10% <ø> (ø) Carriedforward from b57a430
dcm 61.79% <ø> (ø) Carriedforward from b57a430
extensions 61.53% <ø> (ø) Carriedforward from b57a430
global-floating-action-button 71.18% <ø> (ø) Carriedforward from b57a430
global-header 59.71% <ø> (ø) Carriedforward from b57a430
homepage 49.92% <ø> (ø) Carriedforward from b57a430
install-dynamic-plugins 56.77% <ø> (ø) Carriedforward from b57a430
konflux 91.49% <ø> (ø) Carriedforward from b57a430
lightspeed 68.57% <ø> (ø) Carriedforward from b57a430
mcp-integrations 85.46% <ø> (ø) Carriedforward from b57a430
orchestrator 38.02% <ø> (ø) Carriedforward from b57a430
quickstart 63.76% <ø> (ø) Carriedforward from b57a430
sandbox 79.56% <ø> (ø) Carriedforward from b57a430
scorecard 84.07% <95.00%> (+0.11%) ⬆️
theme 61.26% <ø> (ø) Carriedforward from b57a430
translations 7.25% <ø> (ø) Carriedforward from b57a430
x2a 78.68% <ø> (ø) Carriedforward from b57a430

*This pull request uses carry forward flags. Click here to find out more.


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b57a430...1874c31. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sonarqubecloud

Copy link
Copy Markdown

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 24, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 12:49 AM UTC · Completed 1:03 AM UTC
Commit: b57a430 · View workflow run →

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

Critical

  • [breaking-change] workspaces/scorecard/plugins/scorecard-node/src/api/MetricProvider.ts — Five required/optional methods removed from the public MetricProvider interface without deprecation: getMetricType(), getMetric(), getMetricThresholds(), calculateMetric(), and getMetricIds(). Additionally, getMetrics() and calculateMetrics() changed from optional to required. Any downstream plugin implementing MetricProvider will fail to compile.
    Remediation: Either keep removed methods as deprecated optional members with default implementations and remove in a future major version, or treat this as a major/breaking semver bump for @red-hat-developer-hub/backstage-plugin-scorecard-node.

  • [breaking-change] workspaces/scorecard/plugins/scorecard-common/src/types/Metric.ts — A new required property threshold: ThresholdConfig was added to the public Metric type. All existing code that constructs Metric objects without supplying threshold will fail to compile. This affects every downstream consumer of @red-hat-developer-hub/backstage-plugin-scorecard-common.
    Remediation: Either make threshold optional (threshold?: ThresholdConfig) so existing code remains compatible, or perform a major semver bump for @red-hat-developer-hub/backstage-plugin-scorecard-common.

  • [missing-version-bump] workspaces/scorecard/plugins/scorecard-node/report.api.md — No changeset for the breaking changes to @red-hat-developer-hub/backstage-plugin-scorecard-node and @red-hat-developer-hub/backstage-plugin-scorecard-common. These are backward-incompatible changes that require at minimum a major semver bump.
    Remediation: Add a changeset with major bump for both packages and include migration notes describing how downstream providers should update.

Medium

  • [logic-error] workspaces/scorecard/plugins/scorecard-backend/src/threshold/ThresholdResolver.ts:47 — In setConfiguredThresholds, configured thresholds are stored per providerId (not per metricId) and validated using metrics[0].type. For batch providers, a single config-override threshold applies to all metrics uniformly. This is a pre-existing limitation preserved by the PR, but it is now more visible since individual metrics carry their own defaults that could differ.

  • [breaking-change] workspaces/scorecard/plugins/scorecard-backend/src/threshold/ThresholdResolver.tsresolveProviderThresholds(provider) renamed to resolveMetricThresholds(metric, providerId) and resolveEntityThresholds signature changed. These are internal to the scorecard-backend plugin (not exported from the public API surface), so the impact is limited to in-repo consumers.

  • [breaking-change] workspaces/scorecard/plugins/scorecard-backend/src/utils/mergeEntityAndProviderThresholds.ts — Signature changed from (entity, provider, baseThresholds?) to (entity, metric, providerId, baseThresholds?). Also internal to scorecard-backend, not exported publicly.

Low

  • [missing-deprecation] workspaces/scorecard/plugins/scorecard-node/src/api/MetricProvider.ts — No deprecation period for removed interface members with @public JSDoc tags. Best practice is to deprecate for at least one release before removal.

  • [naming-consistency] workspaces/scorecard/plugins/scorecard-backend/__fixtures__/mockProviders.ts:52 — The abstract method getDefaultThresholds() diverges from production provider patterns where thresholds are inlined into getMetrics().

  • [api-shape-consistency] workspaces/scorecard/plugins/scorecard-backend/src/scheduler/tasks/PullMetricsByProviderTask.ts — The logger.warn call for per-entity calculation failures was removed during code path consolidation. Errors are still recorded in the database but no longer logged.

  • [edge-case] workspaces/scorecard/plugins/scorecard-backend/src/providers/MetricProvidersRegistry.ts:39 — If provider.getMetrics() returns an empty array, no metrics are registered and no validation occurs. The provider is silently ignored.

  • [error-handling-idiom] workspaces/scorecard/plugins/scorecard-backend/src/providers/MetricProvidersRegistry.ts:108calculateMetric throws plain Error while getMetric uses NotFoundError from @backstage/errors for the same class of condition.

  • [code-organization] workspaces/scorecard/plugins/scorecard/dev/legacy.tsx — The fallback { rules: [] } is repeated in 5 files. Consider extracting an EMPTY_THRESHOLD_CONFIG constant.

  • [test-weakened] workspaces/scorecard/plugins/scorecard-backend/src/providers/MetricProvidersRegistry.test.ts:125 — Error assertion loosened from exact string match to regex.

Info

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

metric: Metric,
providerId: string,
): ThresholdConfig {
return mergeEntityAndProviderThresholds(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] logic-error

In setConfiguredThresholds, configured thresholds are stored per providerId (not per metricId) and validated using metrics[0].type. Pre-existing limitation preserved by the PR but now more visible.

const providerDatasource = metricProvider.getProviderDatasourceId();
const metricType = metricProvider.getMetricType();
const providerId = metricProvider.getProviderId();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] edge-case

If provider.getMetrics() returns empty array, provider is silently ignored.

return provider.getMetric();
throw new NotFoundError(
`Metric '${metricId}' not found in provider '${provider.getProviderId()}'`,
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] error-handling-idiom

calculateMetric throws plain Error while getMetric uses NotFoundError for same condition.

@christoph-jerolimov

Copy link
Copy Markdown
Member

/fs-fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ❌ Failure · Started 2:42 AM UTC · Completed 4:16 AM UTC
Commit: b57a430 · View workflow run →

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor Scorecard: move ThresholdConfig from MetricProvider to individual Metrics

1 participant