Maven combined detector experiment

[zhenghao104]

MavenWithFallbackDetector

Overview

The MavenWithFallbackDetector is an experimental Maven detector that combines Maven CLI detection with static pom.xml parsing fallback. It provides resilient Maven dependency detection even when the Maven CLI fails (e.g., due to authentication errors with private repositories).

Key Features

• Primary detection: Uses Maven CLI (mvn dependency:tree) for full transitive dependency resolution
• Automatic fallback: Falls back to static pom.xml parsing when Maven CLI fails
• Partial failure handling: Can use Maven CLI results for successful projects while falling back to static parsing for failed ones
• Authentication error detection: Identifies and reports authentication failures with guidance
• Nested pom.xml filtering: Optimizes multi-module projects by only processing root pom.xml files with Maven CLI
• Telemetry: Reports detection method, fallback reasons, and component counts

Detection Flow

┌─────────────────────────────────────────────────────────────────────────┐
│                        OnPrepareDetectionAsync                          │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
                    ┌───────────────────────────────┐
                    │ Check CD_MAVEN_DISABLE_CLI    │
                    │ environment variable          │
                    └───────────────────────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    │                               │
                    ▼                               ▼
            [Disabled=true]                 [false/unset]
                    │                               │
                    ▼                               ▼
        ┌───────────────────┐           ┌───────────────────────┐
        │ Use Static Parser │           │ Check Maven CLI exists │
        │ for ALL pom.xml   │           └───────────────────────┘
        └───────────────────┘                       │
                                    ┌───────────────┴───────────────┐
                                    │                               │
                                    ▼                               ▼
                            [CLI not found]                 [CLI available]
                                    │                               │
                                    ▼                               ▼
                        ┌───────────────────┐       ┌───────────────────────────┐
                        │ Use Static Parser │       │ Create temp Maven local   │
                        │ for ALL pom.xml   │       │ repository (isolated)     │
                        └───────────────────┘       │ RemoveNestedPomXmls       │
                                                    └───────────────────────────┘
                                                                │
                                                                ▼
                                                ┌───────────────────────────────┐
                                                │ Run mvn dependency:tree       │
                                                │ -Dmaven.repo.local=<tempdir>  │
                                                │ for each root pom.xml         │
                                                └───────────────────────────────┘
                                                                │
                                                                ▼
                                                ┌───────────────────────────────┐
                                                │ Scan for bcde-fallback.mvndeps│
                                                │ files (generated by Maven CLI)│
                                                └───────────────────────────────┘
                                                                │
                                                                ▼
                                                ┌───────────────────────────────┐
                                                │ Compare: which pom.xml        │
                                                │ directories have              │
                                                │ bcde-fallback.mvndeps?        │
                                                └───────────────────────────────┘
                                                                │
                                ┌───────────────────────┬───────┴───────────────┐
                                │                       │                       │
                                ▼                       ▼                       ▼
                        [All succeeded]         [All failed]            [Partial]
                                │                       │                       │
                                ▼                       ▼                       ▼
                    ┌─────────────────┐     ┌─────────────────────┐ ┌─────────────────────┐
                    │ Return          │     │ AnalyzeMvnCliFailure│ │ AnalyzeMvnCliFailure│
                    │ bcde-fallback.  │     │ Return ALL pom.xml  │ │ Return bcde-fallback│
                    │ mvndeps only    │     │ for static parsing  │ │ .mvndeps + failed   │
                    │                 │     │ (nested restored)   │ │ pom.xml for static  │
                    │ Method: MvnCli  │     │ Method: StaticOnly  │ │ Method: Mixed       │
                    └─────────────────┘     └─────────────────────┘ └─────────────────────┘
                                                        │
                                                        ▼
                                        ┌───────────────────────────────┐
                                        │ OnDetectionFinishedAsync:     │
                                        │ Cleanup temp local repository │
                                        └───────────────────────────────┘

Core Components

1. Environment Variable Control

internal const string DisableMvnCliEnvVar = "CD_MAVEN_DISABLE_CLI";

CD_MAVEN_DISABLE_CLI:

• Set to true to bypass Maven CLI entirely and use only static parsing

Useful when:

• Maven CLI is known to fail in the environment
• Faster scans are preferred (at the cost of transitive dependencies)
• Debugging static parser behavior

2. Separate Maven Local Repository

To avoid race conditions with MvnCliComponentDetector (which also uses Maven CLI), this detector uses a temporary, isolated Maven local repository:

mvn dependency:tree -Dmaven.repo.local=<temp-directory> ...

Why? When both detectors run in parallel:

• They may process the same pom.xml files simultaneously
• Concurrent writes to the default ~/.m2/repository cause file locking conflicts
• The separate local repository ensures complete isolation

Lifecycle:

1. OnPrepareDetectionAsync: Creates temp directory (e.g., maven-fallback-detector-<guid>)
2. Maven CLI execution: Uses -Dmaven.repo.local=<tempdir> parameter
3. OnDetectionFinishedAsync: Cleans up the temp directory

3. Cancellation Token Propagation

Experimental detectors have a 4-minute timeout enforced by the orchestrator. The MavenWithFallbackDetector properly propagates this cancellation token through the entire call chain:

┌───────────────────────────────────────────────────────────────────────────┐
│ Orchestrator (4-min timeout CancellationToken)                            │
└─────────────────────────────┬─────────────────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ MavenWithFallbackDetector.OnPrepareDetectionAsync(cancellationToken)      │
│   └── GenerateDependenciesFileAsync(..., cancellationToken)               │
└─────────────────────────────┬─────────────────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ MavenCommandService.GenerateDependenciesFileAsync()                       │
│   └── CancellationTokenSource.CreateLinkedTokenSource(cancellationToken)  │
│       (Links framework token with optional file-level timeout)            │
└─────────────────────────────┬─────────────────────────────────────────────┘
                              │
                              ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ CommandLineInvocationService.ExecuteCommandAsync(cancellationToken)       │
│   └── cancellationToken.Register(() => process.Kill())                    │
│       (Kills Maven process if token is cancelled)                         │
└───────────────────────────────────────────────────────────────────────────┘

Key points:

• The 4-minute detector timeout from the orchestrator propagates to the Maven CLI process
• If the timeout is reached, the Maven process is killed immediately via Process.Kill()
• Additional file-level timeouts can be configured via MvnCLIFileLevelTimeoutSeconds environment variable
• The CreateLinkedTokenSource combines both timeouts, so whichever triggers first will cancel the operation

4. RemoveNestedPomXmls

Filters pom.xml files to keep only root-level ones for Maven CLI processing.

Why? In multi-module Maven projects:

/project/
  pom.xml              ← ROOT (parent pom) - KEEP
  /module-a/
    pom.xml            ← NESTED (child) - SKIP
  /module-b/
    pom.xml            ← NESTED (child) - SKIP

Running mvn dependency:tree on the root pom automatically processes all child modules. Processing each nested pom separately would be redundant and slow.

Algorithm:

1. For each pom.xml file, walk up the directory tree
2. If a parent directory contains a pom.xml → skip this file (it's nested)
3. If we reach the filesystem root without finding a parent pom.xml → keep this file (it's a root)

5. Maven CLI Execution

Uses IMavenCommandService.GenerateDependenciesFileAsync() to run:

mvn dependency:tree -B -DoutputFile=bcde-fallback.mvndeps -DoutputType=text -f<pom.xml> -Dmaven.repo.local=<tempdir>

This generates a bcde-fallback.mvndeps file containing the full dependency tree.

Note: The detector uses a different output filename (bcde-fallback.mvndeps) than the standard MvnCliComponentDetector (bcde.mvndeps) and a separate local repository to enable parallel execution.

6. Failure Detection

After Maven CLI runs, the detector scans for bcde-fallback.mvndeps files:

• Directory has bcde-fallback.mvndeps → Maven CLI succeeded for that pom.xml
• Directory missing bcde-fallback.mvndeps → Maven CLI failed for that pom.xml

7. Static Parsing Fallback

When Maven CLI fails, static parsing extracts dependencies directly from pom.xml:

<dependencies>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.12.0</version>
    </dependency>
</dependencies>

Limitations of static parsing:

• Only detects direct dependencies (no transitive)
• Cannot resolve version ranges (e.g., [1.0,2.0))
• May not resolve all property references (e.g., ${project.version})

8. Nested Pom Restoration

When Maven CLI fails (partial or complete), nested pom.xml files that were filtered out by RemoveNestedPomXmls are restored for static parsing.

Why? Maven CLI on the root pom would have processed nested modules. Since it failed, we need to parse each nested pom.xml individually.

Implementation:

• GetAllPomFilesForStaticParsing() - Re-scans ALL pom.xml files (used for complete failure)
• GetStaticParsingRequestsForFailedFiles() - Re-scans pom.xml files only under failed directories (used for partial failure)

9. Authentication Error Analysis

When Maven CLI fails, the detector analyzes error patterns:

private static readonly string[] AuthErrorPatterns =
[
    "401",
    "403",
    "Unauthorized",
    "Access denied",
];

If authentication errors are detected:

1. Sets fallbackReason = MavenFallbackReason.AuthenticationFailure
2. Extracts failed endpoint URLs from error messages
3. Logs guidance for resolving authentication issues

Detection Methods

Method	When Used	What's Detected
MvnCliOnly	Maven CLI succeeds for all pom.xml files	Full transitive dependencies
StaticParserOnly	Maven CLI disabled, unavailable, or fails completely	Direct dependencies only
Mixed	Maven CLI succeeds for some, fails for others	Transitive (where CLI worked) + Direct (where CLI failed)

Fallback Reasons

Reason	Description
None	No fallback needed (MvnCli succeeded)
MvnCliDisabledByUser	User set CD_MAVEN_DISABLE_CLI=true
MavenCliNotAvailable	mvn command not found in PATH
AuthenticationFailure	Maven CLI failed with 401/403 errors
OtherMvnCliFailure	Maven CLI failed for other reasons

Telemetry

The detector records the following telemetry:

Key	Description
DetectionMethod	MvnCliOnly, StaticParserOnly, or Mixed
FallbackReason	Why fallback occurred (if applicable)
MvnCliComponentCount	Components detected via Maven CLI
StaticParserComponentCount	Components detected via static parsing
TotalComponentCount	Total components detected
MavenCliAvailable	Whether Maven CLI was found
OriginalPomFileCount	Number of root pom.xml files processed
FailedEndpoints	URLs of repositories that had auth failures

Version Property Resolution

Static parsing supports resolving version properties:

<properties>
    <commons.version>3.12.0</commons.version>
</properties>
<dependencies>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>${commons.version}</version>  <!-- Resolved to 3.12.0 -->
    </dependency>
</dependencies>

The resolver:

1. Checks <properties> section in current document
2. Checks other loaded pom.xml documents (for multi-module projects)
3. If unresolved, skips the dependency with a debug log

Usage

As Part of Component Detection

The detector is registered as an experimental detector and runs automatically when Component Detection scans a directory containing pom.xml files.

Parallel Execution with MvnCliComponentDetector

This detector can run in parallel with MvnCliComponentDetector without conflicts because:

1. It uses a separate Maven local repository (temp directory) for dependency resolution
2. It uses a different output filename (bcde-fallback.mvndeps vs bcde.mvndeps)

This enables A/B testing and gradual migration without requiring environment variable switches.

Disabling Maven CLI (Static Parsing Only)

Set the environment variable to use only static parsing:

# Bash
export CD_MAVEN_DISABLE_CLI=true

# PowerShell
$env:CD_MAVEN_DISABLE_CLI = "true"

# Azure DevOps Pipeline
variables:
  CD_MAVEN_DISABLE_CLI: 'true'

Experiment Configuration

The detector is paired with MavenWithFallbackExperiment which compares results against the standard MvnCliComponentDetector:

• Control group: MvnCliComponentDetector
• Experiment group: MavenWithFallbackDetector

Enable experiments via:

export CD_DETECTOR_EXPERIMENTS=true

---

Unit Tests

The following unit tests are implemented in MavenWithFallbackDetectorTests.cs:

Maven CLI Scenarios

Test	Description
WhenMavenCliNotAvailable_FallsBackToStaticParsing_Async	Verifies fallback to static parsing when Maven CLI is not installed
WhenMavenCliNotAvailable_DetectsMultipleDependencies_Async	Verifies static parsing detects multiple dependencies correctly
WhenMavenCliSucceeds_UsesMvnCliResults_Async	Verifies Maven CLI results are used when available
WhenMavenCliSucceeds_PreservesTransitiveDependencies_Async	Verifies transitive dependency relationships are preserved
WhenMavenCliProducesNoOutput_FallsBackToStaticParsing_Async	Verifies fallback when Maven CLI runs but produces no output

Static Parser Edge Cases

Test	Description
StaticParser_IgnoresDependenciesWithoutVersion_Async	Verifies dependencies without version tags are skipped
StaticParser_IgnoresDependenciesWithVersionRanges_Async	Verifies version ranges (e.g., [1.0,2.0)) are skipped
StaticParser_ResolvesPropertyVersions_Async	Verifies ${property} versions are resolved from <properties>
StaticParser_IgnoresDependenciesWithUnresolvablePropertyVersions_Async	Verifies unresolvable properties are skipped

Empty/Missing File Scenarios

Test	Description
WhenNoPomXmlFiles_ReturnsSuccessWithNoComponents_Async	Verifies graceful handling of no pom.xml files
WhenPomXmlHasNoDependencies_ReturnsSuccessWithNoComponents_Async	Verifies handling of pom.xml with no dependencies

Environment Variable Control

Test	Description
WhenDisableMvnCliTrue_UsesStaticParsing_Async	Verifies CD_MAVEN_DISABLE_CLI=true bypasses Maven CLI entirely and uses static parsing
WhenDisableMvnCliEnvVarIsFalse_UsesMvnCliNormally_Async	Verifies CD_MAVEN_DISABLE_CLI=false allows Maven CLI to run normally
WhenDisableMvnCliEnvVarNotSet_UsesMvnCliNormally_Async	Verifies Maven CLI runs normally when env var is not set
WhenDisableMvnCliEnvVarSetToInvalidValue_UsesMvnCliNormally_Async	Verifies Maven CLI runs normally when env var has invalid value

Cancellation Token Propagation

The cancellation token from the orchestrator's 4-minute experimental detector timeout is propagated through the call chain:

1. OnPrepareDetectionAsync receives cancellationToken parameter from the framework
2. GenerateDependenciesFileAsync receives and passes the token to the CLI service
3. MavenCommandService links the token with CreateLinkedTokenSource for combined timeouts
4. CommandLineInvocationService registers cancellationToken.Register(() => process.Kill()) to kill Maven

This ensures that if the 4-minute timeout is reached, the Maven CLI process is immediately terminated.

Nested Pom.xml Handling

Test	Description
WhenMvnCliSucceeds_NestedPomXmlsAreFilteredOut_Async	Verifies nested pom.xml files are filtered when Maven CLI is used
WhenMvnCliFailsCompletely_AllNestedPomXmlsAreRestoredForStaticParsing_Async	Verifies all nested pom.xml files are restored when Maven CLI fails completely
WhenMvnCliPartiallyFails_NestedPomXmlsRestoredOnlyForFailedDirectories_Async	Verifies nested pom.xml files are restored only for failed directories in partial failure

Error Analysis and Telemetry

Test	Description
WhenMvnCliFailsWithAuthError_LogsFailedEndpointAndSetsTelemetry_Async	Verifies auth errors (401/403) are detected, failed endpoint URLs are extracted, and telemetry is set to AuthenticationFailure
WhenMvnCliFailsWithNonAuthError_SetsFallbackReasonToOther_Async	Verifies non-auth errors set telemetry to OtherMvnCliFailure without extracting endpoints

---

File Structure

src/Microsoft.ComponentDetection.Detectors/maven/
├── MavenWithFallbackDetector.cs      # Main detector implementation
├── MavenWithFallbackDetector.md      # This documentation
├── MvnCliComponentDetector.cs        # Standard Maven CLI detector
├── MavenCommandService.cs            # Maven CLI execution service
├── IMavenCommandService.cs           # Service interface
└── ...

src/Microsoft.ComponentDetection.Orchestrator/Experiments/Configs/
└── MavenWithFallbackExperiment.cs    # Experiment configuration

test/Microsoft.ComponentDetection.Detectors.Tests/
└── MavenWithFallbackDetectorTests.cs # Unit tests

[codecov]

Codecov Report

❌ Patch coverage is 96.62162% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.6%. Comparing base (6748194) to head (28dd56d).

Files with missing lines	Patch %	Lines
...ction.Detectors/maven/MavenWithFallbackDetector.cs	93.8%	22 Missing and 8 partials ⚠️
....Detectors.Tests/MavenWithFallbackDetectorTests.cs	99.1%	0 Missing and 6 partials ⚠️
...Experiments/Configs/MavenWithFallbackExperiment.cs	66.6%	2 Missing and 1 partial ⚠️
...ntDetection.Detectors/maven/MavenCommandService.cs	90.9%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #1628     +/-   ##
=======================================
+ Coverage   90.4%   90.6%   +0.1%     
=======================================
  Files        441     445      +4     
  Lines      38314   39495   +1181     
  Branches    2347    2407     +60     
=======================================
+ Hits       34674   35815   +1141     
- Misses      3159    3183     +24     
- Partials     481     497     +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR introduces MavenWithFallbackDetector, an experimental Maven detector that provides resilient dependency detection by combining Maven CLI execution with static pom.xml parsing fallback. The detector automatically falls back to static parsing when Maven CLI fails (e.g., due to authentication errors or missing CLI), ensuring components are still detected even in challenging build environments.

Changes:

• Added MavenWithFallbackDetector with dual detection strategy (Maven CLI primary, static parsing fallback) and comprehensive telemetry
• Registered new experiment configuration comparing the new detector against the standard MvnCliComponentDetector
• Added 13 comprehensive unit tests covering CLI scenarios, static parser edge cases, environment variable control, and nested pom.xml handling

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
src/Microsoft.ComponentDetection.Detectors/maven/MavenWithFallbackDetector.cs	Main detector implementation with Maven CLI detection, static XML parsing fallback, nested pom filtering, and telemetry tracking
src/Microsoft.ComponentDetection.Orchestrator/Experiments/Configs/MavenWithFallbackExperiment.cs	Experiment configuration to compare new detector against existing MvnCliComponentDetector
src/Microsoft.ComponentDetection.Orchestrator/Extensions/ServiceCollectionExtensions.cs	Registered new detector and experiment in DI container
test/Microsoft.ComponentDetection.Detectors.Tests/MavenWithFallbackDetectorTests.cs	Comprehensive test suite with 13 tests covering all detection scenarios and edge cases

[github-actions]

👋 Hi! It looks like you modified some files in the Detectors folder.
You may need to bump the detector versions if any of the following scenarios apply:

• The detector detects more or fewer components than before
• The detector generates different parent/child graph relationships than before
• The detector generates different devDependencies values than before

If none of the above scenarios apply, feel free to ignore this comment 🙂

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

[grvillic]

only record if either MavenWithFallbackDetector or MvnCliComponentDetector have at least one component, otherwise you will be just flushing experiments without any meaningful data.

[zhenghao104]

Added numComponents > 0

[grvillic]

Aren't this already covered by line 106 and 107?

[zhenghao104]

Trimmed it to

private static readonly string[] AuthErrorPatterns =
[
"401",
"403",
"Unauthorized",
"Access denied",
];

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[grvillic]

See other experiments, just looking at components will cause to skip the whole experiment altogether, you should always return true, unless the new detector does not return any dependencies.

[zhenghao104]

updated with

    public bool ShouldRecord(IComponentDetector componentDetector, int numComponents)
    {
        // Only record telemetry if a Maven detector found components,
        // indicating Maven projects were detected and scanned.
        if (componentDetector is MvnCliComponentDetector or MavenWithFallbackDetector)
        {
            return numComponents > 0;
        }

        return true;
    }

Looking at the other experiments and the one I wrote for nuget, this should be correct.

[grvillic]

Is this the preexisting behavior of MvnCLI? Seem risky to be enumerating streams within the same file.

[grvillic]

Confirm this method is protected by the experimental coverage. IIRC it does not have the same guardrails as "OnFileFound", hence wrap all of this with a try catch and chain an additional cancellation token so the execution stops within 4-6 minutes. Assuming most repos complete the full scan within this window. That way we can see execution hit limits in our telemetry but no unexpected hangs for existing users.

[grvillic]

I haven't manually validated these changes, but have we validated that if a root pom.xml file succeeded with mvn cli, we are excluding child ones? It will be unnecessary work if run CLI or parse directly pom.xml on child pom.xml when the root one covered them. Although the manual parsing is faster, the slow one that significantly impacts perf is MvnCLI.

[grvillic]

nit: I don't think there is the need to add the async/await here, we only need to have a Task output.