Skip to content

fix(mcp): logged, single-flight, proactive OAuth refresh for MCP servers - #1697

Closed
Aaronontheweb wants to merge 5 commits into
netclaw-dev:devfrom
Aaronontheweb:fix/mcp-oauth-refresh
Closed

fix(mcp): logged, single-flight, proactive OAuth refresh for MCP servers#1697
Aaronontheweb wants to merge 5 commits into
netclaw-dev:devfrom
Aaronontheweb:fix/mcp-oauth-refresh

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Defects (from #1696)

  1. Refresh failures were invisible — the live path delegated to the MCP SDK's ClientOAuthProvider, which refreshes silently and falls back to a headless-fatal interactive flow; Netclaw's own logged refresh path (McpOAuthService.GetValidTokenAsync) had zero callers.
  2. No single-flight on refresh — concurrent callers against an expired or rejected token could redeem a rotating refresh token twice, which providers treat as theft and revoke the grant. Concurrent reconnects could also replace each other's newly-created client.
  3. Refresh was reactive-only — nothing refreshed ahead of the persisted expiry, so idle/restarted daemons only discovered a dead token via a 401.
  4. Token sets without a refresh token dead-ended silently at expiry with no advance warning.

Fix

  • McpOAuthService is now the sole runtime token authority. A Netclaw-owned HTTP authorization handler attaches access tokens, routes both proactive and on-401 refresh through the same per-server single-flight, and retries the rejected request once. The MCP SDK no longer receives the refresh token or independently redeems it; the former ITokenCache bridge was removed.
  • On a 401, refresh is tied to the exact rejected access token. Concurrent requests waiting behind the winner reuse its replacement token instead of redeeming the rotated refresh token again.
  • McpClientManager.TryReconnectAsync now uses a version-aware per-server gate. Overlapping reconnect callers reuse the first successful replacement client; a failed leader allows the next waiter to retry. Terminal teardown uses the same gate and observes cancellation.
  • Proactive refresh rides McpReconnectionService's existing 30s tick: connected OAuth servers refresh once within a 10-minute window of ExpiresAt (constant, TimeProvider-driven).
  • invalid_grant, invalid_client, and unauthorized_client are terminal: tokens are cleared, the connection is torn down, and diagnostics report a distinct AuthFailed status with reauthorization guidance. Client-registration errors also clear the cached DCR client ID so reauthorization registers a fresh client. Transient failures retain tokens and remain retryable.
  • Missing-refresh-token and unknown-expiry conditions are surfaced without log spam. Static Authorization headers remain operator-owned and bypass managed OAuth refresh.
  • netclaw-operations diagnostics and netclaw doctor explain the distinct statuses.

Tests

  • Concurrent proactive refresh produces exactly one token-endpoint request.
  • Concurrent resource-server 401s produce exactly one refresh request, and every request retries with the replacement token.
  • Retried POST requests preserve their body and headers.
  • Concurrent reconnects create/reuse one successful replacement; a failed leader permits a waiter to retry; teardown waiting observes cancellation.
  • A real HTTP MCP client is connected, terminally rejected during proactive refresh, torn down, and moved to the distinct AuthFailed status.
  • Terminal OAuth error parsing, transient retention, proactive timing, missing refresh-token warnings, doctor output, and encrypted persistence remain covered.

Validation

  • Full dotnet test --no-restore passed.
  • dotnet slopwatch analyze passed with zero findings.
  • pwsh ./scripts/Add-FileHeaders.ps1 -Verify and git diff --check passed.
  • Behavioral eval execution was attempted but the required NETCLAW_EVAL_PROVIDER_TYPE, NETCLAW_EVAL_PROVIDER_ENDPOINT, and NETCLAW_EVAL_MODEL_ID are not configured in this workspace.

Fixes #1696

…h rejections

Parse the RFC 6749 error body at the token endpoint (4xx only) and surface
invalid_grant, invalid_client, and unauthorized_client as a typed
OAuthTokenRefreshRejectedException. McpOAuthService handles all three in one
shared terminal path (clear + persist + alert with the code as reason); client
errors additionally drop the cached DCR client_id so re-auth performs a fresh
registration instead of reusing the dead one. 5xx, network errors, and
unrecognized/unparsable bodies stay transient.
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Addressed review: invalid_client / unauthorized_client are now terminal alongside invalid_grant. The token endpoint's 4xx error body is parsed (RFC 6749 §5.2) into a typed OAuthTokenRefreshRejectedException; McpOAuthService handles all three codes in one shared terminal path (clear tokens + persist + alert with the actual code as reason), and client errors additionally drop the cached DCR client_id — EnsureClientRegisteredAsync short-circuits on it, so without the clear a re-auth would have reused the dead registration. 5xx / network / unrecognized or unparsable bodies remain transient. New tests cover terminal classification per code, the 5xx-with-terminal-shaped-body case, and a full invalid_client → re-auth → fresh DCR recovery.

@Aaronontheweb Aaronontheweb added mcp Model context protocol server / client issues. security Security-related changes labels Jul 19, 2026
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Refresh was reactive-only — nothing refreshed ahead of the persisted expiry, so idle/restarted daemons only discovered a dead token via a 401.

Yeah, this is the real killer. I noticed that my Notion MCP connections were needing to be reset every couple of weeks or so, and this is why.

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rejected. This is massively over-engineered for the problem we were trying to solve.

// servers — e.g. an OAuth token with no refresh token, which
// will hard-fail at expiry with no chance of silent
// recovery. See McpClientManager.BuildOAuthAdvisory.
statusMessages.Add(string.IsNullOrEmpty(error)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

[InlineData("invalid_grant", HttpStatusCode.BadRequest)]
[InlineData("invalid_client", HttpStatusCode.Unauthorized)]
[InlineData("unauthorized_client", HttpStatusCode.BadRequest)]
public async Task RefreshToken_TerminalOAuthError_ThrowsRejectedWithErrorCode(

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forcibly tests the types of error codes we get back on token exchange to ensure that there's a visible error that appears in the logs this time around

var results = await Task.WhenAll(reconnects);

Assert.All(results, Assert.True);
Assert.Equal(1, Volatile.Read(ref reconnectCount));

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

ct);
}

private McpReconnectGate GetReconnectGate(McpServerName name)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

populates the McpReconnectGate for this MCP server on-demand

/// state so `netclaw doctor` and the daemon status API can show it ahead
/// of time (see McpServersDoctorCheck's "Connected" case).
/// </summary>
private static string? BuildOAuthAdvisory(McpServerName serverName, McpOAuthTokenSet? tokenSet)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - give user a heads up that they'll need to log back in in N days.

client = null;
_statuses[name] = new McpServerStatus(name, McpConnectionState.Connected, tools.Count, null);

// Advance warning for OAuth-managed servers whose token can never

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

// refresh path before constructing the HTTP transport. Runtime
// requests use McpOAuthAuthorizationHandler, so the SDK never
// receives or independently redeems the refresh token.
await _oauthService.GetValidTokenAsync(name, entry, ct);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refresh happens internally inside here otherwise

/// refresh. The MCP SDK's OAuth provider cannot participate here because it
/// redeems cached refresh tokens outside Netclaw's per-server single-flight.
/// </summary>
internal sealed class McpOAuthAuthorizationHandler : DelegatingHandler

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this looks like we basically ripped out what was previously supported by the ModelContextProtocol SDK and started doing it ourselves to work around, what is fundamentally a race condition? That seems:

  1. A bad idea
  2. Are you sure we aren't solving a problem that the ModelContextProtocol SDK already handles perfectly fine on its own?

Going to lock this PR down until we figure that out definitively. This stinks.

@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Closing after revisiting the OAuth ownership boundary. This PR made Netclaw responsible for bearer injection, request replay, reactive 401 handling, and token refresh in order to provide single-flight behavior. The MCP SDK maintainers have confirmed that concurrency defect in modelcontextprotocol/csharp-sdk#1595 and are implementing the fix inside ClientOAuthProvider in modelcontextprotocol/csharp-sdk#1708, which is the correct ownership layer. No published SDK version contains that fix yet. Netclaw #1696 has been converted into an upstream dependency tracker and now records the remaining Netclaw-owned lifecycle and diagnostic work. We will upgrade to a published SDK fix and validate it through the real SDK transport rather than maintain a parallel OAuth stack. The branch is being retained for auditability.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mcp Model context protocol server / client issues. observability security Security-related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP OAuth: track upstream single-flight refresh and harden Netclaw diagnostics

1 participant