Skip to content

Fix MQTT reconnection failures due to expired AWS credentials#27

Merged
eman merged 2 commits intomainfrom
fix/mqtt-reconnection-aws-credentials
Oct 26, 2025
Merged

Fix MQTT reconnection failures due to expired AWS credentials#27
eman merged 2 commits intomainfrom
fix/mqtt-reconnection-aws-credentials

Conversation

@eman
Copy link
Copy Markdown
Owner

@eman eman commented Oct 26, 2025

Problem

The MQTT client was experiencing reconnection failures with the following errors:

AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
AWS_ERROR_HTTP_WEBSOCKET_UPGRADE_FAILURE: Failed to upgrade HTTP connection to Websocket.

Root Cause

AWS IoT credentials have a separate expiration time from JWT tokens. The reconnection logic was:

  1. Only checking JWT token expiration before reconnecting
  2. Not checking if AWS credentials had expired
  3. Attempting reconnection with expired AWS credentials
  4. Failing with WebSocket upgrade errors

Since the token refresh endpoint doesn't return new AWS credentials (only new JWT tokens), expired AWS credentials couldn't be renewed through token refresh alone.

Solution

Core Changes to src/nwp500/auth.py

  1. Added AWS credential expiration tracking

    • New _aws_expires_at field to track when AWS credentials expire separately from JWT tokens
    • Calculated from authorization_expires_in during token initialization
  2. Added expiration check property

    • New are_aws_credentials_expired property validates AWS credential validity
    • Uses 5-minute buffer before expiration (same as JWT tokens)
    • Returns False if expiration time unknown (backward compatible)
  3. Enhanced credential validation

    • Modified ensure_valid_token() to check AWS credentials first (before JWT token)
    • Triggers full re-authentication when AWS credentials expire (not just token refresh)
    • Only does token refresh if JWT expired but AWS credentials still valid
  4. Preserved AWS expiration timestamps

    • Updated token refresh logic to preserve _aws_expires_at timestamp
    • Ensures expiration tracking persists across token refreshes

New Test Suite - tests/test_auth.py

Created comprehensive test coverage with 40 tests organized into 5 categories:

  1. UserInfo Tests (5 tests) - Dataclass creation, parsing, properties
  2. AuthTokens Tests (12 tests) - Expiration logic, AWS credentials, token handling
  3. AuthenticationResponse Tests (1 test) - Response parsing
  4. Exception Classes (4 tests) - Error handling and inheritance
  5. NavienAuthClient Tests (18 tests) - Client functionality, async operations

Critical Tests for MQTT Fix

  • test_auth_tokens_aws_credentials_expired_false/true - Validates AWS credential expiration detection
  • test_ensure_valid_token_aws_credentials_expired - Ensures re-authentication when AWS credentials expire
  • test_aws_credentials_preservation_in_token_refresh - Validates credential timestamp preservation

Testing

  • All 80 tests passing (40 new auth tests + 40 existing tests)
  • Type checking: mypy - PASSED
  • Linting: ruff - PASSED
  • Coverage: auth.py improved from 31% → 60%

Impact

This fix prevents MQTT reconnection failures that occur when:

  • Connection is interrupted after AWS credentials expire
  • But before JWT tokens expire (AWS credentials typically expire sooner)
  • Reconnection attempts would use expired AWS credentials

The client now automatically re-authenticates to obtain fresh AWS credentials when needed for reconnection, ensuring reliable MQTT connectivity.

Backward Compatibility

  • ✅ No breaking changes to the API
  • ✅ Automatic handling - users don't need to change their code
  • ✅ Gracefully handles cases where authorization_expires_in is not provided

**Problem:**
MQTT client was failing to reconnect after connection interruptions with
AWS_ERROR_HTTP_WEBSOCKET_UPGRADE_FAILURE errors. The root cause was that
AWS IoT credentials have a separate expiration time from JWT tokens, but
the reconnection logic only checked JWT expiration before attempting to
reconnect with potentially expired AWS credentials.

**Solution:**
- Added _aws_expires_at field to AuthTokens to track AWS credential expiration
- Added are_aws_credentials_expired property to check AWS credential validity
- Modified ensure_valid_token() to prioritize AWS credential expiration check
- Triggers full re-authentication (not just token refresh) when AWS credentials expire
- Preserves AWS credential expiration timestamps during token refresh

**Changes:**
- src/nwp500/auth.py: Added AWS credential expiration tracking and validation
- tests/test_auth.py: Created comprehensive test suite (40 tests) for auth module

**Testing:**
- All 80 tests passing (40 new auth tests + 40 existing tests)
- Type checking: mypy - PASSED
- Linting: ruff - PASSED
- Coverage: auth.py improved from 31% to 60%

**Impact:**
Prevents MQTT reconnection failures that occur when:
- Connection is interrupted after AWS credentials expire
- But before JWT tokens expire (AWS credentials typically expire sooner)
- Reconnection attempts would use expired AWS credentials

Fixes #<issue_number> (if applicable)
@eman eman requested a review from Copilot October 26, 2025 04:32
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes MQTT reconnection failures caused by expired AWS IoT credentials. The core issue was that the authentication system only checked JWT token expiration, not AWS credential expiration, leading to failed reconnections when AWS credentials expired before JWT tokens. The solution adds separate tracking and validation for AWS credential expiration, triggering full re-authentication when AWS credentials expire (since token refresh doesn't provide new AWS credentials).

Key Changes:

  • Added AWS credential expiration tracking with _aws_expires_at field
  • Implemented are_aws_credentials_expired property to validate AWS credential validity
  • Modified ensure_valid_token() to check AWS credentials before JWT tokens and trigger re-authentication when needed
  • Added comprehensive test suite (40 new tests) covering authentication flows and expiration scenarios

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/nwp500/auth.py Added AWS credential expiration tracking, validation property, and enhanced token refresh logic to preserve AWS credential timestamps
tests/test_auth.py New comprehensive test suite with 40 tests covering UserInfo, AuthTokens, exceptions, and NavienAuthClient functionality

…ired

The docstring incorrectly stated 'True if AWS credentials are expired or
expiration time is unknown', but the function returns False when expiration
time is unknown. Updated to: 'True if AWS credentials are expired, False if
expiration time is unknown or credentials are still valid'
@eman eman merged commit 83b8c34 into main Oct 26, 2025
10 checks passed
@eman eman deleted the fix/mqtt-reconnection-aws-credentials branch November 8, 2025 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants