Bug 2027142 - The github-etl script is failing due to various errors such as bad credentials, network connection closed, etc.

[dklawren]

No description provided.

[Copilot]

Pull request overview

This PR hardens the github-etl orchestration and GitHub API access layer to better tolerate transient failures (network issues, 5xxs, HTML error pages) and expired/invalid credentials, while expanding the test suite to cover the new behaviors.

Changes:

• Add robust retry/backoff + request timeouts to github_get, including optional auth refresh on 401s and HTML error-page detection.
• Propagate refresh_auth through extraction helpers and make _main() continue processing subsequent repos after a repo-level failure, returning a partial-failure exit code.
• Update/add tests for retries and orchestration flows; remove GITHUB_TOKEN usage from tests and docker-compose.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
main.py	Adds retry/backoff/timeouts in github_get, threads refresh_auth through extractors, and continues ETL across repos on failure.
tests/test_main.py	Updates orchestration tests to run without auth env vars; adds a repo-failure-continues test.
tests/test_main_integration.py	Adds a broad set of orchestration/integration-style tests (currently largely duplicative of tests/test_main.py).
tests/test_github_get.py	Adds focused tests for retry behaviors (5xx, connection errors, 401 refresh, HTML error pages).
tests/test_logging.py	Adds a test validating setup_logging() configuration.
tests/test_extract_pull_requests.py	Adjusts tests to align with new retry behavior and header expectations.
tests/test_extract_commits.py	Adjusts tests to align with new retry behavior and header expectations.
tests/test_extract_reviewers.py	Adjusts tests to align with new retry behavior and header expectations.
tests/test_extract_comments.py	Ensures mocked responses include headers used by HTML detection.
docker-compose.yml	Removes GITHUB_TOKEN env var from the compose example (token no longer used).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.