Skip to content

Fixed cloudflare fetch#292

Open
IJIJI wants to merge 5 commits into
sparkmicro:mainfrom
Synapt-Technologies:main
Open

Fixed cloudflare fetch#292
IJIJI wants to merge 5 commits into
sparkmicro:mainfrom
Synapt-Technologies:main

Conversation

@IJIJI

@IJIJI IJIJI commented Jun 21, 2026

Copy link
Copy Markdown

Fixed the scraping of Cloudflare-protected sites. I have only tested this with LCSC (#291
) and Digikey, but I expect it to fix the Cloudflare blocking of other platforms that where failing before?
Also, the LCSC mappings have changed. I have changed it to only look at the keys of values that don't change as often, to improve long-term stability.

Copilot AI review requested due to automatic review settings June 21, 2026 16:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to improve scraping/fetching reliability for Cloudflare-protected supplier endpoints (notably LCSC/Digi-Key) and make the LCSC API “test” check less brittle against upstream text changes.

Changes:

  • Add a Cloudflare-aware fallback path in download() by retrying API fetches via cloudscraper.
  • Adjust LCSC test_api() expectations to rely on stable identity fields plus a loose check on the free-text description.
  • Regenerate poetry.lock (Poetry 2.4.1) and update .gitignore entries.

Reviewed changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 2 comments.

File Description
kintree/common/tools.py Adds cloudscraper helper and introduces a requests→cloudscraper→urllib fallback chain for JSON API downloads.
kintree/search/lcsc_api.py Makes the LCSC API test less sensitive to changing description strings.
poetry.lock Lockfile regenerated; includes notable dependency source changes (e.g., digikey-api via Git).
.gitignore Ignores the whole .vscode folder and a Copilot instructions file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread kintree/common/tools.py
Comment thread kintree/search/lcsc_api.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants