Account for scrapy-poet Zyte API parameters when fingerprinting

[AdrianAtZyte]

Before this change, one of the 2 URLs in the example below would be skipped:

from scrapy import Request, Spider
from scrapy_poet import DummyResponse
from zyte_common_items import JobPostingNavigation

class TestSpider(Spider):
    name = "test"
    custom_settings = {
        "ZYTE_API_LOG_REQUESTS": True,
    }

    def start_requests(self):
        for slug in ("foo", "bar"):
            yield Request(
                f"https://toscrape.com/#/{slug}",
                meta={
                    "zyte_api_provider": {
                        "jobPostingNavigationOptions": {"extractFrom": "browserHtml"},
                    },
                },
            )

    async def parse(self, response: DummyResponse, item: JobPostingNavigation):
        yield item

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.81%. Comparing base (8783e37) to head (82fb964).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #281      +/-   ##
==========================================
+ Coverage   97.69%   97.81%   +0.11%     
==========================================
  Files          15       15              
  Lines        2172     2375     +203     
  Branches      401      432      +31     
==========================================
+ Hits         2122     2323     +201     
- Misses         21       24       +3     
+ Partials       29       28       -1     

Files with missing lines	Coverage Δ
scrapy_zyte_api/_request_fingerprinter.py	99.42% <100.00%> (+0.51%)	⬆️
scrapy_zyte_api/providers.py	95.29% <100.00%> (+0.14%)	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wRAR]

My only question is whether this makes costly calculations for requests passed to it that are already made for them somewhere else

[AdrianAtZyte]

I’ve done my best to minimize this now (as of d5b4863).

[Copilot]

Pull request overview

This PR updates request fingerprinting to incorporate scrapy-poet/Zyte API provider parameters so that requests to the same URL with different provider options no longer share the same fingerprint (and therefore won’t be incorrectly deduplicated/skipped).

Changes:

• Extend ScrapyZyteAPIRequestFingerprinter to compute a provider-aware fingerprint (via scrapy-poet’s dependency plan) and combine it with the “regular” Zyte API fingerprint when appropriate.
• Refactor Zyte API provider meta construction into a reusable helper (_build_zyte_api_provider_meta) and add coverage around AnyResponse behavior when HttpResponse is already available.
• Add/adjust tests around the new fingerprint combination logic and provider validation behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scrapy_zyte_api/_request_fingerprinter.py	Adds provider-aware fingerprinting and combines provider + regular fingerprints, including special handling for provider-only requests.
scrapy_zyte_api/providers.py	Introduces _build_zyte_api_provider_meta helper and reuses it inside ZyteApiProvider.__call__.
tests/test_request_fingerprinter.py	Adds unit tests for provider/regular fingerprint combination and provider-only behavior.
tests/test_providers.py	Imports and tests _build_zyte_api_provider_meta, plus adds a test for unannotated Actions dependencies.
tests/test_sessions_enabled.py	Sets RETRY_TIMES = 0 for test stability/reproducibility.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.