Skip to content

Commit fb1356a

Browse files
authored
test: poll for dataset items in crawlee e2e tests (#1086)
`test_playwright_crawler` failed on master ([run 31375518173](https://github.com/apify/apify-sdk-python/actions/runs/31375518173/job/93413747650)) with `AssertionError: Missing product: Widget A` / `assert 'Widget A' in {}`, while the run itself succeeded and `assert items.count == 3` passed on the line above. The dataset read in `verify_crawler_results` happens immediately after the run finishes. The API returned an empty item list while its pagination headers already reported the final item count, and `DatasetItemsPage.count` is `max(x-apify-pagination-count, len(items))` - so the count assertion passed on the headers alone and the test only broke once it looked at the items. The read now polls with `poll_until_condition` (30s ceiling) until all pushed items are returned, and asserts on `len(items.items)` instead of the header-derived `count`, which was the misleading part. The helper is shared by all six crawler e2e tests. Verified by replaying the exact response shape from CI (empty `items`, `count=3`): the old helper reproduces `Missing product: Widget A`, the new one passes after polling, an immediately consistent read still takes a single call, and wrong or extra items still fail fast. `tests/e2e/test_crawlee` against the platform: `test_playwright_crawler`, `test_adaptive_playwright_crawler`, `test_http_crawler` and `test_basic_crawler` pass; `test_parsel_crawler` and `test_beautifulsoup_crawler` could not run on my account (Actor-count limit), they fail before reaching this helper. *✍️ Drafted by Claude Code*
1 parent 3212e1c commit fb1356a

1 file changed

Lines changed: 12 additions & 3 deletions

File tree

tests/e2e/test_crawlee/conftest.py

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
from pathlib import Path
55
from typing import TYPE_CHECKING
66

7+
from ..._utils import poll_until_condition
8+
79
if TYPE_CHECKING:
810
from apify_client._models import Run
911
from apify_client._resource_clients import ActorClientAsync
@@ -39,9 +41,16 @@ async def verify_crawler_results(
3941
"""Verify dataset items and KVS record after a crawler Actor run."""
4042
assert run_result.status == 'SUCCEEDED'
4143

42-
# Verify dataset items.
43-
items = await actor.last_run().dataset().list_items()
44-
assert items.count == 3
44+
# Verify dataset items. The dataset is eventually consistent - right after a run finishes, the API can already
45+
# report the final item count in the pagination headers while the items themselves are not returned yet - so poll
46+
# until all the pushed items are readable.
47+
dataset = actor.last_run().dataset()
48+
items = await poll_until_condition(
49+
dataset.list_items,
50+
lambda page: len(page.items) >= len(_EXPECTED_PRODUCTS),
51+
timeout=30,
52+
)
53+
assert len(items.items) == len(_EXPECTED_PRODUCTS)
4554

4655
items_by_name = {item['name']: item for item in items.items}
4756

0 commit comments

Comments
 (0)