Commit fb1356a
authored
test: poll for dataset items in crawlee e2e tests (#1086)
`test_playwright_crawler` failed on master ([run
31375518173](https://github.com/apify/apify-sdk-python/actions/runs/31375518173/job/93413747650))
with `AssertionError: Missing product: Widget A` / `assert 'Widget A' in
{}`, while the run itself succeeded and `assert items.count == 3` passed
on the line above.
The dataset read in `verify_crawler_results` happens immediately after
the run finishes. The API returned an empty item list while its
pagination headers already reported the final item count, and
`DatasetItemsPage.count` is `max(x-apify-pagination-count, len(items))`
- so the count assertion passed on the headers alone and the test only
broke once it looked at the items.
The read now polls with `poll_until_condition` (30s ceiling) until all
pushed items are returned, and asserts on `len(items.items)` instead of
the header-derived `count`, which was the misleading part. The helper is
shared by all six crawler e2e tests.
Verified by replaying the exact response shape from CI (empty `items`,
`count=3`): the old helper reproduces `Missing product: Widget A`, the
new one passes after polling, an immediately consistent read still takes
a single call, and wrong or extra items still fail fast.
`tests/e2e/test_crawlee` against the platform:
`test_playwright_crawler`, `test_adaptive_playwright_crawler`,
`test_http_crawler` and `test_basic_crawler` pass; `test_parsel_crawler`
and `test_beautifulsoup_crawler` could not run on my account
(Actor-count limit), they fail before reaching this helper.
*✍️ Drafted by Claude Code*1 parent 3212e1c commit fb1356a
1 file changed
Lines changed: 12 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
| 8 | + | |
7 | 9 | | |
8 | 10 | | |
9 | 11 | | |
| |||
39 | 41 | | |
40 | 42 | | |
41 | 43 | | |
42 | | - | |
43 | | - | |
44 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
45 | 54 | | |
46 | 55 | | |
47 | 56 | | |
| |||
0 commit comments