Skip to content

Allow iterating over batches of dataset items, instead of single items only #32

@gullmar

Description

@gullmar

E.g., add iterateBatches and greedyIterateBatches functions in ExtDatasetClient.

It would improve cases like this:

for await (const post of client.dataset<Post>(postsRun.defaultDatasetId).greedyIterate({
    offset: state.postsScraperRun.seenPosts,
})) {
    addPost(post);
    ++state.postsScraperRun.seenPosts;
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions