feat: performance evaluation

[jirispilka]

Based on @matyascimbulka's suggestion, I refactored the code and moved preNavigationHooks to a separate function so that selecting blockMedia: true/false does not create a new instance of the crawler.

There may be a better way to block media, but it didn’t work for me—perhaps @metalwarrior665 can help here?

preNavigationHooks: [
    async ({ blockRequests }) => {
        // Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
        await blockRequests({
            extraUrlPatterns: ['adsbygoogle.js'],
        });
    },
],

Another issue (#60) in standby mode causes multiple crawlers to be created without reason. I’ll leave this for a separate PR.

And some number not as good as I hoped for but still it is an improvement

[matyascimbulka]

Thank you for implementing the changes. There was no need to start new crawler for blocking media.

I'm not sure why the blockRequests function doesn't work. But the page.route function seems to be the way to go for this use case (outside of Crawlee).

[MQ37]

LGTM 👍 And thank you for fixing this, I haven't noticed that it spawns another crawler instance.

[metalwarrior665]

Let's test the perf a bit more

[metalwarrior665]

page.route disables native browser cache which is why blockRequests is normally recommended (that is a native Chromium CDP call). The cache disabling is only bad if you do more requests for the same site. I would do a perf test on more URLs of the same site and test more sites because this could slow us down as well.